{"id":5644,"date":"2018-01-17T08:53:30","date_gmt":"2018-01-17T08:53:30","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=5644"},"modified":"2018-09-18T11:36:55","modified_gmt":"2018-09-18T06:06:55","slug":"create-sparkdataframes","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/","title":{"rendered":"Ways to Create SparkDataFrames in SparkR"},"content":{"rendered":"

1. Objective<\/h2>\n

Today, in this SparkDataFrames Tutorial, we will learn the whole concept of creating DataFrames in SparkR<\/a>.\u00a0<\/strong>Data is organized as a distributed collection of data into named columns. Basically, that we call a SparkDataFrames in SparkR. Also, there are many ways to create DataFrames in SparkR.<\/p>\n

So, let’s start SparkDataFrames Tutorial.<\/p>\n

\"Creating<\/a>

Creating SparkDataframes in Spark R<\/p><\/div>\n

2. What is SparkDataFrames?<\/h2>\n

Data is organized as a distributed collection of data into named columns. Basically, that we call a SparkDataFrame. Although, it is as same as a table in a relational database or a data frame in R<\/a><\/strong>. Moreover, we can construct a SparkR DataFrame from a wide array of sources. For example, structured data files, tables in Hive, external databases. Also, existing local R data frames are used for construction.<\/span><\/p>\n

3. Ways to Create SparkDataFrames<\/h2>\n

Applications can create DataFrames in Spark, with a SparkSession. Apart from it, we can also create it from several methods. Such as local R data frame, a Hive table, or other data sources. Let\u2019s discuss all in brief. <\/span><\/p>\n

a. From local data frames<\/h3>\n

To create a SparkDataframe, there is one simplest way. That is the conversion of a local R data frame into a SparkDataFrame. Although, we can create by using as DataFrame or createDataFrame. Also, by passing in the local R data frame to create a SparkDataFrame.<\/span>
\nFor example,
\n<\/span><\/p>\n

df <- as.DataFrame(faithful)\r\n# Displays the first part of the SparkDataFrame\r\nhead(df)\r\n## \u00a0eruptions waiting\r\n##1 \u00a0\u00a0\u00a0\u00a03.600 \u00a0\u00a0\u00a0\u00a0\u00a079\r\n##2 \u00a0\u00a0\u00a0\u00a01.800 \u00a0\u00a0\u00a0\u00a0\u00a054\r\n##3 \u00a0\u00a0\u00a0\u00a03.333 \u00a0\u00a0\u00a0\u00a0\u00a074<\/pre>\n
b. From Data Sources<\/h3>\n
Through the SparkR SparkDataFrame interface, SparkR supports operating on a variety of data sources. Basically, for creating SparkDataFrames, the general method from data sources is read.df. Generally, this method takes in the path for the file to load. Also, the type of data source and the currently active SparkSession will be automatically used. Moreover, SparkR supports reading JSON, CSV and parquet files natively.<\/span>
\n
\n<\/span>In addition, we can add these packages by specifying two conditions.<\/span> Such as, if packages with spark-submit or sparkR commands. Else, if initializing SparkSession with the sparkPackages parameter. Either in an interactive R shell or from RStudio.
\n<\/span><\/p>\n
sparkR.session(sparkPackages = \"com.databricks:spark-avro_2.11:3.0.0\")<\/pre>\n
Basically, we have seen how to use data sources using an example, JSON input file.<\/span> Although the file that is used here is not a typical JSON file. Basically, each line in the file must contain a separate, valid JSON object.
\n
\n<\/span><\/p>\n
people <- read.df(\".\/examples\/src\/main\/resources\/people.json\", \"json\")\r\nhead(people)\r\n## \u00a0age \u00a0\u00a0\u00a0name\r\n##1 \u00a0NA Michael\r\n##2 \u00a030 \u00a0\u00a0\u00a0Andy\r\n##3 \u00a019 \u00a0Justin\r\n# SparkR automatically infers the schema from the JSON file\r\nprintSchema(people)\r\n# root\r\n# \u00a0|-- age: long (nullable = true)\r\n# \u00a0|-- name: string (nullable = true)\r\n# Similarly, multiple files can be read with read.json\r\npeople <- read.json(c(\".\/examples\/src\/main\/resources\/people.json\", \".\/examples\/src\/main\/resources\/people2.json\"))\r\nThe data sources API natively supports CSV formatted input files. For more information please refer to SparkR read.df API documentation.\r\ndf <- read.df(csvPath, \"csv\", header = \"true\", inferSchema = \"true\", na.strings = \"NA\")<\/pre>\n
In addition, we can also use data sources API to save out SparkDataFrames into multiple file formats.<\/span>
\nFor example, we can save the SparkDataFrame from the previous example to a Parquet file using write.df.<\/span><\/p>\n
write.df(people, path = \"people.parquet\", source = \"parquet\", mode = \"overwrite\")<\/pre>\n
c. From Hive tables<\/h3>\n
We can also use Hive tables to create SparkDataFrames.<\/span> For this, we will need to create a SparkSession with Hive support.\u00a0Also can help to access tables in the Hive MetaStore. Although it is very important to note that Spark should have been built with Hive support.\u00a0Although, SparkR attempt to create a SparkSession with Hive support enabled by default.<\/span> (enableHiveSupport = TRUE).
\nsparkR.session()<\/b>
\nFor example,<\/p>\n
sql(\"CREATE TABLE IF NOT EXISTS src (key INT, value STRING)\")\r\nsql(\"LOAD DATA LOCAL INPATH 'examples\/src\/main\/resources\/kv1.txt' INTO TABLE src\")\r\n\r\n# Queries can be expressed in HiveQL.\r\nresults <- sql(\"FROM src SELECT key, value\")\r\n# results is now a SparkDataFrame\r\nhead(results)\r\n## \u00a0key \u00a0\u00a0value\r\n## 1 238 val_238\r\n## 2 \u00a086 \u00a0val_86\r\n## 3 311 val_311<\/pre>\n
So, this was all in SparkDataFrames. Hope you like our explanation.<\/p>\n
4. Conclusion<\/h2>\n
Hence, we have seen all the methods to construct a SparkR SparkDataFrame. Also, we have learned different ways to create Data frames in spark with local R data frame, a Hive table, and data sources. Although, we have covered all the insights regarding. Still, if any query arises, feel free to ask in the comment section.
\nFor reference<\/a>.
1. Objective Today, in this SparkDataFrames Tutorial, we will learn the whole concept of creating DataFrames in SparkR.\u00a0Data is organized as a distributed collection of data into named columns. Basically, that we call a...<\/p>\n","protected":false},"author":6,"featured_media":6215,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[3062,3088,3536,4409,13047,15438],"class_list":["post-5644","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-create-dataframes-in-spark-r","tag-create-sparkdataframes","tag-dataframe-in-spark","tag-examples-of-sparkdataframe","tag-spark-dataframe","tag-ways-to-create-sparkdataframe"],"yoast_head":"\nWays to Create SparkDataFrames in SparkR - DataFlair<\/title>\n<meta name=\"description\" content=\"SparkDataFrames in SparkR, Ways to Create SparkDataFrame from local dataframes, create SparkDataFrame from hive table,create SparkDataFrame from Data source\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Ways to Create SparkDataFrames in SparkR - DataFlair\" \/>\n<meta property=\"og:description\" content=\"SparkDataFrames in SparkR, Ways to Create SparkDataFrame from local dataframes, create SparkDataFrame from hive table,create SparkDataFrame from Data source\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-17T08:53:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-09-18T06:06:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Creating-SparkDataFrames-in-SparkR.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Ways to Create SparkDataFrames in SparkR - DataFlair","description":"SparkDataFrames in SparkR, Ways to Create SparkDataFrame from local dataframes, create SparkDataFrame from hive table,create SparkDataFrame from Data source","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/","og_locale":"en_US","og_type":"article","og_title":"Ways to Create SparkDataFrames in SparkR - DataFlair","og_description":"SparkDataFrames in SparkR, Ways to Create SparkDataFrame from local dataframes, create SparkDataFrame from hive table,create SparkDataFrame from Data source","og_url":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-01-17T08:53:30+00:00","article_modified_time":"2018-09-18T06:06:55+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Creating-SparkDataFrames-in-SparkR.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Ways to Create SparkDataFrames in SparkR","datePublished":"2018-01-17T08:53:30+00:00","dateModified":"2018-09-18T06:06:55+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/"},"wordCount":559,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Creating-SparkDataFrames-in-SparkR.jpg","keywords":["Create DataFrames in Spark R","Create SparkDataFrames","Dataframe in Spark","Examples of SparkDataFrame","Spark DataFrame","Ways to Create SparkDataFrame"],"articleSection":["Apache Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/","url":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/","name":"Ways to Create SparkDataFrames in SparkR - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Creating-SparkDataFrames-in-SparkR.jpg","datePublished":"2018-01-17T08:53:30+00:00","dateModified":"2018-09-18T06:06:55+00:00","description":"SparkDataFrames in SparkR, Ways to Create SparkDataFrame from local dataframes, create SparkDataFrame from hive table,create SparkDataFrame from Data source","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Creating-SparkDataFrames-in-SparkR.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/Creating-SparkDataFrames-in-SparkR.jpg","width":1200,"height":628,"caption":"Create SparkDataFrame in SparkR"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/create-sparkdataframes\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/spark\/"},{"@type":"ListItem","position":3,"name":"Ways to Create SparkDataFrames in SparkR"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5644","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=5644"}],"version-history":[{"count":4,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5644\/revisions"}],"predecessor-version":[{"id":34575,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5644\/revisions\/34575"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/6215"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=5644"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=5644"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=5644"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}