{"id":5628,"date":"2018-01-17T05:03:20","date_gmt":"2018-01-17T05:03:20","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=5628"},"modified":"2018-11-16T15:07:41","modified_gmt":"2018-11-16T09:37:41","slug":"sparksession-and-rstudio","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/","title":{"rendered":"Ways to start SparkR – Using Sparksession or RStudio"},"content":{"rendered":"

1. Objective<\/h2>\n

A distributed collection of data organized into named columns is SparkDataFrame. Basically, there are two ways to which we can start SparkR<\/strong><\/a>. For example, by using Sparksession or\u00a0R<\/strong><\/a>studio. In this article, we will learn both the process in detail. Before, ways to start SparkR, we will also learn a brief introduction\u00a0to SparkDataFrame.<\/span><\/p>\n

\"Ways<\/a>

Ways to start SparkR – Sparksession and RStudio<\/p><\/div>\n

2. SparkDataFrame<\/h2>\n

A distributed collection of data organized into named columns is SparkDataFrame. Basically, it is as same as a table in a relational database or a data frame<\/strong><\/a> in R. Moreover, it has some richer optimizations under the hood. <\/span>
\nIn addition, we can construct SparkDataFrames<\/a><\/strong> from a wide array of sources. For example, structured data files, tables in Hive<\/a><\/strong>, external databases. Also can construct from existing local R data frames<\/strong><\/a>.<\/span>
\n<\/span>
\nWe can start SparkR using two kinds of methods. Either using Sparksession or R studio.<\/span><\/p>\n

3. Ways to start SparkR<\/h2>\n

There are several ways to start SparkR,\u00a0let\u2019s discuss\u00a0them in detail.<\/p>\n

a. Using SparkSession<\/h3>\n

Basically, SparkSession is the entry point into SparkR. Also connects your R program to a Spark cluster<\/a><\/strong>. Although, by using sparkR.session we can easily<\/span> create a SparkSession. Moreover, it passes in options. Such as the application name, any spark packages depended on, and many more. Afterwards, we can also work with SparkDataFrames via SparkSession. Since we are working from the sparkR shell. Hence it is important that SparkSession should already be created<\/span> for us. Then we would not need to call sparkR.session.<\/span>
\n<\/span>
\n<\/span>sparkR.session()<\/b><\/p>\n

b.\u00a0Using RStudio<\/h3>\n

Apart from SparkSession, we can also start SparkR from RStudio.<\/span>\u00a0Also, from RStudio, R shell, Rscript or other R IDEs, we can connect our R <\/a><\/strong>program to a Spark cluster. Basically, we need to make sure that to start, SPARK_HOME is set in the environment.
\nMoreover, to check Sys.getenv, load the SparkR package, and call sparkR.session as same as below. Hence, that will check for the Spark installation. Although, if not found, it will be downloaded and cached automatically. Therefore, as an alternative, we can also run install.spark manually.
\n<\/span>
\n<\/span>Furthermore, to call sparkR.session, we could also specify certain Spark driver properties. Although, we can not set these application properties, Runtime Environment programmatically<\/span>.<\/span>
\n Since the driver JVM process would have been started<\/span>. Basically<\/span>, in this case, SparkR<\/a><\/strong> takes care of this for us. Moreover, we can also set in other ways. By passing them as other configuration properties in the sparkConfig argument to sparkR.session()<\/span>.<\/span>
\n<\/span>
\n<\/span>if (nchar(Sys.getenv(“SPARK_HOME”)) < 1) {<\/b>
\n<\/b> \u00a0Sys.setenv(SPARK_HOME = “\/home\/spark”)<\/b>
\n<\/b>}<\/b>
\n<\/b>library(SparkR, lib.loc = c(file.path(Sys.getenv(“SPARK_HOME”), “R”, “lib”)))<\/b>
\n<\/b>sparkR.session(master = “local[*]”, sparkConfig = list(spark.driver.memory = “2g”))<\/b><\/p>\n

c. Spark Driver Properties<\/h3>\n

In addition<\/span>, there are many Spark driver Properties. Basically<\/span>, we can set following Spark driver properties. Although, it is possible by using sparkConfig with sparkR.session from RStudio. <\/span>
\n<\/span><\/p>\n\n\n\n\n\n\n\n\n\n\n
Property Name<\/b><\/td>\nProperty group<\/b><\/td>\nspark-submit equivalent<\/b><\/td>\n<\/tr>\n
spark.master<\/b><\/td>\nApplication Properties<\/b><\/td>\n–master<\/b><\/b><\/td>\n<\/tr>\n
spark.yarn.keytab<\/b><\/td>\nApplication Properties<\/b><\/td>\n–keytab<\/b><\/td>\n<\/tr>\n
spark.yarn.principal<\/b><\/td>\nApplication Properties<\/b><\/td>\n–principal<\/b><\/b><\/td>\n<\/tr>\n
spark.driver.memory<\/b><\/td>\nApplication Properties<\/b><\/td>\n–driver-memory<\/b><\/td>\n<\/tr>\n
spark.driver.extraClassPath<\/b><\/td>\nRuntime Environment<\/b><\/td>\n–driver-class-path<\/b><\/b><\/td>\n<\/tr>\n
spark.driver.extraJavaOptions<\/b><\/td>\nRuntime Environment<\/b><\/td>\n–driver-java-options<\/b><\/td>\n<\/tr>\n
spark.driver.extraLibraryPath<\/b><\/td>\nRuntime Environment<\/b><\/td>\n–driver-library-path<\/b><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

4. Conclusion<\/h2>\n

As a result, we have seen ways to start SparkR by Sparksession and RStudio. However, we have tried to cover all the insights regarding same. Despite that, if any query occurs, feel free to ask in the comment section.
\nA list of best books<\/a><\/strong> to Learn Spark.
\nFor reference<\/strong><\/a>

1. Objective A distributed collection of data organized into named columns is SparkDataFrame. Basically, there are two ways to which we can start SparkR. For example, by using Sparksession or\u00a0Rstudio. In this article, we...<\/p>\n","protected":false},"author":6,"featured_media":42385,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[6329,13054,15446,15966],"class_list":["post-5628","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-how-to-start-sparkr","tag-spark-driver-properties","tag-ways-to-start-sparkr","tag-what-is-sparkdataframe"],"yoast_head":"\nWays to start SparkR - Using Sparksession or RStudio - DataFlair<\/title>\n<meta name=\"description\" content=\"SparkR-what is SparkDataFrames, Ways to start SparkR- from RStudio, Starting Up from SparkSession, Spark Driver Properties with property name & group\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Ways to start SparkR - Using Sparksession or RStudio - DataFlair\" \/>\n<meta property=\"og:description\" content=\"SparkR-what is SparkDataFrames, Ways to start SparkR- from RStudio, Starting Up from SparkSession, Spark Driver Properties with property name & group\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-17T05:03:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-11-16T09:37:41+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/How-to-start-SparkR-Using-sparksession-or-R-studio-01-2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Ways to start SparkR - Using Sparksession or RStudio - DataFlair","description":"SparkR-what is SparkDataFrames, Ways to start SparkR- from RStudio, Starting Up from SparkSession, Spark Driver Properties with property name & group","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/","og_locale":"en_US","og_type":"article","og_title":"Ways to start SparkR - Using Sparksession or RStudio - DataFlair","og_description":"SparkR-what is SparkDataFrames, Ways to start SparkR- from RStudio, Starting Up from SparkSession, Spark Driver Properties with property name & group","og_url":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-01-17T05:03:20+00:00","article_modified_time":"2018-11-16T09:37:41+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/How-to-start-SparkR-Using-sparksession-or-R-studio-01-2.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Ways to start SparkR – Using Sparksession or RStudio","datePublished":"2018-01-17T05:03:20+00:00","dateModified":"2018-11-16T09:37:41+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/"},"wordCount":598,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/How-to-start-SparkR-Using-sparksession-or-R-studio-01-2.jpg","keywords":["How to start SparkR","Spark Driver Properties","Ways to start SparkR","what is SparkDataFrame"],"articleSection":["Apache Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/","url":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/","name":"Ways to start SparkR - Using Sparksession or RStudio - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/How-to-start-SparkR-Using-sparksession-or-R-studio-01-2.jpg","datePublished":"2018-01-17T05:03:20+00:00","dateModified":"2018-11-16T09:37:41+00:00","description":"SparkR-what is SparkDataFrames, Ways to start SparkR- from RStudio, Starting Up from SparkSession, Spark Driver Properties with property name & group","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/How-to-start-SparkR-Using-sparksession-or-R-studio-01-2.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/01\/How-to-start-SparkR-Using-sparksession-or-R-studio-01-2.jpg","width":1200,"height":628,"caption":"Ways to start SparkR - Sparksession and RStudio"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/sparksession-and-rstudio\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/spark\/"},{"@type":"ListItem","position":3,"name":"Ways to start SparkR – Using Sparksession or RStudio"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5628","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=5628"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5628\/revisions"}],"predecessor-version":[{"id":42387,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5628\/revisions\/42387"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/42385"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=5628"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=5628"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=5628"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}