{"id":18957,"date":"2018-06-29T04:15:16","date_gmt":"2018-06-29T04:15:16","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=18957"},"modified":"2021-05-12T11:09:09","modified_gmt":"2021-05-12T05:39:09","slug":"pyspark-sparkconf","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/","title":{"rendered":"PySpark SparkConf &#8211; Attributes and Applications"},"content":{"rendered":"<p><span style=\"font-weight: 400\">In our last <strong>Pyspark tutorial<\/strong>, we saw <strong>Pyspark Serializers<\/strong>. Today, we will discuss PySpark SparkConf. Moreover, we will see attributes in PySpark SparkConf and running Spark Applications. <\/span><\/p>\n<p><span style=\"font-weight: 400\">Also, we\u00a0will learn PySpark SparkConf example. As we need to set a few configurations and parameters, to run a <strong>Spark<\/strong> application on the local\/cluster for that we use SparkConf. So, to learn to run SparkConf using PySpark, this document will help.\u00a0<\/span><\/p>\n<p>So, let&#8217;s start PySpark SparkConf.<\/p>\n<h2>What is PySpark SparkConf?<\/h2>\n<p><span style=\"font-weight: 400\">We need to set a few configurations and parameters, to run a Spark application on the local\/cluster, this is what SparkConf helps with. Basically, \u00a0to run a Spark application, it offers configurations. <\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><strong>Code<\/strong><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">For PySpark, here is the code block which has the details of a SparkConf class:<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">class pyspark.SparkConf (\r\n loadDefaults = True,\r\n _jvm = None,\r\n _jconf = None\r\n)<\/pre>\n<p><span style=\"font-weight: 400\">Basically, with SparkConf() we will create a SparkConf object first. So, that will load the values from spark. Even <strong>Java<\/strong> system properties. Hence, by using the SparkConf object, now we can set different parameters and their parameters will take priority over the system properties.<\/span><\/p>\n<p><span style=\"font-weight: 400\">However, there are better methods, which support chaining, in a SparkConf class. Let\u2019s say, we can write conf.setAppName(\u201cPySpark App\u201d).setMaster(\u201clocal\u201d). Though, it cannot be modified by any user once we pass a SparkConf object to Apache Spark.<\/span><\/p>\n<h2>Attributes of PySpark SparkConf<\/h2>\n<p><span style=\"font-weight: 400\">Thus here are the most commonly used attributes of SparkConf:<\/span><\/p>\n<div id=\"attachment_19487\" style=\"width: 1210px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf.png\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-19487\" class=\"wp-image-19487 size-full\" src=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf.png\" alt=\"Attributes of PySpark SparkConf\" width=\"1200\" height=\"628\" srcset=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf.png 1200w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf-150x79.png 150w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf-300x157.png 300w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf-768x402.png 768w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Attributes-of-PySpark-SparkConf-1024x536.png 1024w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><p id=\"caption-attachment-19487\" class=\"wp-caption-text\">Attributes of PySpark SparkConf<\/p><\/div>\n<h3>i. set(key, value)<\/h3>\n<p><span style=\"font-weight: 400\">It helps to set a configuration property.<\/span><\/p>\n<h3>ii. setMaster(value)<\/h3>\n<p><span style=\"font-weight: 400\">In order to set the master URL, we use it.<\/span><\/p>\n<h3>iii. setAppName(value)<\/h3>\n<p><span style=\"font-weight: 400\">We use it to set an application name.<\/span><\/p>\n<h3>iv. get(key, defaultValue=None)<\/h3>\n<p><span style=\"font-weight: 400\">It helps to get a configuration value of a key.<\/span><\/p>\n<h3>v. setSparkHome(value)<\/h3>\n<p><span style=\"font-weight: 400\">In order to set <strong>Spark installation<\/strong> path on worker nodes, we use it.<\/span><br \/>\n<span style=\"font-weight: 400\">In the following code, we can use to create SparkConf and <strong>SparkContext <\/strong>objects as part of our applications. Also, using sbt console on base directory of our application we can validate:<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">from pyspark import SparkConf,SparkContext\r\nconf = SparkConf().setAppName(\"Spark Demo\").setMaster(\"local\")\r\nsc = SparkContext(conf=conf)<\/pre>\n<h2>Running Spark Applications Using SparkConf<\/h2>\n<p><span style=\"font-weight: 400\">In addition, here are some different contexts in which we can run spark applications:<\/span><\/p>\n<ul>\n<li><strong>local \u2013 conf<\/strong><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">SparkConf.setAppName(&#8220;Spark Demo&#8221;).setMaster(&#8220;local&#8221;)<\/span><\/p>\n<ul>\n<li><strong>yarn-client \u2013 conf<\/strong><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">SparkConf.setAppName(&#8220;Spark Demo&#8221;).setMaster(&#8220;yarn-client&#8221;)<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><strong>mesos URL<\/strong><\/li>\n<li style=\"font-weight: 400\"><strong>spark URL \u2013 conf<\/strong><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\">SparkConf.setAppName(&#8220;Spark Demo&#8221;).setMaster(&#8220;spark master URL&#8221;)<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400\"><strong>Code snippet to get all the properties<\/strong><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400\"> for i in sc.getConf.getAll: print(i)<\/span><br \/>\nSo, this was all about Pyspark SparkConf. Hope you like our explanation.<\/p>\n<h2><span style=\"font-weight: 400\">Conclusion<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Hence, we have learned all about PySpark SparkConf, including its code which will help to create one. Moreover, we discussed different attributes of PySpark SparkConf and also running Spark applications. Still, if any doubt, comment below.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>In our last Pyspark tutorial, we saw Pyspark Serializers. Today, we will discuss PySpark SparkConf. Moreover, we will see attributes in PySpark SparkConf and running Spark Applications. Also, we\u00a0will learn PySpark SparkConf example. As&#46;&#46;&#46;<\/p>\n","protected":false},"author":6,"featured_media":19483,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[5087,10323,12774,12775,12776,12782,15330,15856],"class_list":["post-18957","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pyspark","tag-getkey","tag-pyspark-sparkconf","tag-setkey","tag-setappnamevalue","tag-setmastervalue","tag-setsparkhomevalue","tag-value","tag-what-is-pyspark-sparkconf"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>PySpark SparkConf - Attributes and Applications - DataFlair<\/title>\n<meta name=\"description\" content=\"Pyspark tutorial, PySpark SparkConf, Pyspark SparkConf examples, attributes of PySpark SparkConf, running Spark Applications using PySpark Spark Conf\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark SparkConf - Attributes and Applications - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Pyspark tutorial, PySpark SparkConf, Pyspark SparkConf examples, attributes of PySpark SparkConf, running Spark Applications using PySpark Spark Conf\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-29T04:15:16+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-12T05:39:09+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-SparkConf-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"PySpark SparkConf - Attributes and Applications - DataFlair","description":"Pyspark tutorial, PySpark SparkConf, Pyspark SparkConf examples, attributes of PySpark SparkConf, running Spark Applications using PySpark Spark Conf","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/","og_locale":"en_US","og_type":"article","og_title":"PySpark SparkConf - Attributes and Applications - DataFlair","og_description":"Pyspark tutorial, PySpark SparkConf, Pyspark SparkConf examples, attributes of PySpark SparkConf, running Spark Applications using PySpark Spark Conf","og_url":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-29T04:15:16+00:00","article_modified_time":"2021-05-12T05:39:09+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-SparkConf-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"PySpark SparkConf &#8211; Attributes and Applications","datePublished":"2018-06-29T04:15:16+00:00","dateModified":"2021-05-12T05:39:09+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/"},"wordCount":454,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-SparkConf-01-1.jpg","keywords":["get(key","PySpark SparkConf","set(key","setAppName(value)","setMaster(value)","setSparkHome(value)","value)","What is PySpark SparkConf?"],"articleSection":["PySpark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/","url":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/","name":"PySpark SparkConf - Attributes and Applications - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-SparkConf-01-1.jpg","datePublished":"2018-06-29T04:15:16+00:00","dateModified":"2021-05-12T05:39:09+00:00","description":"Pyspark tutorial, PySpark SparkConf, Pyspark SparkConf examples, attributes of PySpark SparkConf, running Spark Applications using PySpark Spark Conf","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-SparkConf-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-SparkConf-01-1.jpg","width":1200,"height":628,"caption":"PySpark SparkConf - Attributes and Applications"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkconf\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"PySpark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pyspark\/"},{"@type":"ListItem","position":3,"name":"PySpark SparkConf &#8211; Attributes and Applications"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18957","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=18957"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18957\/revisions"}],"predecessor-version":[{"id":94284,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18957\/revisions\/94284"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/19483"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=18957"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=18957"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=18957"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}