

{"id":18719,"date":"2018-06-23T04:04:10","date_gmt":"2018-06-23T04:04:10","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=18719"},"modified":"2021-05-12T11:09:11","modified_gmt":"2021-05-12T05:39:11","slug":"pyspark-rdd","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/","title":{"rendered":"PySpark RDD With Operations and Commands"},"content":{"rendered":"<p><span style=\"font-weight: 400\">In our last article, we discussed <strong>PySpark SparkContext<\/strong>. Today in this <strong>PySpark Tutorial<\/strong>, we will see PySpark RDD with operations. After installation and configuration of PySpark on our system, we can easily program in <strong>Python<\/strong> on Apache Spark. <\/span><\/p>\n<p><span style=\"font-weight: 400\">So, this document focus on manipulating PySpark RDD by applying operations (Transformation and Actions). Well to understand PySpark RDD, we have to learn the basic concept of Spark RDD. Moreover, we will see how to create PySpark RDD.<\/span><\/p>\n<p><span style=\"font-weight: 400\">So, let\u2019s begin PySpark RDD.<\/span><\/p>\n<h2><span style=\"font-weight: 400\">What is Spark RDD?<\/span><\/h2>\n<p><span style=\"font-weight: 400\">An Acronym RDD\u00a0refers to Resilient Distributed Dataset. Basically, RDD is the key abstraction of Apache <strong>Spark<\/strong>. <\/span><\/p>\n<p><span style=\"font-weight: 400\">In order\u00a0to do parallel processing on a cluster, these are the elements that run and operate on multiple nodes. Moreover, it is immutable in nature,\u00a0that says\u00a0as soon as we create an RDD\u00a0we cannot change it.<\/span><\/p>\n<p><span style=\"font-weight: 400\">In addition, RDDs have the best feature that is &#8220;fault tolerance&#8221;. It means if any failure occurs they recover automatically.<\/span><\/p>\n<h3>a. Ways to create Spark RDD<\/h3>\n<p>So,\u00a0 to create Spark RDDs,\u00a0there are 3 ways:<br \/>\ni. Parallelized collections<br \/>\nii. External datasets<br \/>\niii. Existing RDDs<\/p>\n<h3>b. Spark RDDs operations<\/h3>\n<p>Moreover, to achieve a certain task, we can apply multiple operations on these RDDs.<br \/>\ni. Transformation Operations<br \/>\nTransformation Operations creates a new Spark RDD from the existing one. In addition,\u00a0this passes the dataset to the function and then returns new dataset as a result.<br \/>\nii. Action Operations<br \/>\nAnd, this operation returns final result to driver program or also writes it to the external data store.<\/p>\n<h2><span style=\"font-weight: 400\">How to Create PySpark RDD?<\/span><\/h2>\n<p><span style=\"font-weight: 400\">At very first, we need to create a PySpark RDD to apply any operation in PySpark. For that, here is a code block which has the full detail of a PySpark RDD Class \u2212<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">class pyspark.RDD (\r\n  jrdd,\r\n  ctx,\r\n  jrdd_deserializer = AutoBatchedSerializer(PickleSerializer())\r\n)<\/pre>\n<p><span style=\"font-weight: 400\">Further, let&#8217;s see the way to run a few basic operations using PySpark. So, here is the following code in a Python file creates RDD words,\u00a0basically, that stores a set of words which is mentioned here.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">words = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)<\/pre>\n<h2><span style=\"font-weight: 400\">Operations in PySpark RDD<\/span><\/h2>\n<p><span style=\"font-weight: 400\">Now, we will now run a few operations on those words:<\/span><\/p>\n<div id=\"attachment_19429\" style=\"width: 1210px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-19429\" class=\"wp-image-19429 size-full\" src=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01.jpg\" alt=\"PySpark RDD\" width=\"1200\" height=\"628\" srcset=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01.jpg 1200w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01-150x79.jpg 150w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01-300x157.jpg 300w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01-768x402.jpg 768w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Operations-in-PySpark-RDD-01-1024x536.jpg 1024w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><p id=\"caption-attachment-19429\" class=\"wp-caption-text\">Operations in PySpark RDD<\/p><\/div>\n<h3><span style=\"font-weight: 400\">i. count()<\/span><\/h3>\n<p><span style=\"font-weight: 400\">With this operation, the number of elements in the RDD is returned.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------count.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"count app\")\r\nwords = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)\r\ncounts = words.count()\r\nprint \"Number of elements in RDD -&gt; %i\" % (counts)\r\n----------------------------------------count.py---------------------------------------<\/pre>\n<h4><strong>Command <\/strong><\/h4>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit count.py<\/pre>\n<p><strong>Outcome<\/strong><br \/>\n<b>Number of elements in RDD \u2192 8<\/b><\/p>\n<h3><span style=\"font-weight: 400\">ii. collect()<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Basically, this operation returns all the elements in the RDD.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">---------------------------------------collect.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"Collect app\")\r\nwords = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)\r\ncoll = words.collect()\r\nprint \"Elements in RDD -&gt; %s\" % (coll)\r\n----------------------------------------collect.py---------------------------------------<\/pre>\n<p><b>Command<\/b><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit collect.py<\/pre>\n<p><strong>Output <\/strong><br \/>\n<b>Elements in RDD -&gt; [<\/b><br \/>\n<b> \u00a0\u00a0&#8216;scala&#8217;,https:\/\/data-flair.training\/blogs\/python-tutorial-for-beginners\/ <\/b><br \/>\n<b> \u00a0\u00a0&#8216;java&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;hadoop&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;spark&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;akka&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;spark vs hadoop&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;pyspark&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;pyspark and spark&#8217;<\/b><br \/>\n<b>]<\/b><\/p>\n<h3><span style=\"font-weight: 400\">iii. foreach(f)<\/span><\/h3>\n<p><span style=\"font-weight: 400\">foreach(f) operations returns only those elements which meet the condition of the function inside foreach. Here, to prints all the elements in the RDD, we will call a print function in foreach.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------foreach.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"ForEach app\")\r\nwords = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)\r\ndef f(x): print(x)\r\nfore = words.foreach(f)\r\n----------------------------------------foreach.py---------------------------------------<\/pre>\n<p><strong><span style=\"font-family: Verdana, Geneva, sans-serif\">Command<\/span><\/strong><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit foreach.py<\/pre>\n<p><strong>Output<\/strong><br \/>\n<b>scala<\/b><br \/>\n<b>java<\/b><br \/>\n<b>hadoop<\/b><br \/>\n<b>spark<\/b><br \/>\n<b>akka<\/b><br \/>\n<b>spark vs hadoop<\/b><br \/>\n<b>pyspark<\/b><br \/>\n<b>pyspark and spark<\/b><\/p>\n<h3><span style=\"font-weight: 400\">iv.\u00a0cc<\/span><\/h3>\n<p><span style=\"font-weight: 400\">After applying this operation, we will get a new RDD which contains the elements,\u00a0those satisfy the function inside the filter. Now, here we filter out the strings containing &#8221;spark&#8221;, in the following example.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------filter.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"Filter app\")\r\nwords = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)\r\nwords_filter = words.filter(lambda x: 'spark' in x)\r\nfiltered = words_filter.collect()\r\nprint \"Fitered RDD -&gt; %s\" % (filtered)\r\n----------------------------------------filter.py----------------------------------------<\/pre>\n<p><strong>Command<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit filter.py<\/pre>\n<h4><strong>Output <\/strong><\/h4>\n<p><b>Fitered RDD -&gt; [<\/b><br \/>\n<b> \u00a0\u00a0&#8216;spark&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;spark vs hadoop&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;pyspark&#8217;, <\/b><br \/>\n<b> \u00a0\u00a0&#8216;pyspark and spark&#8217;<\/b><br \/>\n<b>]<\/b><\/p>\n<h3><span style=\"font-weight: 400\">v. map(f, preservesPartitioning = False)<\/span><\/h3>\n<p><span style=\"font-weight: 400\">By applying a function to each element in the RDD, a new RDD is returned. Now, here, we form a key-value pair and map every string with a value of 1 in the following example.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------map.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"Map app\")\r\nwords = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)\r\nwords_map = words.map(lambda x: (x, 1))\r\nmapping = words_map.collect()\r\nprint \"Key value pair -&gt; %s\" % (mapping)\r\n----------------------------------------map.py---------------------------------------<\/pre>\n<p><strong>Command<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit map.py<\/pre>\n<p><strong>Output<\/strong><br \/>\n<b>Key value pair -&gt; [<\/b><br \/>\n<b> \u00a0\u00a0(&#8216;scala&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;java&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;hadoop&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;spark&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;akka&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;spark vs hadoop&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;pyspark&#8217;, 1), <\/b><br \/>\n<b> \u00a0\u00a0(&#8216;pyspark and spark&#8217;, 1)<\/b><br \/>\n<b>]<\/b><\/p>\n<h3><span style=\"font-weight: 400\">vi. reduce(f)<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Here, the element in the RDD is returned, after performing the specified commutative and associative binary operation. So, we are importing add package from the operator and also to carry out a simple addition operation we are applying it on \u2018num\u2019, in the following example.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------reduce.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nfrom operator import add\r\nsc = SparkContext(\"local\", \"Reduce app\")\r\nnums = sc.parallelize([1, 2, 3, 4, 5])\r\nadding = nums.reduce(add)\r\nprint \"Adding all the elements -&gt; %i\" % (adding)\r\n----------------------------------------reduce.py---------------------------------------<\/pre>\n<p><strong>Command<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit reduce.py<\/pre>\n<p><strong>Output<\/strong><br \/>\n<b>Adding all the elements -&gt; 15<\/b><\/p>\n<h3><span style=\"font-weight: 400\">vii. join(other, numPartitions = None)<\/span><\/h3>\n<p><span style=\"font-weight: 400\">This operation returns RDD with a pair of elements with the matching keys as well as all the values for that particular key. So, there is two pair of elements in two different RDDs, in the following example. So, we get an RDD with elements having matching keys and their values, after joining these two RDDs.<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------join.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"Join app\")\r\nx = sc.parallelize([(\"spark\", 1), (\"hadoop\", 4)])\r\ny = sc.parallelize([(\"spark\", 2), (\"hadoop\", 5)])\r\njoined = x.join(y)\r\nfinal = joined.collect()\r\nprint \"Join RDD -&gt; %s\" % (final)\r\n----------------------------------------join.py---------------------------------------<\/pre>\n<p><strong>Command<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit join.py<\/pre>\n<p><strong>Output<\/strong><br \/>\n<b>Join RDD -&gt; [<\/b><br \/>\n<b> \u00a0\u00a0(&#8216;spark&#8217;, (1, 2)), \u00a0<\/b><br \/>\n<b> \u00a0\u00a0(&#8216;hadoop&#8217;, (4, 5))<\/b><br \/>\n<b>]<\/b><\/p>\n<h3><span style=\"font-weight: 400\">viii. cache()<\/span><\/h3>\n<p><span style=\"font-weight: 400\">Moreover, this command, with the default storage level (MEMORY_ONLY), Persist this RDD. Also, we can check if the RDD is cached or not with cache() command<\/span><\/p>\n<pre class=\"EnlighterJSRAW\">----------------------------------------cache.py---------------------------------------\r\nfrom pyspark import SparkContext\r\nsc = SparkContext(\"local\", \"Cache app\")\r\nwords = sc.parallelize (\r\n  [\"scala\",\r\n  \"java\",\r\n  \"hadoop\",\r\n  \"spark\",\r\n  \"akka\",\r\n  \"spark vs hadoop\",\r\n  \"pyspark\",\r\n  \"pyspark and spark\"]\r\n)\r\nwords.cache()\r\ncaching = words.persist().is_cached\r\nprint \"Words got chached &gt; %s\" % (caching)\r\n----------------------------------------cache.py---------------------------------------<\/pre>\n<p><strong>Command<\/strong><\/p>\n<pre class=\"EnlighterJSRAW\">$SPARK_HOME\/bin\/spark-submit cache.py<\/pre>\n<p><strong>Output<\/strong><br \/>\n<b>Words got cached -&gt; True<\/b><br \/>\nSo, this was all about PySpark RDD and its operations. Hope you like our explanation.<\/p>\n<h2><span style=\"font-weight: 400\">Conclusion: PySpark RDD<\/span><\/h2>\n<p>Hence, we have seen the concept of\u00a0PySpark RDD. Thus it includes some of the most important operations\u00a0which are done on PySpark RDD. Also, we have seen the way to create a PySpark RDD in detail. Still, if any doubt, ask in the comment tab.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In our last article, we discussed PySpark SparkContext. Today in this PySpark Tutorial, we will see PySpark RDD with operations. After installation and configuration of PySpark on our system, we can easily program in&#46;&#46;&#46;<\/p>\n","protected":false},"author":6,"featured_media":19427,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[2269,2629,3010,4692,4856,7804,8517,10287,10319,11450,13109,15437,15961],"class_list":["post-18719","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pyspark","tag-cache","tag-collect","tag-count","tag-filter","tag-foreach","tag-join","tag-map","tag-pyspark-rdd","tag-pyspark-rdd-operations","tag-reduce","tag-spark-rdd-operations","tag-ways-to-create-spark-rdd","tag-what-is-spark-rdd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>PySpark RDD With Operations and Commands - DataFlair<\/title>\n<meta name=\"description\" content=\"In this document, we are focusing on manipulating PySpark RDD by applying several operations (Transformation and Actions).\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark RDD With Operations and Commands - DataFlair\" \/>\n<meta property=\"og:description\" content=\"In this document, we are focusing on manipulating PySpark RDD by applying several operations (Transformation and Actions).\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-23T04:04:10+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-12T05:39:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-RDD-with-Operations-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"PySpark RDD With Operations and Commands - DataFlair","description":"In this document, we are focusing on manipulating PySpark RDD by applying several operations (Transformation and Actions).","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/","og_locale":"en_US","og_type":"article","og_title":"PySpark RDD With Operations and Commands - DataFlair","og_description":"In this document, we are focusing on manipulating PySpark RDD by applying several operations (Transformation and Actions).","og_url":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-23T04:04:10+00:00","article_modified_time":"2021-05-12T05:39:11+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-RDD-with-Operations-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"PySpark RDD With Operations and Commands","datePublished":"2018-06-23T04:04:10+00:00","dateModified":"2021-05-12T05:39:11+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/"},"wordCount":804,"commentCount":2,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-RDD-with-Operations-01.jpg","keywords":["cache","collect","count","filter","foreach","join","MAP","PySpark - RDD","PySpark RDD operations","reduce","Spark RDD Operations","Ways to create Spark RDD","what is spark RDD"],"articleSection":["PySpark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/","url":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/","name":"PySpark RDD With Operations and Commands - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-RDD-with-Operations-01.jpg","datePublished":"2018-06-23T04:04:10+00:00","dateModified":"2021-05-12T05:39:11+00:00","description":"In this document, we are focusing on manipulating PySpark RDD by applying several operations (Transformation and Actions).","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/pyspark-rdd\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-RDD-with-Operations-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/PySpark-RDD-with-Operations-01.jpg","width":1200,"height":628,"caption":"PySpark RDD With Operations and Commands"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/pyspark-rdd\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"PySpark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pyspark\/"},{"@type":"ListItem","position":3,"name":"PySpark RDD With Operations and Commands"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=18719"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18719\/revisions"}],"predecessor-version":[{"id":94293,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/18719\/revisions\/94293"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/19427"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=18719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=18719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=18719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}