{"id":2621,"date":"2017-05-22T11:19:40","date_gmt":"2017-05-22T11:19:40","guid":{"rendered":"http:\/\/data-flair.training\/blogs\/?p=2621"},"modified":"2018-11-16T13:56:55","modified_gmt":"2018-11-16T08:26:55","slug":"apache-spark-rdd-features","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/","title":{"rendered":"11 Shining Features of Spark RDD You Must Know"},"content":{"rendered":"<h2>1. Objective<\/h2>\n<p>In this Spark tutorial, we will come across various twinkling<strong> features of Spark RDD<\/strong>. Before moving forward to this blog make yourself familiar with the concepts of <strong><a href=\"http:\/\/data-flair.training\/blogs\/why-learn-big-data-use-cases\/\">Bigdata<\/a> <\/strong>and Apache Spark. This blog also contains the<a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-tutorial-quickstart-introduction\/\"> introduction to Apache Spark<\/a> RDD and its operations along with the methods to create RDD.<\/p>\n<p>To play with RDD<a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-installation-in-standalone-mode\/\"> learn Apache Spark Installation in standalone mode<\/a> and <a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-installation-on-multi-node-cluster-step-by-step-guide\/\">Multi-cluster node.<\/a><\/p>\n<div id=\"attachment_42327\" style=\"width: 1210px\" class=\"wp-caption aligncenter\"><a href=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg\"><img loading=\"lazy\" decoding=\"async\" aria-describedby=\"caption-attachment-42327\" class=\"size-full wp-image-42327\" src=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg\" alt=\"11 Shining Features of Spark RDD You Must Know\" width=\"1200\" height=\"628\" srcset=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg 1200w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1-150x79.jpg 150w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1-300x157.jpg 300w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1-768x402.jpg 768w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1-1024x536.jpg 1024w, https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1-520x272.jpg 520w\" sizes=\"auto, (max-width: 1200px) 100vw, 1200px\" \/><\/a><p id=\"caption-attachment-42327\" class=\"wp-caption-text\">11 Shining Features of Spark RDD You Must Know<\/p><\/div>\n<h2>2. Apache Spark RDD<\/h2>\n<p><a href=\"http:\/\/data-flair.training\/blogs\/rdd-in-apache-spark\/\"><strong>RDD<\/strong><\/a> stands <strong>Resilient Distributed Dataset<\/strong>. RDDs are the fundamental abstraction of Apache Spark. It is an immutable distributed collection of the dataset. Each dataset in RDD is divided into logical partitions. On the different node of the cluster, we can compute These partitions. RDDs are a read-only partitioned collection of record. we can<a href=\"http:\/\/data-flair.training\/blogs\/how-to-create-rdds-in-apache-spark\/\">\u00a0create RDD in three ways<\/a>:<\/p>\n<ul>\n<li><strong>Parallelizing<\/strong> already existing collection in driver program.<\/li>\n<li><strong>Referencing a dataset<\/strong> in an external storage system (e.g. <a href=\"http:\/\/data-flair.training\/blogs\/apache-hadoop-hdfs-introduction-tutorial\/\">HDFS<\/a>, <a href=\"http:\/\/data-flair.training\/blogs\/hbase-tutorial-beginners-guide\/\">Hbase<\/a>, shared file system).<\/li>\n<li>Creating RDD <strong>from already existing RDDs<\/strong>.<\/li>\n<\/ul>\n<p>There are two operations in RDD namely <a href=\"http:\/\/data-flair.training\/blogs\/rdd-transformations-actions-apis-apache-spark\/\">transformation and Action<\/a>.<\/p>\n<h2>3. Sparkling Features of Spark RDD<\/h2>\n<p>There are several advantages of using RDD. Some of them are-<\/p>\n<h3>i. In-memory computation<\/h3>\n<p>The data inside RDD are stored in memory for as long as you want to store. Keeping the data in-memory improves the performance by an order of magnitudes. refer this comprehensive guide to <a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-in-memory-computing\/\">Learn Spark in-memory computation<\/a> in detail.<\/p>\n<h3>ii. Lazy Evaluation<\/h3>\n<p>The data inside RDDs are not evaluated on the go. The changes or the computation is performed only after an action is triggered. Thus, it limits how much work it has to do. Follow this guide to <a href=\"http:\/\/data-flair.training\/blogs\/lazy-evaluation-in-apache-spark-guide\/\">learn Spark lazy evaluation in great detail<\/a>.<\/p>\n<h3>iii. Fault Tolerance<\/h3>\n<p>Upon the failure of worker node, using lineage of operations we can re-compute the lost partition of RDD from the original one. Thus, we can easily recover the lost data. <a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-streaming-fault-tolerance\/\">Learn Fault tolerance is Spark in detail<\/a>.<\/p>\n<h3>iv. Immutability<\/h3>\n<p>RDDS are immutable in nature meaning once we create an RDD we can not manipulate it. And if we perform any transformation, it creates new RDD. We achieve consistency through immutability.<\/p>\n<h3>v. Persistence<\/h3>\n<p>We can store the frequently used RDD in in-memory and we can also\u00a0retrieve them directly from memory without going to disk, this speedup the execution. We can perform Multiple operations on the same data, this happens by storing the data explicitly in memory by calling persist() or cache() function. Follow this guide for the <a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-rdd-persistence-caching\/\">detailed study of RDD persistence in Spark<\/a>.<\/p>\n<h3>vi. Partitioning<\/h3>\n<p>RDD partition the records logically and distributes the data across various nodes in the cluster. The logical divisions are only for processing and internally it has no division. Thus, it provides parallelism.<\/p>\n<h3>vii. Parallel<strong>\u00a0<\/strong><\/h3>\n<p>Rdd, process the data\u00a0parallelly over the cluster.<\/p>\n<h3>viii. Location-Stickiness<\/h3>\n<p>RDDs are capable of defining placement preference to compute partitions. Placement preference refers to information about the location of RDD. The <strong>DAGScheduler<\/strong> places the partitions in such a way that task is close to data as much as possible. Thus speed up computation. Follow this guide to <a href=\"http:\/\/data-flair.training\/blogs\/directed-acyclic-graph-dag-in-apache-spark\/\">learn What is DAG?<\/a><\/p>\n<h3>ix. Coarse-grained Operation<\/h3>\n<p>We apply coarse-grained transformations to RDD<strong>.<\/strong> Coarse-grained meaning the operation applies to the whole dataset not on an individual element in the data set of RDD.<\/p>\n<h3>x. Typed<\/h3>\n<p>We can have RDD of various types like: RDD [int], RDD [long], RDD [string].<\/p>\n<h3><span id=\":7l.co\" class=\"tL8wMe EMoHub\" dir=\"ltr\">xi. No limitation<\/span><\/h3>\n<div id=\":86.ma\" class=\"Mu SP\">\n<div id=\":86.at\" class=\"xH\">we can have any number of RDD. there is no limit to its number. the limit depends on the size of disk and memory.<\/div>\n<\/div>\n<h2>4. Conclusion<\/h2>\n<div class=\"xH\">Hence, using RDD we can recover the shortcoming of Hadoop MapReduce and can handle the large volume of data, as a result, it decreases the time complexity of the system. Thus the above-mentioned features of Spark RDD make them useful for fast computations and increase the performance of the system.<\/div>\n<div class=\"xH\"><\/div>\n<div class=\"xH\">If you like this post or feel that I have missed some features of Spark RDD, please do leave a comment.<\/div>\n<div class=\"xH\"><\/div>\n<div class=\"xH\"><strong>See Also-<\/strong><\/div>\n<ul>\n<li class=\"xH\"><a href=\"http:\/\/data-flair.training\/blogs\/apache-spark-features\/\">Features of Apache Spark<\/a><\/li>\n<li class=\"xH\"><a href=\"http:\/\/data-flair.training\/blogs\/limitations-of-apache-spark-overcome-spark-drawbacks\/\">Limitations of Apache Spark<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>1. Objective In this Spark tutorial, we will come across various twinkling features of Spark RDD. Before moving forward to this blog make yourself familiar with the concepts of Bigdata and Apache Spark. This&#46;&#46;&#46;<\/p>\n","protected":false},"author":6,"featured_media":42327,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[943,4627,13107,15514,15896],"class_list":["post-2621","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-apache-spark-rdd-features","tag-features-of-rdd","tag-spark-rdd-features","tag-what-are-the-features-of-rdd-in-spark","tag-what-is-rdd"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>11 Shining Features of Spark RDD You Must Know - DataFlair<\/title>\n<meta name=\"description\" content=\"Learn what is Rdd in Apache Spark and why do we need RDD in Spark. features of Spark RDD and how Spark RDD overcomes the disadvantages of Hadoop MapReduce.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"11 Shining Features of Spark RDD You Must Know - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Learn what is Rdd in Apache Spark and why do we need RDD in Spark. features of Spark RDD and how Spark RDD overcomes the disadvantages of Hadoop MapReduce.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2017-05-22T11:19:40+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-11-16T08:26:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"11 Shining Features of Spark RDD You Must Know - DataFlair","description":"Learn what is Rdd in Apache Spark and why do we need RDD in Spark. features of Spark RDD and how Spark RDD overcomes the disadvantages of Hadoop MapReduce.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/","og_locale":"en_US","og_type":"article","og_title":"11 Shining Features of Spark RDD You Must Know - DataFlair","og_description":"Learn what is Rdd in Apache Spark and why do we need RDD in Spark. features of Spark RDD and how Spark RDD overcomes the disadvantages of Hadoop MapReduce.","og_url":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2017-05-22T11:19:40+00:00","article_modified_time":"2018-11-16T08:26:55+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"11 Shining Features of Spark RDD You Must Know","datePublished":"2017-05-22T11:19:40+00:00","dateModified":"2018-11-16T08:26:55+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/"},"wordCount":681,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg","keywords":["Apache spark RDD features","features of RDD","Spark RDD features","what are the features of RDD in spark","what is RDD"],"articleSection":["Apache Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/","url":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/","name":"11 Shining Features of Spark RDD You Must Know - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg","datePublished":"2017-05-22T11:19:40+00:00","dateModified":"2018-11-16T08:26:55+00:00","description":"Learn what is Rdd in Apache Spark and why do we need RDD in Spark. features of Spark RDD and how Spark RDD overcomes the disadvantages of Hadoop MapReduce.","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/features-of-rdd-1.jpg","width":1200,"height":628,"caption":"11 Shining Features of Spark RDD You Must Know"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/apache-spark-rdd-features\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/spark\/"},{"@type":"ListItem","position":3,"name":"11 Shining Features of Spark RDD You Must Know"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/2621","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=2621"}],"version-history":[{"count":5,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/2621\/revisions"}],"predecessor-version":[{"id":42328,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/2621\/revisions\/42328"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/42327"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=2621"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=2621"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=2621"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}