Resilient Distributed Datasets (RDD)<\/strong><\/a> is the fundamental data structure of Spark. RDDs are immutable and fault tolerant in nature. These are distributed collections of objects. The datasets are divided into a logical partition, which is further computed on different nodes over the cluster. Thus, RDD is just the way of representing dataset distributed across multiple machines, which can be operated around in parallel. RDDs are called resilient because they have the ability to always re-compute an RDD. Let us revise Spark RDDs in depth here. Learn: RDD Persistence and Caching Mechanism in Apache Spark<\/a><\/strong> In the initial stage when we learn Spark, RDDs<\/a><\/strong> are generally created by parallelized collection i.e. by taking an existing collection in the program and passing it to SparkContext\u2019s<\/strong><\/a> parallelize() method. This method is used in the initial stage of learning Spark since it quickly creates our own RDDs in Spark shell and performs operations on them. This method is rarely used outside testing and prototyping because this method requires entire dataset on one machine. In Spark, the distributed dataset can be formed from any data source supported by Hadoop, including the local file system, HDFS<\/a><\/strong>, Cassandra, HBase <\/a><\/strong>etc. In this, the data is loaded from the external dataset. To create text file RDD, we can use SparkContext\u2019s textFile method. It takes URL of the file and read it as a collection of line. URL can be a local path on the machine or a hdfs:\/\/, s3n:\/\/, etc. It loads a CSV file and returns the result as a Dataset<Row>. It loads a JSON file (one object per line) and returns the result as a Dataset<Row> It loads text files and returns a Dataset of String. Transformation mutates one RDD into another RDD, thus transformation is the way to create an RDD from already existing RDD. This creates difference between Apache Spark and Hadoop MapReduce<\/strong>. <\/a>Transformation acts as a function that intakes an RDD and produces one. The input RDD does not get changed, because RDDs are immutable in nature but it produces one or more RDD by applying operations. Some of the operations applied on RDD are: filter, count, distinct, Map, FlatMap<\/a><\/strong> etc. Now as we have seen how to create RDDs in Apache Spark, let us learn RDD\u00a0transformations and Actions in Apache Spark<\/a><\/strong>\u00a0with the help of examples. If you like this blog or you have any query to create RDDs in Apache Spark, so let us know by leaving a comment in the comment box. <\/span><\/p>\n","protected":false},"excerpt":{"rendered":" 1. Objective In this Spark tutorial, we are going to understand different ways of how to create RDDs in Apache Spark. We will understand Spark RDDs and 3 ways of creating RDDs in Spark...<\/p>\n","protected":false},"author":6,"featured_media":73667,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[946,11340,11351,11352,11575,13106,13142],"class_list":["post-1830","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-apache-spark-rdds","tag-rdd","tag-rdds-in-apache-spark","tag-rdds-in-spark","tag-resilient-distributed-dataset-in-apache-spark","tag-spark-rdd-creation","tag-spark-tutorial"],"yoast_head":"\n![\"how](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/03\/ways-to-create-RDDs-in-spark-2.jpg\")
<\/a><\/p>\n2. How to Create RDDs in Apache Spark?<\/h2>\n
\nNow as we have already seen what is RDD in Spark, let us see how to create Spark RDDs.
\nThere are three ways to create an RDD in Spark.<\/p>\n\n
\nLet us learn these in details below:<\/p>\ni. Parallelized collection (parallelizing)<\/h3>\n
\nConsider the following example of sortByKey(). In this, the data to be sorted is taken through parallelized collection:
\n[php]val data=spark.sparkContext.parallelize(Seq((“maths”,52),(“english”,75),(“science”,82), (“computer”,65),(“maths”,85)))
\nval sorted = data.sortByKey()
\nsorted.foreach(println)[\/php]
\nThe key point to note in parallelized collection is the number of partition the dataset is cut into. Spark will run one task for each partition of cluster. We require two to four partitions for each CPU in cluster<\/a><\/strong>. Spark sets number of partition based on our cluster. But we can also manually set the number of partitions. This is achieved by passing number of partition as second parameter to parallelize .
\ne.g. sc.parallelize(data, 10), here we have manually given number of partition as 10.
\nConsider one more example, here we have used parallelized collection and manually given the number of partitions:
\n[php]val rdd1 = spark.sparkContext.parallelize(Array(“jan”,”feb”,”mar”,”april”,”may”,”jun”),3)
\nval result = rdd1.coalesce(2)
\nresult.foreach(println)[\/php]<\/p>\nii. External Datasets (Referencing a dataset)<\/h3>\n
\nThe point to jot down is that the path of the local file system and worker node should be the same. The file should be present at same destinations both in the local file system and worker node.\u00a0 We can copy the file to the worker\u00a0nodes or use a network mounted shared file system.
\nLearn:<\/strong> Spark Shell Commands to Interact with Spark-<\/strong>Scala<\/a>
\nDataFrameReader Interface is used to load a Dataset from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.read to access an instance of DataFrameReader.DataFrameReader supports many file formats-<\/p>\na.\u00a0<\/strong>csv (String path)<\/strong><\/h4>\n
\nExample:<\/strong>
\n[php]import org.apache.spark.sql.SparkSession
\ndef main(args: Array[String]):Unit = {
\nobject DataFormat {
\nval spark =\u00a0 SparkSession.builder.appName(“AvgAnsTime”).master(“local”).getOrCreate()
\nval dataRDD = spark.read.csv(“path\/of\/csv\/file”).rdd[\/php]
\nNote<\/strong> – Here .rdd method is used to convert Dataset<Row> to RDD<Row>.<\/p>\nb. json (String path)<\/h4>\n
\n[php]val dataRDD = spark.read.json(“path\/of\/json\/file”).rdd[\/php]<\/p>\nc. textFile (String path)<\/h4>\n
\n[php]val dataRDD = spark.read.textFile(“path\/of\/text\/file”).rdd[\/php]
\nLearn: RDD lineage in Spark: ToDebugString Method<\/a><\/strong><\/p>\niii. Creating RDD from existing RDD<\/h3>\n
\nExample:<\/strong>
\n[php]val words=spark.sparkContext.parallelize(Seq(“the”, “quick”, “brown”, “fox”, “jumps”, “over”, “the”, “lazy”, “dog”))
\nval wordPair = words.map(w => (w.charAt(0), w))
\nwordPair.foreach(println)[\/php]
\nNote<\/strong> – In above code RDD “wordPair” is created from existing RDD “word” using map() transformation which contains word and its starting character together.
\nRead: Limitations of Spark RDD<\/a><\/strong><\/p>\n4. Conclusion<\/h2>\n
\nReference:<\/strong>
\nhttp:\/\/spark.apache.org\/<\/a><\/strong>
\nSee Also-<\/strong><\/p>\n\n