Before starting you must have Spark installed. follow this guide to install Apache Spark.<\/a> Spark Shell Commands to Interact with Spark-Scala<\/p><\/div>\n Start the Spark Shell<\/strong><\/p>\n Apache Spark<\/a><\/strong> is shipped with an interactive shell\/scala prompt, as the spark is developed in Scala<\/a><\/strong>. Using the interactive shell we will run different commands (RDD<\/strong>\u00a0transformation\/action<\/strong><\/a>) to process the data. a)<\/strong> Read File from local filesystem and create an RDD<\/strong>.<\/p>\n [php]scala> val data = sc.textFile(“data.txt”)[\/php]<\/p>\n Note: sc is the object of SparkContext<\/em><\/p>\n Note: You need to create a file data.txt in Spark_Home directory<\/em><\/p>\n b) Create an RDD through Parallelized\u00a0Collection<\/strong><\/p>\n [php]scala> val no = Array(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)<\/p>\n scala> val noData = sc.parallelize(no)[\/php]<\/p>\n c) From Existing RDDs<\/strong><\/p>\n [php]scala> val newRDD = no.map(data => (data * 2))[\/php]<\/p>\n These are three methods to create the RDD<\/a><\/strong>. We can use the first method, when data is already available with the external systems like a local filesystem, HDFS<\/a><\/strong>, HBase<\/a><\/strong>, Cassandra, S3<\/strong>, etc. One can create an RDD by calling a textFile<\/strong> method of Spark Context<\/strong><\/a> with path \/ URL as the argument. The second approach can be used with the existing collections and the third one is a way to create new RDD from the existing one.<\/p>\n Count the number of items available in the RDD. To count the items we need to call an Action:<\/p>\n [php]scala> data.count()[\/php]<\/p>\n Filter the RDD and create new RDD of items which contain word \u201cDataFlair\u201d. To filter, we need to call transformation filter, which will return a new RDD with subset of items.<\/p>\n [php]scala> val DFData = data.filter(line => line.contains(“DataFlair”))[\/php]<\/p>\n For complex requirements, we can chain multiple operations together like filter transformation and count action together:<\/p>\n [php]scala> data.filter(line => line.contains(“DataFlair”)).count()[\/php]<\/p>\n To read the first item from the file,\u00a0you\u00a0can use the following command:<\/p>\n [php]scala> data.first()[\/php]<\/p>\n To read the first 5 item from the file, you can use the following command:<\/p>\n [php]scala> data.take(5)[\/php]<\/p>\n An RDD is made up of multiple partitions, to count the number of partitions:<\/p>\n [php]scala> data.partitions.length[\/php]<\/p>\n Note: Minimum no. of partitions in the RDD is 2 (by default). When we create RDD from HDFS<\/strong><\/a> file then a number of blocks will be equals to the number of partitions.<\/em><\/p>\n Caching is the optimization technique. Once we cache the RDD<\/strong><\/a> in the memory all future computation will work on the in-memory<\/strong><\/a> data, which saves disk seeks and improve the performance.<\/p>\n [php]scala> data.cache()[\/php]<\/p>\n RDD will not be cached once you run above operation, you can visit the web UI:<\/p>\n http:\/\/localhost:4040\/storage, it will be blank. RDDs are not explicitly cached once we run cache()<\/strong>, rather<\/p>\n RDDs will be cached once we run the Action, which actually needs data read from the disk.<\/p>\n Let\u2019s run some actions<\/strong><\/p>\n [php]scala> data.count()[\/php]<\/p>\n [php]scala> data.collect()[\/php]<\/p>\n Now as we have run some actions on the data file, which needs to be read from the disk to perform those operations. During this process, Spark will cache the file, so that for all future operations will get the data from the memory (no need for any disk interaction). Now if we run any transformation or action it will be done in-memory and will be much faster.<\/p>\n To read data from HDFS<\/a> file we can specify complete hdfs URL like hdfs:\/\/IP:PORT\/PATH<\/strong><\/p>\n [php]scala> var hFile = sc.textFile(“hdfs:\/\/localhost:9000\/inp”)[\/php]<\/p>\n One of the most popular operations of MapReduce<\/a><\/strong> \u2013 Wordcount<\/strong>. Count all the words available in the file.<\/p>\n [php]scala> val wc = hFile.flatMap(line => line.split(” “)).map(word => (word, 1)).reduceByKey(_ + _)[\/php]<\/p>\n Read the result on console<\/p>\n [php]scala> wc.take(5)[\/php]<\/p>\n It will display first 5 results<\/p>\n To write the data from HFDS<\/a>:<\/p>\n [php]scala> wc.saveAsTextFile(“hdfs:\/\/localhost:9000\/out”)[\/php]<\/p>\n In conclusion, we can say that using Spark Shell commands we can create RDD (In three ways), read from RDD, and partition RDD. We can even cache the file, read and write data from and to HDFS file and perform various operation on the data using the\u00a0Apache Spark Shell commands. See Also-<\/strong><\/p>\n 1. Objective The shell acts as an interface to access the operating system\u2019s service. Apache Spark is shipped with an interactive shell\/scala prompt with the interactive shell we can run different commands to process...<\/p>\n","protected":false},"author":6,"featured_media":42919,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[896,13021,13022,13041,13114,13142,13153],"class_list":["post-50","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-apache-spark","tag-spark","tag-spark-scala","tag-spark-commands","tag-spark-shell-commands","tag-spark-tutorial","tag-spark-shell"],"yoast_head":"\n
\nAfter Spark installation, You can create RDDs<\/strong><\/a> and perform various transformations and actions like filter(), partitions(), cache(), count(), collect, etc. In this blog, we will also discuss the integration of Spark with Hadoop<\/a>, how spark reads the data from HDFS and write to HDFS?.<\/p>\n![\"Spark](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/05\/SPARK-Shell-Commands-01.jpg\")
<\/a>2. Scala – Spark Shell Commands<\/h2>\n
\nThe command to start the Apache Spark Shell:
\n[php] $bin\/spark-shell [\/php]<\/p>\n2.1. Create a new RDD<\/h3>\n
2.2. Number of Items in the RDD<\/h3>\n
2.3. Filter Operation<\/h3>\n
2.4. Transformation and Action together<\/h3>\n
2.5. Read the first item from the RDD<\/h3>\n
2.6. Read the first 5 item from the RDD<\/h3>\n
2.7. RDD Partitions<\/h3>\n
2.8. Cache the file<\/h3>\n
2.9. Read Data from HDFS file<\/h3>\n
2.10. Spark WordCount Program in Scala<\/h3>\n
2.11 Write the data to HDFS file<\/h3>\n
3. Conclusion<\/h2>\n
\nNow you can create your first Spark Scala project<\/a>.<\/p>\n\n