Apache Spark is an open-source cluster computing system that provides high-level API in Java, Scala, Python and R. It can access data from HDFS, Cassandra, HBase<\/strong><\/a>, Hive<\/strong><\/a>, Tachyon, and any Hadoop data source. And run in Standalone, YARN and Mesos cluster manager<\/strong><\/a>. What is Spark – Apache Spark Tutorial for Beginners<\/p><\/div>\n Apache Spark is a general-purpose & lightning fast cluster computing system. It provides a high-level API. For example, Java<\/strong><\/a>, Scala<\/a><\/strong>, Python<\/strong><\/a>, and R<\/a><\/strong>. Apache Spark is a tool for Running Spark Applications. Spark is 100 times faster than Bigdata<\/a><\/strong> Hadoop and 10 times faster than accessing data from disk. Apache Spark was introduced in 2009 in the UC Berkeley R&D Lab, later it becomes AMPLab. It was open sourced in 2010 under BSD license. In 2013 spark was donated to Apache Software Foundation where it became top-level Apache project in 2014.<\/p>\n After studying Apache Spark introduction lets discuss, why Spark come into existence? Hence in the industry, there is a big demand for a powerful engine that can process the data in real-time (streaming) as well as in batch mode. There is a need for an engine that can respond in sub-second and perform in-memory processing.<\/strong><\/a> Apache Spark puts the promise for faster data processing and easier development. How Spark achieves this? To answer this question, let\u2019s introduce the Apache Spark ecosystem which is the important topic in Apache Spark introduction that makes Spark fast and reliable. These components of Spark resolves the issues that cropped up while using Hadoop MapReduce.<\/p>\n What is Spark – Spark Ecosystem Components<\/p><\/div>\n Here we are going to discuss Spark Ecosystem Components one by one<\/p>\n It is the kernel of Spark, which provides an execution platform for all the Spark applications. It is a generalized platform\u00a0to support a wide array\u00a0of applications.<\/p>\n It enables users to run SQL\/HQL queries on the top of Spark. Using Apache Spark SQL, we can process structured as well as semi-structured data. It also provides an engine for Hive to run unmodified queries up to 100 times faster on existing deployments. Refer Spark SQL Tutorial<\/a><\/strong> for detailed study.<\/p>\n Apache Spark Streaming enables powerful interactive and data analytics<\/a><\/strong> application across live streaming data. The live streams are converted into micro-batches which are executed on top of spark core. Refer our Spark Streaming tutorial<\/a><\/strong> for detailed study of Apache Spark Streaming.<\/p>\n It is the scalable machine learning<\/strong><\/a> library which delivers both efficiencies as well as the high-quality algorithm. Apache Spark MLlib is one of the hottest choices for Data Scientist<\/a><\/strong> due to its capability of in-memory data processing, which improves the performance of iterative algorithm drastically.<\/p>\n Apache Spark GraphX<\/strong> <\/a>is the graph computation engine built on top of spark that enables to process graph data at scale.<\/p>\n It is R package<\/a><\/strong> that gives light-weight frontend to use Apache Spark from R. It allows data scientists to analyze large datasets and interactively run jobs on them from the R shell. The main idea behind SparkR<\/strong><\/a> was to explore different techniques to integrate the usability of R with the scalability of Spark. In this section of Apache Spark Tutorial, we will discuss the key abstraction of Spark knows as RDD. Learn How to create RDD in Spark in detail.<\/a><\/strong> Refer this link to learn RDD Transformations and Actions APIs with examples<\/a>.<\/strong><\/p>\n
\nWhat is Spark tutorial will cover Spark ecosystem components, Spark video tutorial, Spark abstraction \u2013 RDD, transformation, and action in Spark RDD. The objective of this introductory guide is to provide Spark Overview in detail, its history, Spark architecture, deployment model and RDD in Spark.<\/p>\n![\"What](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/09\/Spark-Tutorial-for-Beginners-01.jpg\")
<\/a>2. What is Spark?<\/h2>\n
\nSpark is written in Scala but provides rich APIs in Scala, Java, Python, and R.
\nIt can be integrated with Hadoop <\/a><\/strong>and can process existing Hadoop HDFS<\/a><\/strong> data. Follow this guide to learn How Spark is compatible with Hadoop?<\/a><\/strong>
\nIt is saying that the images are the worth of a thousand words. To keep this in mind we have also provided Spark video tutorial for more understanding of Apache Spark.<\/p>\n3. History Of Apache Spark<\/h2>\n
4. Why Spark?<\/h2>\n
\nIn the industry, there is a need for a general-purpose cluster computing tool as:<\/p>\n\n
\nApache Spark Definition says it is a powerful open-source engine that provides real-time stream processing, interactive processing, graph processing, in-memory processing as well as batch processing with very fast speed, ease of use and standard interface. This creates the difference between Hadoop vs Spark<\/a><\/strong> and also makes a huge comparison between Spark vs Storm<\/a><\/strong>.
\nIn this What is Spark tutorial, we discussed a definition of spark, history of spark and importance of spark. Now let’s move towards spark components.<\/p>\n5. Apache Spark Components<\/h2>\n
![\"Spark](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/07\/spark-ecosystem-components.jpg\")
<\/a>i. Spark Core<\/h3>\n
ii. Spark SQL<\/h3>\n
iii. Spark Streaming<\/h3>\n
iv. Spark MLlib<\/h3>\n
v. Spark GraphX<\/h3>\n
vi. SparkR<\/h3>\n
\nRefer Spark Ecosystem Guide<\/a><\/strong> for detailed study of Spark components.<\/p>\n6. Resilient Distributed Dataset – RDD<\/h2>\n
\nResilient Distributed Dataset<\/strong><\/a> (RDD) is the fundamental unit of data in Apache Spark, which is a distributed collection of elements across cluster nodes and can perform parallel operations. Spark RDDs are immutable but can generate new RDD by transforming existing RDD.
\nThere are three\u00a0ways to create RDDs in Spark:<\/p>\n\n
\nApache Spark RDDs support two types of operations:<\/p>\n\n
7. Spark Shell<\/h2>\n