![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/05\/apachr-spark-ecosystem-components-1.jpg\")
<\/a>Apache Spark Ecosystem – Complete Spark Components Guide<\/p><\/div>\n
2. What is Apache Spark?<\/h2>\n
Apache Spark<\/strong> is general purpose cluster computing system. It provides high-level API in Java, Scala<\/a><\/strong>, Python, and R<\/strong><\/a>. Spark provide an optimized engine that supports general execution graph. It also has abundant high-level tools for structured data processing, machine learning, graph processing and streaming. The Spark can either run alone or on an existing cluster manager<\/a>. Follow this link to\u00a0Learn more about Apache Spark.<\/a><\/p>\n Apache Spark Ecosystem – Spark Core, Spark SQL, Spark Streaming, MLlib, GraphX, SparkR.<\/p><\/div>\n Following are 6 components in Apache Spark Ecosystem which empower to Apache Spark- Spark Core, Spark SQL, Spark Streaming, Spark MLlib, Spark GraphX, and SparkR.<\/p>\n Let us now learn about these Apache Spark ecosystem components in detail below:<\/p>\n All the functionalities being provided by Apache Spark are built on the top of Spark Core<\/strong>. It delivers speed by providing in-memory computation<\/strong> <\/a>capability. Thus Spark Core is the foundation of parallel and distributed processing of huge dataset. Spark Core<\/strong> is embedded with a special collection called RDD <\/strong>(resilient distributed dataset)<\/a>. RDD is among the abstractions of Spark. Spark RDD<\/em> handles partitioning data across all the nodes in a cluster. It holds them in the memory pool of the cluster as a single unit. There are two operations performed on RDDs: Transformation<\/em>\u00a0and\u00a0Action-<\/em><\/p>\n Refer these guides to learn more about\u00a0Spark RDD Transformations & Actions API<\/a>\u00a0and Different ways to create RDD in Spark<\/a>.<\/p>\n The\u00a0Spark SQL<\/a><\/strong> component is a distributed framework for structured<\/em> data processing. Using Spark SQL, Spark gets more information about the structure of data and the computation. With this information, Spark can perform extra optimization. It uses same execution engine while computing an output. It does not depend on API\/ language to express the computation. It is an add-on to core Spark API which allows scalable, high-throughput, fault-tolerant stream processing of live data streams. Spark can access data from sources like Kafka<\/strong>, Flume<\/a>,<\/strong> Kinesis<\/strong> or TCP socket.<\/strong>\u00a0It can operate using various algorithms. Finally, the data so received is given to file system, databases and live dashboards. Spark uses Micro-batching<\/em>\u00a0for real-time streaming. b. PROCESSING<\/strong> MLlib<\/strong> in Spark is a scalable Machine learning library that discusses both high-quality algorithm and high speed. GraphX<\/strong> in Spark is API for graphs and graph parallel execution. It is network graph analytics engine and data store. Clustering, classification, traversal, searching, and pathfinding<\/em> is also possible in graphs. Furthermore, GraphX extends Spark RDD by bringing in light a new Graph abstraction: a directed multigraph with properties attached to each vertex and edge. SparkR was Apache Spark 1.4 release. The key component of SparkR is SparkR DataFrame.\u00a0DataFrames are a fundamental data structure for data processing in R.<\/strong>\u00a0The concept of DataFrames extends to other languages with libraries like Pandas<\/em> etc.3. Introduction to Apache Spark Ecosystem Components<\/h2>\n
<\/a>3.1. Apache Spark Core<\/h3>\n
\nThe key features of Apache Spark Core are:<\/strong><\/p>\n\n
\n
3.2. Apache Spark SQL<\/h3>\n
\nSpark SQL works to access structured and semi-structured information. It also enables powerful, interactive, analytical application across both streaming and historical data. Spark SQL is Spark module for structured data processing. Thus, it acts as a distributed SQL query engine.
\nFeatures of Spark SQL include:<\/strong><\/p>\n\n
3.3. Apache Spark Streaming<\/h3>\n
\nMicro-batching is a technique that allows a process or task to treat a stream as a sequence of small batches of data. Hence Spark Streaming, groups the live data into small batches. It then delivers it to the batch system for processing. It also provides fault tolerance characteristics. Learn Spark Streaming in detail from this Apache Spark Streaming Tutorial<\/a>.
\nHow does Spark Streaming Works?<\/strong>
\nThere are 3 phases of Spark Streaming:
\na. GATHERING<\/strong>
\nThe Spark Streaming<\/em>\u00a0provides two categories of built-in\u00a0streaming sources:<\/em><\/p>\n\n
\nThe gathered data is processed using complex algorithms expressed with a high-level function. For example, map,\u00a0reduce,\u00a0join\u00a0and\u00a0window. Refer this guide to learn Spark Streaming transformations operations<\/a>.
\nc. DATA STORAGE<\/strong>
\nThe Processed data is pushed out to file systems, databases, and live dashboards.
\nSpark Streaming also provides high-level abstraction. It is known as discretized stream or DStream.
\nDStream in Spark<\/strong> signifies continuous stream of data. We can form DStream in two ways either from sources such as Kafka, Flume, and Kinesis or by high-level operations on other DStreams. Thus, DStream is internally a sequence of RDDs.<\/p>\n3.4. Apache Spark MLlib (Machine Learning Library)<\/h3>\n
\nThe motive behind MLlib creation is to make machine learning scalable and easy. It contains machine learning libraries that have an implementation of various machine learning algorithms. For example,\u00a0clustering, regression, classification and collaborative filtering.<\/em> Some lower level machine learning primitives like generic gradient descent optimization algorithm are also present in MLlib.
\nIn Spark Version 2.0 the RDD-based API in spark.mllib<\/em> package entered in maintenance mode. In this release, the DataFrame-based API is the primary Machine Learning API for Spark<\/a>. So, from now MLlib will not add any new feature to the RDD based API.
\nThe reason MLlib is switching to DataFrame-based API is that it is more user-friendly than RDD. Some of the benefits of using DataFrames are it includes Spark Data sources, SQL DataFrame queries Tungsten and Catalyst optimizations<\/em>, and uniform APIs across languages. MLlib also uses the linear algebra package Breeze<\/em>. Breeze is a collection of libraries for numerical computing and machine learning.<\/p>\n3.5. Apache Spark GraphX<\/h3>\n
\nGraphX also optimizes the way in which we can represent vertex and edges when they are primitive data types. To support graph computation it supports fundamental operators (e.g.,\u00a0subgraph,\u00a0join Vertices, and\u00a0aggregate Messages) as well as an optimized variant of the\u00a0Pregel\u00a0API<\/em>.<\/p>\n3.6. Apache SparkR<\/h3>\n
\nR also provides software facilities for data manipulation, calculation, and graphical display. Hence, the main idea behind SparkR was to explore different techniques to integrate the usability of R with the scalability of Spark. It is R package that gives light-weight frontend to use Apache Spark from R.
\nThere are various benefits of SparkR:<\/p>\n\n
4. Conclusion<\/h2>\n