This blog of Spark Notes, answers to what is Apache Spark, what is the need of Spark, what are the Spark components and what are their roles, how Apache Spark is used by data scientist<\/a><\/strong>? We will also discuss various features and limitations of Spark<\/a><\/strong>\u00a0in this Spark Notes.<\/p>\n So, let’s start Spark Notes tutorial.<\/p>\n Spark Notes for Beginners & Experienced<\/p><\/div>\n Apache Spark<\/strong> is an open source, wide range data processing engine with revealing development API\u2019s, that qualify data workers to accomplish streaming in spark, machine learning or SQL workloads which demand repeated access to data sets. It is designed in such a way that it can perform batch processing<\/strong><\/a> (processing of the previously collected job in a single batch) and stream processing (deal with streaming data). It is a general purpose, cluster computing platform. MapReduce is a programming paradigm\u00a0that allows scalability across thousands of server in Hadoop cluster. Spark is highly accessible and offers simple APIs in Python, Java, Scala, and R<\/a><\/strong>.<\/p>\n There is a common belief that Apache Spark is an extension of Hadoop, which is not true. Spark is independent of Hadoop because it has its own cluster management system.<\/strong> It uses Hadoop for storage purpose only.<\/p>\n The key feature of Spark<\/a><\/strong> is that it has in-memory cluster computation capability. That increases the processing speed of an application. The in-memory computing<\/strong><\/a> means using a type of middleware software that allows one to store data in RAM, across a cluster of computers, and process it in parallel.<\/p>\n As we know that the images are the worth of a thousand words. To keep this in mind we have also provided Spark video tutorial<\/strong><\/a>\u00a0for more understanding of Apache Spark.<\/p>\n Apache Spark is a subproject of Hadoop developed in the year 2009 by Matei Zaharia<\/strong> in UC Berkeley\u2019s AMPLab. The first users of Spark were the group inside UC Berkeley including machine learning researchers, which used Spark to monitor and predict traffic congestion in the San Francisco Bay Area. Spark has open sourced in the year 2010 under BSD license. Spark became a project of Apache Software Foundation in the year 2013 and is now the biggest project of Apache foundation.<\/p>\n This section of Spark notes, we cover the needs\u00a0of Spark. Apache Spark was developed to overcome the limitations of Hadoop<\/a><\/strong> MapReduce cluster computing paradigm. Some of the drawbacks of Hadoop MapReduce are:<\/p>\n Due to\u00a0these weaknesses of Hadoop MapReduce<\/strong><\/a> Apache Spark come into the picture. Features of Spark listed below explains that how Spark overcomes the limitations of Hadoop MapReduce.<\/p>\n In this section of Spark notes, we will discuss various Spark features, which takes Spark to the limelight.<\/p>\n The key feature required for Bigdata evaluation is speed. With Apache Spark, we get swift processing speed of up to 100x faster in memory and 10x faster than Hadoop even when running on disk. It is achieved by reducing the number of read-write to disk.<\/p>\n Because of 80 high-level operators present in Apache Spark, it makes it possible to develop parallel applications. Scala<\/strong><\/a> being defaulted language for Spark. We can also work with Java, Python<\/strong><\/a>, R. Hence, it provides dynamicity and overcomes the limitation of Hadoop MapReduce that it can build applications only in java.<\/p>\n Disk seeks is becoming\u00a0very costly with increasing volumes of data. Reading terabytes to petabytes of data from disk and writing back to disk, again and again, is not acceptable. Hence in-memory processing in Spark works as a\u00a0boon to increasing the processing speed. Spark keeps data in memory for faster access. Keeping data in servers’ RAM as it makes accessing stored data quickly \/faster. Apache Spark provides the provision of code reusability for batch processing, join streams against historical data, or run adhoc queries on stream state.<\/p>\n Spark\u00a0RDD<\/a>\u00a0<\/strong>(Resilient Distributed Dataset), abstraction are designed to seamlessly handle failures of any worker nodes in the cluster. Thus, the loss of data and information is negligible. Follow this guide to learn how fault tolerance is achieved in Spark?<\/a><\/strong><\/p>\n Spark Streaming can handle real-time stream processing along with the integration of other frameworks which concludes that sparks streaming ability are easy, fault tolerance and Integrated. Refer Spark Streaming Tutorial<\/a><\/strong> for detailed study.<\/p>\n Spark comes with tools for interactive\/declarative queries, streaming data, machine learning which is an addition to the simple map and reduces, so that user can combine all this into a single workflow.<\/p>\n Apache Spark is compatible with both versions of Hadoop ecosystem. Be it YARN<\/a><\/strong> (Yet Another Resource Negotiator) or SIMR (Spark in MapReduce). It can read anything existing Hadoop data that\u2019s what makes it suitable for migration of pure Hadoop applications. It can run independently too. Follow this link if you would like to know how Spark is compatible with Hadoop?<\/a><\/strong><\/p>\n Lazy Evaluation\u00a0is an Another outstanding feature of Apache Spark is called by need or memorization. It waits for instructions before providing a final result which saves time. Learn Spark Lazy Evaluation<\/a><\/strong> feature in detail.<\/p>\n A wide set of developers from over 50 companies build Apache Spark. It has active mailing state and JIRA for issue tracking.<\/p>\n Till now in Spark notes, we have studied what is Spark and way spark is needed? Now, we will cover the Spark Ecosystem. Following are the ecosystem components of Apache Spark.<\/p>\n Apache Spark Ecosystem Components<\/p><\/div>\n It is the foundation of Spark. Spark core is also shelter to API that contains the backbone of Spark i.e. RDDs (resilient distributed datasets). The basic functionality of Spark is present in Spark Core like memory management, fault recovery, interaction with the storage system. It is in charge of essential I\/O functionalities like:<\/p>\n Spark SQL component is a distributed framework. It is a Spark package that allows working with structured and semi-structured data. Spark SQL allows querying in SQL<\/strong> and HQL <\/strong>too which provides declarative query with optimized storage running in parallel. Refer our Spark SQL Tutorial<\/a><\/strong> for the detailed study.<\/p>\n Spark Streaming enables processing of the large stream of data. It makes easy for the programmer to move between an application that manipulates data stored in memory, on disk and arriving in real time.\u00a0Micro-batching is used for real time streaming. Firstly the small batches have formed from the live data and then delivered to the Spark batch processing System for processing. In short, it provides data abstraction known as DStream. It also provides fault tolerance characteristics. Refer\u00a0Spark Streaming Tutorial<\/a>\u00a0<\/strong>for detailed study.<\/p>\n Spark MLlib (Machine learning library) provides algorithms like machine learning<\/strong><\/a> as well as statistical, which further includes classification, regression, linear algebra, clustering, collaborative filtering\u00a0as well as supporting functions such as model evaluation and data import.<\/p>\n MLlib is a scalable learning library that discusses high-quality algorithms and high speed. It also contains some lower level primitives.<\/p>\n GraphX<\/strong> <\/a>is an API for graphs and graph parallel execution. It is a network graph analytics engine. GraphX is a library that performs graph-parallel computation and manipulates graph. It has various Spark RDD API so it can help to create directed graphs with arbitrary properties linked to its vertex and edges.<\/p>\n Spark GraphX also provides various operator and algorithms to manipulate graph. Clustering, classification, traversal, searching, and pathfinding is possible in GraphX.<\/p>\n In Apache Spark 1.4 SparkR has released.\u00a0It allows data scientists to analyze large datasets and interactively run jobs on them from the R<\/strong><\/a> shell. The main idea behind SparkR was to explore different techniques to integrate the usability of R with the scalability of Spark. It is R package that gives light-weight frontend to use Apache Spark from R.<\/p>\n There are some limitations of Apache Spark<\/p>\n To install Apache Spark follow this Spark installation on Ubuntu guide<\/a><\/strong>. Once you have completed the installation, you can play with Spark shell and run various commands, please follow this Quickstart guide\u00a0to playing with Spark<\/a>.<\/strong><\/p>\n So, this was all in Spark Notes. Hope you like our explanation.<\/p>\n In conclusion, Apache Spark\u00a0notes defines that Apache spark has designed in such a way that it can scale up from one to thousands of computer node. Moreover, the simplicity of APIs in spark and its processing speed makes Spark a popular framework among data scientists. So, for the\u00a0data exploration, Spark uses SQL shell. MLlib supports data analysis and Machine Learning. It lets the data scientist handle the problem with large data size.<\/p>\n Hence Spark provides a simple way to parallelize the applications across Clusters and hides the complexity of distributed systems programming, network developer, and fault tolerance.<\/p>\n Still, if you felt that I have missed any point in this Spark Notes, so please let me know by leaving a comment.![\"Spark](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/06\/Spark-Notes-for-Beginners-Experienced-01.jpg\")
<\/a>2. Spark Notes – What is Spark?<\/h2>\n
\nSpark is designed in such a way that it integrates with all the Big data<\/a><\/strong> tools. For example, Spark can access any Hadoop data source and can run on Hadoop clusters<\/a><\/strong>. Spark extends Hadoop\u00a0MapReduce to next level which includes iterative queries and stream processing.<\/p>\n3. Spark Notes – Spark\u00a0History<\/h2>\n
4. Why Spark?<\/h2>\n
\n
5. Features of\u00a0Apache Spark<\/h2>\n
a. Swift Processing<\/h3>\n
b. Dynamic<\/h3>\n
c. In – Memory Processing<\/h3>\n
\nSpark owns advanced DAG <\/a><\/strong>execution engine which facilitates in-memory computation and acyclic data flow resulting high speed. You can find out the more detail about Spark in-memory computation<\/a><\/strong>.<\/p>\nd. Reusability<\/h3>\n
e. Fault Tolerance<\/h3>\n
f. Real-Time Stream Processing<\/h3>\n
g. Pillar to Sophisticated Analytics<\/h3>\n
h. Compatibility with Hadoop & existing Hadoop Data<\/h3>\n
i. Lazy Evaluation<\/h3>\n
j. Active, Progressive and Expanding Community<\/h3>\n
6. Apache Spark Ecosystem<\/h2>\n
![\"Apache](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/Apache-Spark-Ecosystem-01.jpg\")
<\/a>a. Spark Core<\/h3>\n
\n
b. Spark SQL<\/h3>\n
\nIt enables powerful interactive and analytical application across both streaming and historical data. SparkSQL allows accessing data from multiple sources like Hive<\/a><\/strong> table, Parquet, and JSON. It also lets you intermix SQL query with the programmatic data manipulations supported by RDDs in Python, Java, and Scala, all within a single application, thus combining SQL with complex analytics.<\/p>\nc. Spark Streaming<\/h3>\n
d. MLlib<\/h3>\n
e. GraphX<\/h3>\n
f. SparkR<\/h3>\n
7. Limitations of Apache Spark<\/h2>\n
\n
8. Spark Notes – Install Spark<\/h2>\n
9. Conclusion – Spark Notes<\/h2>\n
\nSee Also<\/strong><\/p>\n