So, let’s start exploring Apache Spark Interview Questions.<\/p>\n
![\"50](\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2017\/06\/50-frequently-asked-apacha-spark-interview-questions-2.jpg\")
<\/a>50 Frequently Asked Apache Spark Interview Questions<\/p><\/div>\n
2. Top 50 Apache Spark Interview Questions and Answers<\/h2>\n
Let’s proceed further with Apache Spark Interview Questions and Answer-<\/p>\n
1)\u00a0What is Apache Spark? What is the reason behind the evolution of this framework?<\/strong><\/p>\n Ans.\u00a0<\/strong>Spark<\/a><\/strong>\u00a0is an open source big data framework. It has an expressive APIs to allow big data professionals to efficiently execute streaming as well as the batch. Apache Spark provides faster and more general data processing platform engine. It basically designed for fast computation and developed at UC Berkeley in 2009. Spark is an Apache project which is also call as \u201clighting fast cluster computing\u201c. It distributes data in a file system across the cluster, and process that data in parallel. Spark covers a wide range of workloads like batch applications, iterative algorithms, interactive queries and streaming. It lets you write an application in Java, Python or Scala.<\/p>\n It was developed to overcome the limitations of\u00a0MapReduce<\/a><\/strong>\u00a0cluster computing paradigm. Spark keeps things in memory whereas map reduces keep shuffling things in and out of disk. It allows to cache data in memory which is beneficial in iterative algorithm those used in machine learning.<\/p>\n Spark is easier to develop as it knows how to operate on data. It supports SQL queries, streaming data as well as graph data processing. Spark doesn\u2019t need Hadoop to run, it can run on its own using other storages like Cassandra, S3 from which spark can read and write. In terms of speed spark run programs up to 100x faster in memory or 10x faster on disk than Map Reduce.<\/p>\n 2)\u00a0Explain the features of Apache Spark because of which it is superior to Apache MapReduce?<\/strong><\/p>\n Ans.\u00a0Hadoop<\/a><\/strong>\u00a0is designed for batch processing. Batch processing is very efficient in the processing of high volume data. Spark\u00a0also supports batch processing system as well as stream processing.<\/p>\n Spark streaming<\/a><\/strong>\u00a0processes data streams in micro-batches, Micro batches are an essentially collect and then process kind of 3) Why is Apache Spark faster than Apache Hadoop?<\/strong><\/p>\n Ans.<\/strong>\u00a0Apache Spark\u00a0is faster than\u00a0Apache Hadoop\u00a0due to below reasons:<\/p>\n 4) List down the languages supported by Apache Spark.<\/strong><\/p>\n Ans.\u00a0<\/strong>Apache Spark supports\u00a0Scala,\u00a0Python<\/strong>,\u00a0Java<\/strong>, and\u00a0R. 5) What are the components of Apache Spark Eco-system? <\/strong><\/p>\n Ans.\u00a0<\/strong>Spark Core\u00a0<\/strong><\/p>\n Spark Core is the base of Spark for parallel and distributed processing of huge datasets. It is in charge of all the essential I\/O functionalities, programming, and observance the roles on spark clusters. It is also responsible for task dispatching, and networking with different storage systems,\u00a0fault tolerance<\/a><\/strong>, and economical memory management. Core uses special collection referred to as\u00a0RDD (Resilient Distributed Datasets)<\/a><\/strong>.<\/p>\n SparkSQL is module\/component in Apache Spark that is employed to access structured and semi-structured information. It is a distributed framework that is tightly integrated with varied spark programming like\u00a0Scala<\/a><\/strong>, Python, and Java. Spark SQL supports relative process in Spark programs via RDD further as on external data source. It is used for manipulating and taking information in varied formats. The means through that {we can|we will|we square measure able to} act with Spark SQL are SQL\/HQL,\u00a0DataFrame<\/a><\/strong>\u00a0API, and Datasets API.<\/p>\n The main abstraction in SparkSQL is information sets that act on structured data. It translates ancient SQL and HiveQL queries into Spark jobs creating Spark accessible wide. It supports real-time data analytics, data streaming SQL.SparkSQL defines 3 varieties of function:<\/strong><\/p>\n Different type of APIs for accessing SparkSQL:<\/strong> SQL:\u00a0<\/strong>Executing SQL queries or\u00a0Hive<\/a><\/strong>\u00a0queries, the result is going to become in the variety of DataFrame.<\/p>\n Spark Streaming is a light-weight API that permits developers to perform execution and streaming of information application. Discretized Streams kind the bottom abstraction in Spark Streaming. It makes use of an endless stream of {input information|input file|computer file} to method data in the time period. It leverages the quick programming capability of Apache Spark core to perform streaming analytics by ingesting information in mini-batches. Information in Spark Streaming accepts from varied information sources and live streams like Twitter, Apache Kafka, IoT Sensors, Amazon response, Apache Flume, etc. in an event-driven, fault-tolerant, and type-safe applications.<\/p>\n For more details, please read\u00a0Apache Spark Eco-system.<\/a><\/strong><\/p>\n 6) Is it possible to run Apache Spark without Hadoop?<\/strong><\/p>\n Ans.\u00a0\u00a0<\/strong>Yes,\u00a0Apache Spark\u00a0can run without\u00a0Hadoop, standalone, or in the cloud. Spark doesn\u2019t need a Hadoop cluster to work. It can read and then process data from other file systems as well.\u00a0HDFS<\/a><\/strong>\u00a0is just one of the file systems that Spark supports.<\/p>\n Spark does not have any storage layer, so it relies on one of the distributed storage systems for distributed computing like HDFS, Cassandra etc.<\/p>\n However, there are a lot of advantages to running Spark on top of Hadoop (HDFS (for storage) +\u00a0YARN<\/a><\/strong>\u00a0(resource manager)), but it\u2019s not the mandatory requirement. It meant for distributed computing. In this case, the data distribute across the computers and Hadoop\u2019s distributed file system HDFS is used to store data that does not fit in memory.<\/p>\n One more reason for using Hadoop with Spark is they both are open source and both can integrate with each other rather easily as compared to other data storage system.<\/p>\n For more details, please refer:\u00a0Apache Spark Compatibility with Hadoop<\/a><\/strong><\/p>\n 7)\u00a0What is RDD in Apache Spark? How are they computed in Spark? what are the various ways in which it can create?<\/strong><\/p>\n Ans.\u00a0<\/strong>RDD in\u00a0Apache Spark\u00a0is the representation of a set of records, it is the immutable collection of objects with distributed computing. RDD is the large collection of data or an array of reference of partitioned objects. Each and every dataset in RDD is logically partitioned across many servers so that they can compute on different nodes of the cluster.\u00a0RDDs are fault tolerant<\/a><\/strong>\u00a0i.e. self-recovered \/ recomputed in the case of failure. The dataset could data load externally by the users which can be in the form of JSON file, CSV file, text file or database via JDBC with no specific data structure.<\/p>\n RDD is Lazily Evaluated<\/a><\/strong>\u00a0i.e. it is memorized or called when required or needed, which saves lots of time. RDD is a read-only, partitioned collection of data. RDDs can creates through deterministic operations or on stable storage or from other RDDs. It can also generates by parallelizing an existing collection in your application or referring a dataset in an external storage system. It is cacheable. As it operates on data over multiple jobs in computations such as logistic regression, k-means clustering, PageRank algorithms, which makes it reuse or share data among multiple jobs.<\/p>\n To learn more about the RDD follow:\u00a0RDD Tutorial<\/a><\/strong><\/p>\n 8)\u00a0What are the features of RDD, that makes RDD an important abstraction of Spark?<\/strong><\/p>\n Ans.\u00a0\u00a0<\/strong>RDD (Resilient Distributed Dataset) is a basic abstraction in\u00a0Apache Spark<\/a><\/strong>. Spark RDD is an immutable, partitioned collection of elements on the cluster which can operates in parallel.<\/p>\n Each RDD is characterized by five main properties :<\/strong><\/p>\n Below operations are lineage operations.<\/em><\/p>\n Below operations are used for optimization during execution.<\/em><\/p>\n Examples :<\/p>\n # HadoopRDD:\u00a0<\/strong>HadoopRDD provides core functionality for reading data stored in Hadoop (HDFS<\/a><\/strong>,\u00a0HBase<\/a><\/strong>, Amazon S3..) using the older\u00a0MapReduce\u00a0<\/a><\/strong>API (org.apache.hadoop.mapred)<\/p>\n Properties of HadoopRDD :<\/p>\n #FilteredRDD :\u00a0<\/strong>Properties of FilteredRDD:<\/p>\n Find features of RDD in\u00a0RDD Features in Spark<\/a><\/strong><\/p>\n 9) List out the ways of creating RDD in Apache Spark.<\/strong><\/p>\n Ans.\u00a0<\/strong>There are three ways to create\u00a0RDD<\/a><\/strong><\/p>\n (1) By Parallelizing collections in the driver program<\/p>\n (2) By loading an external dataset<\/p>\n (3) Creating RDD from already existing RDDs.<\/p>\n Create RDD By Parallelizing collections : OR\u00a0<\/strong><\/p>\n Create by loading an external Dataset<\/strong><\/p>\n In Spark, the distributed dataset can form from any data source supported by\u00a0Hadoop, including the local file system,\u00a0HDFS, Cassandra,\u00a0HBase<\/a><\/strong>\u00a0etc. In this, the data is loaded from the external dataset. To create text file RDD, we can use SparkContext\u2019s textFile method. It takes URL of the file and read it as a collection of line. URL can be a local path on the machine or a hdfs:\/\/, s3n:\/\/, etc. Use SparkSession.read to access an instance of DataFrameReader. DataFrameReader supports many file formats-<\/p>\n i) csv (String path)<\/strong><\/p>\n ii) json (String path)<\/strong><\/p>\n iii) textFile (String path)<\/strong><\/p>\n Creating RDD from existing RDD: For a detailed description on creating RDD read\u00a0How to create RDD in Apache Spark<\/a><\/strong><\/p>\n 10)\u00a0Explain Transformation in RDD. How is lazy evaluation helpful in reducing the complexity of the System?<\/strong><\/p>\n Ans.\u00a0<\/strong>Transformations are lazy evaluated operations on RDD that create one or many new\u00a0RDDs, e.g. map, filter, reduceByKey, join, cogroup, randomSplit. Transformations are functions which take an RDD as the input and produces one or many RDDs as output. They don\u2019t change the input RDD as RDDs are immutable and hence cannot change or modify but always produces new RDD by applying the computations operations on them. By applying transformations you incrementally build an RDD lineage with all the ancestor RDDs of the final RDD(s).<\/p>\n Transformations are lazy<\/a><\/strong>, i.e. are not executed immediately. Transformations can execute only when actions are called. After executing a transformation, the result RDD(s) will always be different from their ancestors RDD and can be smaller (e.g. filter, distinct, sample), bigger (e.g. flatMap, union, cartesian) or the same size (e.g. map) or it can vary in size.<\/p>\n RDD allows you to create dependencies b\/w RDDs. Dependencies are the steps for producing results in a program. Each RDD in lineage chain, string of dependencies has a function for operating its data and has a pointer dependency to its ancestor RDD. Spark will divide RDD dependencies into stages and tasks and then send those to workers for execution.<\/p>\n Follow this link to read more<\/a><\/strong><\/p>\n 11)\u00a0What are the types of Transformation in Spark RDD Operations?<\/strong><\/p>\n Ans.\u00a0<\/strong>There are two kinds of transformations:<\/p>\n Narrow transformations:<\/strong><\/p>\n Narrow transformations are the result of map, filter and in which data to transform id from a single partition only, i.e. it is self-sustained. Wide Transformations<\/strong><\/p>\n Wide transformations are the result of groupByKey and reduceByKey.\u00a0The data required to compute the records in a single partition may Wide transformations are also called shuffle transformations as they may or may not depend on a shuffle.\u00a0All of the tuples with the same key must end up in the same partition, processed by the same task.\u00a0To satisfy these operations,\u00a0Spark\u00a0<\/strong>must execute RDD shuffle, which transfers data across the cluster\u00a0and results in a new stage with a new set of partitions.<\/p>\n 12)\u00a0What is the reason behind Transformation being a lazy operation in Apache Spark RDD? How is it useful?<\/strong><\/p>\n Ans.\u00a0<\/strong>Whenever a\u00a0transformation operation<\/strong>\u00a0is performed in\u00a0Apache Spark<\/strong>, it is lazily evaluated. It won\u2019t execute until an action is performed. Apache Spark just adds an entry of the transformation operation to the\u00a0DAG (Directed Acyclic Graph)<\/a><\/strong>\u00a0of computation, which is a directed finite graph with no cycles. In this DAG, all the operations are classified into different stages, with no shuffling of data in a single stage.<\/p>\n By this way, Spark can optimize the execution by looking at the DAG at its entirety, and return the appropriate result to the driver program.<\/p>\n For example, consider a 1TB of a log file in HDFS containing errors, warnings, and other information. Below are the operations being performed in the driver program:<\/p>\n
\nHadoop MapReduce\u00a0is batch-oriented processing tool, it takes large dataset in the input, processes it and produces a result.
\nHadoop MapReduce adopted the batch-oriented model. Batch is essentially processing data at rest, taking a large amount of data\u00a0at once and producing output. MapReduce process is slower than spark because due to produce a lot of intermediary data.<\/p>\n
\ncomputational model. Spark processes faster than map reduces because it caches input data in memory by\u00a0RDD<\/a><\/strong>.<\/p>\n\n
\nApache Spark is written in Scala. Many people use Scala for the purpose of development. But it also has API in Java, Python, and R.<\/p>\n\n
\n
\n<\/strong><\/p>\n\n
\nDataFrameAPI is offered in Scala, Java, and Python.<\/li>\n\n
\n
\nMLlib in Spark stands for machine learning (ML) library. Its goal is to form sensible machine learning effective, ascendible and straightforward. It consists of some learning algorithms and utilities, as well as classification, regression, clustering, collaborative filtering, spatial property reduction, further as lower-level improvement primitives and higher-level pipeline genus Apis.<\/li>\n
\nGraphX, a distributed graph process framework on prime of Apache Spark. because it predicate on RDDs, that square measure immutable, graphs square measure immutable and so GraphX is unsuitable for graphs that require being updated, in addition to in an exceedingly transactional manner sort of a graph info.<\/li>\n<\/ul>\n3. Apache Spark Interview Questions For Beginners<\/h2>\n
\n
\n
\n
\n<\/em><\/li>\n<\/ol>\n\n
\n<\/em><\/li>\n<\/ol>\n
\n<\/strong>
\nParallelized collections are created by calling\u00a0parallelize<\/strong>() method on an existing collection in driver program.<\/p>\nval rdd1 = Array(1,2,3,4,5)\r\nval rdd2 = sc.parallelize(rdd1)<\/pre>\n
val myList = sc.parallelize(List(1 to 1000), 5) where 5 is the number of partitions\r\n [If we do not specify then default partition is 1<\/pre>\n
import org.apache.spark.sql.SparkSession\r\ndef main(args: Array[String]):Unit = {\r\nobject DataFormat {\r\nval spark = SparkSession.builder.appName(\"AvgAnsTime\").master(\"local\").getOrCreate()\r\nval dataRDD = spark.read.csv(\"path\/of\/csv\/file\").rdd<\/pre>\nval dataRDD = spark.read.json(\"path\/of\/json\/file\").rdd<\/pre>\n
val dataRDD = spark.read.textFile(\"path\/of\/text\/file\").rdd<\/pre>\n
\n<\/strong>
\nTransformation<\/a><\/strong>\u00a0mutates one RDD into another RDD, thus transformation is the way to create an RDD from already existing RDD.<\/p>\nval words=spark.sparkContext.parallelize(Seq(\"the\", \"quick\", \"brown\", \"fox\", \"jumps\", \"over\", \"the\", \"lazy\",\r\n \"dog\"))\r\nval wordPair = words.map(w => (w.charAt(0), w))\r\nwordPair.foreach(println)<\/pre>\n
\nAn output RDD has partitions with records that originate from a\u00a0single partition in the parent\u00a0RDD<\/strong>.<\/p>\n
\nreside in many partitions of the parent RDD.<\/p>\n\n