Explain various cluster manager in Apache Spark?

[DataFlair Team]

Explain different Apache Spark Cluster Manager.
What are the types of cluster manager used in Apache Spark?

[DataFlair Team]

Apache Spark uses three types of Cluster Manager:

<li style=”list-style-type: none”>

• Standalon Cluster Manager
• Apache Mesos
• Hadoop YARN

Standalone Cluster: The cluster consists of master and number of worker node. In this mode, the allocation of resources is based on a number of cores. An application grabs all the cores in the cluster.

Apache Mesos: By dynamic resource sharing and isolation Apache Mesos shares the workload in distributed environment. It joins the existing resource of the machine/node in the cluster. It acts as a resource management platform for Hadoop and Bigdata cluster. In this various physical resources are joined in single virtual resources. As a result, it is opposite of virtualization.

Hadoop YARN: YARN stands for Yet another Resource Negotiator. It is a combination of resource manager and node manager which can run on both Linux and Windows. It is also known bt Mapreduce 2.0. It lets different data processing engine like graph processing, stream processing to run and process data stored in HDFS.

For more information on Apache Spark Cluster Manager read Cluster Manager in Spark.

[DataFlair Team]

Spark can have 3 types of cluster managers

1. Standalone scheduler – this is the default cluster manager that comes along with spark in the distributed mode and manages resources on the executor nodes.

2. Hadoop YARN (Yet another resource negotiator) – It has a Resource Manager (scheduler and Applications Manager) and Node manager. the applications are assigned to queues and the resources are shared between queues.

3. Apache Mesos – It has Master and Slave processes. The Master makes offer of the available resources to the applications which decide whether to accept it
or not.

[DataFlair Team]

Apache Spark is an engine for large data processing can be run in distributed mode on a cluster. Spark applications are run as independent sets of processes on a cluster, all coordinated by a central coordinator. This central coordinator can connect with three different cluster managers, Spark’s Standalone, Apache Mesos, and Hadoop YARN.
When running an application in distributed mode on a cluster, Spark uses a master/slave architecture and the central coordinator, also called the driver program

This driver process is responsible for converting a user application into smaller execution units called tasks. These tasks are then executed by executors which are worker processes that run the individual tasks.

Spark Standalone

The Spark Standalone cluster manager is a simple cluster manager available as part of the Spark distribution. It has HA for the master, is resilient to worker failures, has capabilities for managing resources per application, and can run alongside an existing Hadoop deployment and access HDFS (Hadoop Distributed File System) data. The distribution includes scripts to make it easy to deploy either locally or in the cloud on Amazon EC2. It can run on Linux, Windows, or Mac OSX.

Apache Mesos

Mesos handles the workload in a distributed environment by dynamic resource sharing and isolation. It is healthful for deployment and management of applications in large-scale cluster environments Mesos many physical resources are club into a single virtual resource three components of Apache Mesos is Mesos masters, Mesos slave, Frameworks.

Mesos Master is an instance of the cluster. A cluster has many Mesos masters that provide fault tolerance. The slave is Mesos instance that offers resources to the cluster. Mesos Framework allows applications to request the resources from the cluster

YARN>/strong>
YARN data computation framework is a combination of the ResourceManager, the Nodemanager.

Resource Manager-manages resources among all the applications in the system. The Resource Manager has scheduler and Application Manager.

The Scheduler allocates resource to the various running application. It is pure Scheduler, performs monitoring or tracking of status for the application.

The Application Manager manages applications across all the nodes.

Yarn Node Manager contains Application Master and container. A container is a place where a unit of work happens. Application Master is a framework specific library. It aims to negotiate resources from the Resource Manager. It continues with Node Manager(s) to execute and watch the tasks.

[Author]

Posts