Hadoop difference between Capacity scheduler and Fair scheduler

[DataFlair Team]

The default job scheduling strategy is FIFO. But there are multi-user scheduling is also available like Capacity and Fair scheduler. What is the difference between Capacity scheduler and Fair scheduler? What are the deciding factors in choosing a specific scheduler?

[DataFlair Team]

Fair Scheduler
Fair scheduling is a method of assigning resources to jobs such that all jobs get, on average, an equal share of resources over time. When there is a single job running, that job uses the entire cluster. When other jobs are submitted, tasks slots that free up are assigned to the new jobs, so that each job gets roughly the same amount of CPU time.

Capacity Scheduler
The Capacity Scheduler is designed to allow sharing a large cluster while giving eachorganization a minimum capacity guarantee. The central idea is that the available resources in the Hadoop clusterare partitioned among multiple organizations who collectively fund the cluster based on computing needs. There is an added benefit that an organization can access any excess capacity
no being used by others. This provides elasticity for the organizations in a cost-effective manner.

[DataFlair Team]

Fair Scheduler assigns equal amount of resource to all running jobs. When the job completes, free slot is assigned to new job with equal amount of resource. Here, the resource is shared between queues.

Capacity Scheduler on the other hand, it assigns resource based on the capacity required by the organisation. This is set up by queues for each organisation with specified amount of capacity. The queue is based on FIFO scheduling.

[DataFlair Team]

All the above answers are helpful, but if you want to learn about Hadoop Scheduling in detail, follow the link: Hadoop Schedulers Tutorial – Job Scheduling in Hadoop

[Author]

Posts