1. Necessity of Big Data and Hadoop in the industry
2. Paradigm shift – why the industry is shifting to Big Data tools
3. Different dimensions of Big Data
4. Data explosion in the Big Data industry
5. Various implementations of Big Data
6. Different technologies to handle Big Data
7. Traditional systems and associated problems
8. Future of Big Data in the IT industry

1. Why Hadoop is at the heart of every Big Data solution
2. Introduction to the Big Data Hadoop framework
3. Hadoop architecture and design principles
4. Ingredients of Hadoop
5. Hadoop characteristics and data-flow
6. Components of the Hadoop ecosystem
7. Hadoop Flavors – Apache, Cloudera, Hortonworks, and more

Setup and Installation of single-node Hadoop cluster

1. Hadoop environment setup and pre-requisites
2. Hadoop Installation and configuration
3. Working with Hadoop in pseudo-distributed mode
4. Troubleshooting encountered problems

Setup and Installation of Hadoop multi-node cluster

1. Hadoop environment setup on the cloud (Amazon cloud)
2. Installation of Hadoop pre-requisites on all nodes
3. Configuration of masters and slaves on the cluster
4. Playing with Hadoop in distributed mode

1. What is HDFS (Hadoop Distributed File System)
2. HDFS daemons and architecture
3. HDFS data flow and storage mechanism
4. Hadoop HDFS characteristics and design principles
5. Responsibility of HDFS Master – NameNode
6. Storage mechanism of Hadoop meta-data
7. Work of HDFS Slaves – DataNodes
8. Data Blocks and distributed storage
9. Replication of blocks, reliability, and high availability
10. Rack-awareness, scalability, and other features
11. Different HDFS APIs and terminologies
12. Commissioning of nodes and addition of more nodes
13. Expanding clusters in real-time
14. Hadoop HDFS Web UI and HDFS explorer
15. HDFS best practices and hardware discussion

1. What is MapReduce, the processing layer of Hadoop
2. The need for a distributed processing framework
3. Issues before MapReduce and its evolution
4. List processing concepts
5. Components of MapReduce – Mapper and Reducer
6. MapReduce terminologies- keys, values, lists, and more
7. Hadoop MapReduce execution flow
8. Mapping and reducing data based on keys
9. MapReduce word-count example to understand the flow
10. Execution of Map and Reduce together
11. Controlling the flow of mappers and reducers
12. Optimization of MapReduce Jobs
13. Fault-tolerance and data locality
14. Working with map-only jobs
15. Introduction to Combiners in MapReduce
16. How MR jobs can be optimized using combiners

1. Anatomy of MapReduce
2. Hadoop MapReduce data types
3. Developing custom data types using Writable & WritableComparable
4. InputFormats in MapReduce
5. InputSplit as a unit of work
6. How Partitioners partition data
7. Customization of RecordReader
8. Moving data from mapper to reducer – shuffling & sorting
9. Distributed cache and job chaining
10. Different Hadoop case-studies to customize each component
11. Job scheduling in MapReduce

1. The need for an adhoc SQL based solution – Apache Hive
2. Introduction to and architecture of Hadoop Hive
3. Playing with the Hive shell and running HQL queries
4. Hive DDL and DML operations
5. Hive execution flow
6. Schema design and other Hive operations
7. Schema-on-Read vs Schema-on-Write in Hive
8. Meta-store management and the need for RDBMS
9. Limitations of the default meta-store
10. Using SerDe to handle different types of data
11. Optimization of performance using partitioning
12. Different Hive applications and use cases

1. The need for a high level query language – Apache Pig
2. How Pig complements Hadoop with a scripting language
3. What is Pig
4. Pig execution flow
5. Different Pig operations like filter and join
6. Compilation of Pig code into MapReduce
7. Comparison – Pig vs MapReduce

1. NoSQL databases and their need in the industry
2. Introduction to Apache HBase
3. Internals of the HBase architecture
4. The HBase Master and Slave Model
5. Column-oriented, 3-dimensional, schema-less datastores
6. Data modeling in Hadoop HBase
7. Storing multiple versions of data
8. Data high-availability and reliability
9. Comparison – HBase vs HDFS
10. Comparison – HBase vs RDBMS
11. Data access mechanisms
12. Work with HBase using the shell

1. The need for Apache Sqoop
2. Introduction and working of Sqoop
3. Importing data from RDBMS to HDFS
4. Exporting data to RDBMS from HDFS
5. Conversion of data import/export queries into MapReduce jobs

1. What is Apache Flume
2. Flume architecture and aggregation flow
3. Understanding Flume components like data Sources and Sinks
4. Flume channels to buffer events
5. Reliable & scalable data collection tools
6. Aggregating streams using Fan-in
7. Separating streams using Fan-out
8. Internals of the agent architecture
9. Production architecture of Flume
10. Collecting data from different sources to Hadoop HDFS
11. Multi-tier Flume flow for collection of volumes of data using AVRO

1. The need for and the evolution of YARN
2. YARN and its eco-system
3. YARN daemon architecture
4. Master of YARN – Resource Manager
5. Slave of YARN – Node Manager
6. Requesting resources from the application master
7. Dynamic slots (containers)
8. Application execution flow
9. MapReduce version 2 application over Yarn
10. Hadoop Federation and Namenode HA

1. Introducing Scala
2. Installation and configuration of Scala
3. Developing, debugging, and running basic Scala programs
4. Various Scala operations
5. Functions and procedures in Scala
6. Scala APIs for common operations
7. Loops and collections- Array, Map, List, Tuple
8. Pattern-matching and Regex
9. Eclipse with Scala plugin

1. Introduction to OOP – object oriented programming
2. Different oops concepts
3. Constructors, getters, setters, singletons; overloading and overriding
4. Nested Classes and visibility Rules
5. Functional Structures
6. Functional programming constructs
7. Call by Name, Call by Value

1. Problems with older Big Data solutions
2. Batch vs Real-time vs in-Memory processing
3. Limitations of MapReduce
4. Apache Storm introduction and its limitations
5. Need for Apache Spark

1. Introduction to Apache Spark
2. Architecture and design principles of Apache Spark
3. Spark features and characteristics
4. Apache Spark Ecosystem components and their insights

1. Spark environment setup
2. Installing and configuring prerequisites
3. Installation of Spark in local mode
4. Troubleshooting encountered problems

1. Spark installation and configuration in standalone mode
2. Installation and configuration of Spark in YARN mode
3. Installation and configuration of Spark on a real cluster
4. Best practices for Spark deployment

1. Working on the Spark shell
2. Executing Scala and Java statements in the shell
3. Understanding SparkContext and the driver
4. Reading data from local file-system and HDFS
5. Caching data in memory for further use
6. Distributed persistence
7. Spark streaming
8. Testing and troubleshooting

1. Introduction to Spark RDDs
2. How RDDs make Spark a feature rich framework
3. Transformations in Spark RDDs
4. Spark RDDs action and persistence
5. Lazy operations and fault tolerance in Spark
6. Loading data and how to create RDD in Spark
7. Persisting RDD in memory or disk
8. Pairing operations and key-value in Spark
9. Hadoop integration with Spark
10. Apache Spark practicals and workshops

1. The need for stream analytics
2. Comparison with Storm and S4
3. Real-time data processing using streaming
4. Fault tolerance and checkpointing in Spark
5. Stateful Stream Processing
6. DStream and window operations in Spark
7. Spark Stream execution flow
8. Connection to various source systems
9. Performance optimizations in Spark

1. The need for Spark machine learning
2. Introduction to Machine learning in Spark
3. Various Spark libraries
4. Algorithms for clustering, statistical analytics, classification etc.
5. Introduction to Spark GraphX
6. The need for different graph processing engine
7. Graph handling using Apache Spark

1. Introduction to Spark SQL
2. Apache Spark SQL Features and Data flow
3. Architecture and components of Spark SQL
4. Hive and Spark together
5. Data frames and loading data
6. Hive Queries through Spark
7. Various Spark DDL and DML operations
8. Performance tuning in Spark

Live Apache Spark & Hadoop project using Spark & Hadoop components to solve real-world Big Data problems in Hadoop & Spark.