Introduction to Apache Flink – A Quickstart Tutorial 4


1. Objective

In this tutorial we will discuss the introduction to Apache Flink, What is Flink, Why and where to use Flink. This Flink tutorial will answer the question why Apache Flink is called 4G of Big Data? The tutorial also briefs about Flink APIs and features.

Introduction to Apache Flink Tutorial Training

2. Video Tutorial

3. Introduction to Apache Flink

Apache Flink is an open source platform which is a streaming data flow engine that provides communication, fault-tolerance, and data-distribution for distributed computations over data streams. Flink is a top level project of Apache. Flink is a scalable data analytics framework that is fully compatible to Hadoop. Flink can execute both stream processing and batch processing easily.

Apache Flink was started under the project called The Stratosphere. In 2008 Volker Markl formed the idea for Stratosphere and attracted other co-principal Investigators from HU Berlin, TU Berlin, and the Hasso Plattner Institute Potsdam. They jointly worked on a vision and had already put the great efforts on open source deployment and systems building. Later on, several decisive steps had been so that the project can be popular in commercial, research and open source community. A commercial entity named this project as Stratosphere. After applying for Apache incubation in April 2014 Flink name was finalized. Flink is a german word which means swift or agile. To learn more about Flink introduction follow this guide.

4. Why Flink?

The key vision for Apache Flink is to overcome and reduces the complexity that has been faced by other distributed data-driven engines. It is achieved by integrating query optimization, concepts from database systems and efficient parallel in-memory and out-of-core algorithms, with the MapReduce framework. As Apache Flink is mainly based on the streaming model, Apache Flink iterates data by using streaming architecture. The concept of an iterative algorithm is tightly bounded into Flink query optimizer. Apache Flink’s pipelined architecture allows processing the streaming data faster with lower latency than micro-batch architectures (Spark).

To learn various Real world use cases of Apache Flink follow this Flink use case guide.

5. Apache Flink-API’s

Apache Flink provides APIs for creating several applications which use flink engine-

i. DataStream APIs

It is a regular program in Apache Flink that implements the transformation on data streams For example- filtering, aggregating, update state etc. Results are returned through sink which can be generated through write data on files or in a command line terminal.

ii. DataSet APIs

It is a regular program in Apache Flink that implements the transformation on data sets For example joining, grouping, mapping, filtering etc. This API is used for batch processing of data, the data which is already available in the repository.

iii. Table APIs

This API in FLink used for handling relational operations. It is an SQL-like expression language used for relational stream and batch processing which can also be integrated into Datastream APIs and Dataset APIs.

Get the Best Apache Flink Books to become Master of Flink.

6. Features Of Apache Flink

In this section of the tutorial, we will discuss various features of Apache Flink-

i. Low latency and High Performance

Apache Flink provides high performance and Low latency without any heavy configuration. Its pipelined architecture provides the high throughput rate. It processes the data at lightening fast speed, it is also called as 4G of Big Data.

ii. Fault Tolerance

The fault tolerance feature provided by Apache Flink is based on Chandy-Lamport distributed snapshots, this mechanism provides the strong consistency guarantees.

iii. Memory Management

The memory management in apache flink provides control on how much memory is used by certain runtime operations.

iv. Iterations

Apache Flink provides the dedicated support for iterative algorithms (machine learning, graph processing)

v. Integration

Apache Flink can be easily integrated with other open source data processing ecosystem. It can be integrated with Hadoop, streams data from Kafka, It can be run on YARN.

What Next:

Install Apache Flink on Ubuntu and run Wordcount program, to install and configure Flink follow this installation guide


Leave a comment

Your email address will not be published. Required fields are marked *

4 thoughts on “Introduction to Apache Flink – A Quickstart Tutorial

  • Nirmalya Sengupta

    I have been an Apache Flink enthusiast for last ~1 year, playing around with it in my spare time (I am written a few blogs too, to capture my understanding). While I have come to like it very much personally, I am yet to come across a proven Use-Case where it is *decidedly* preferable to Apache Spark, driven by the technical approach and improvements that it brings in (I would love to be made wiser though). Having said that, I think the way it pushes the idea of streaming and computing – even data which has been stored post-facto and hence, a sure candidate for batch processing – quite compelling. The variety and fluidity of Time-based window operators are quite riveting for a programmer like me. The recent addition of CEP-like SQL operators can prove to be the key to its larger adoption. In my view, a good competitive technology for Apache Flink is not Apache Spark but Apache Apex. My 2 cents.