{"id":8882,"date":"2018-02-21T12:21:37","date_gmt":"2018-02-21T12:21:37","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=8882"},"modified":"2021-05-09T13:08:28","modified_gmt":"2021-05-09T07:38:28","slug":"flume-data-flow","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/","title":{"rendered":"Flume Data Flow – Types & Failure Handling in Apache Flume"},"content":{"rendered":"

If you want to know how data flows in Apache Flume then you have landed at the right place. This article explains the complete data flow mechanism in Apache Flume.<\/p>\n

In this article, you will explore different data flow like multi-hop flow, fan-in flow, and fan-out flow. The article also covers failure handling to understand the topic in detail.<\/p>\n

What is Apache Flume?<\/h2>\n

Apache Flume is an open-source distributed system for transferring streaming data from various applications to the central repository like HDFS, or HBase. It is highly reliable, extensible, robust, and fault-tolerant.<\/p>\n

Apache Flume is useful for delivering log data from different sources to Hadoop HDFS. Its components are:<\/p>\n

1. Flume Event<\/h3>\n

Flume event is the independent unit of data to be transferred from external sources to the central repository.<\/p>\n

2. Flume Agent<\/h3>\n

It is a JVM process that consists of components (source, channels, and sink) through which the data flows from the external source to the next destination.<\/p>\n

The Flume agent consists of a source, sink, and channel.<\/p>\n

a. Source<\/h4>\n

It is the component of the flume agent which receives events from the external source like web servers.<\/p>\n

b. Channels<\/h4>\n

The channel is a passive component that stores the event transferred by source until it is consumed by the sink. It is a bridge between the source and the sink.<\/p>\n

c. Sink<\/h4>\n

The sink is the component of the flume agent that consumes data from the flume channel and pushes it to the central repository or forward it to the source of the next flume agent.<\/p>\n

Introduction to Apache Flume Data Flow<\/h2>\n

 <\/p>\n

\"Apache<\/a><\/p>\n

Apache Flume collects data from data generators like web servers and sends them to central repositories like HDFS.<\/p>\n

The Flume collects data from data generators through Flume agents. The web servers have Flume agents running on them.<\/p>\n

There can be an intermediate node in Flume data flow which aggregates data from Flume agents and passes them to the central repository. There can be multiple in Flume flow.<\/p>\n

Agents pass the data to next agent or to the final destination ie external sink like HDFS.<\/p>\n

Types of Data Flow in Flume<\/h2>\n

1. Multi-hop flow<\/h3>\n

 <\/p>\n

\"Multi-hop<\/a><\/p>\n

In Flume flow, there can be more than one agent. The event (data) may travel through multiple agents before reaching the final destination. This is Multi-hop flow.<\/p>\n

2. Fan-in flow<\/h3>\n

\"Fan-in<\/a><\/p>\n

In Flume, data from multiple sources can be transferred through one channel. This is known as Fan-in flow.<\/p>\n

3. Fan-out flow<\/h3>\n

 <\/p>\n

\"Fan-out<\/a><\/p>\n

In fan-out flow, data flows from one source to multiple channels. There are two modes of fan-out flow. They are replicating and multiplexing.<\/p>\n

a. Replicating<\/h4>\n

In the Replicating mode, the event (data) is replicated to all the configured channels.<\/p>\n

b. Multiplexing<\/h4>\n

In multiplexing mode, event (data) is sent to selected channels based on the information in the event header.<\/p>\n

Flume Failure Handling<\/h2>\n

Apache Flume has an in-built feature of Failure Handling. For each event in Flume, two transactions take place. One transaction occurs at the sender side and another at the receiver side. The sender sends the event to the receiver.<\/p>\n

The receiver after receiving the event commits its own transaction and sends back a received signal to the sender. The sender after receiving the signal from the receiver commits its transaction. The sender will not commit its transaction until it receives a \u201creceived signal\u201d from the receiver.<\/p>\n

Summary<\/h2>\n

Apache Flume is a distributed system for collecting, aggregating, and transporting events from external sources to HDFS or HBase. The Flume agent runs on the webserver and collects data from web servers.<\/p>\n

There can be multiple agents in the flume data flow, intermediate agents aggregate data from previous agents and push them on the central repository. Flume data flow is of three types that are multi-hop, fan-in, and fan-out flow.<\/p>\n

In multi-hop, data travels through multiple agents before reaching the destination. In fan-in flow, data from multiple sources get transferred through one channel, and in fan-out flow, data from one source is transferred to multiple channels.<\/p>\n","protected":false},"excerpt":{"rendered":"

If you want to know how data flows in Apache Flume then you have landed at the right place. This article explains the complete data flow mechanism in Apache Flume. In this article, you...<\/p>\n","protected":false},"author":6,"featured_media":78380,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[3310,3311,4537,4539,4540,4812,4813,8921],"class_list":["post-8882","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flume","tag-data-flow-in-apache-flume","tag-data-flow-in-flume","tag-failure-handling","tag-fan-in-flow","tag-fan-out-flow","tag-flume-data-flow","tag-flume-data-flow-diagram","tag-multi-hop-flow"],"yoast_head":"\nFlume Data Flow - Types & Failure Handling in Apache Flume - DataFlair<\/title>\n<meta name=\"description\" content=\"What is flume Data Flow-Types of data flow in Flume, Multi-hop Flow, Fan-out Flow, Fan-in Flow, Failure Handling with flume Data Flow model\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/flume-data-flow\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Flume Data Flow - Types & Failure Handling in Apache Flume - DataFlair\" \/>\n<meta property=\"og:description\" content=\"What is flume Data Flow-Types of data flow in Flume, Multi-hop Flow, Fan-out Flow, Fan-in Flow, Failure Handling with flume Data Flow model\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/flume-data-flow\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-21T12:21:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:38:28+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/data-flow-in-apache-flume.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Flume Data Flow - Types & Failure Handling in Apache Flume - DataFlair","description":"What is flume Data Flow-Types of data flow in Flume, Multi-hop Flow, Fan-out Flow, Fan-in Flow, Failure Handling with flume Data Flow model","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/","og_locale":"en_US","og_type":"article","og_title":"Flume Data Flow - Types & Failure Handling in Apache Flume - DataFlair","og_description":"What is flume Data Flow-Types of data flow in Flume, Multi-hop Flow, Fan-out Flow, Fan-in Flow, Failure Handling with flume Data Flow model","og_url":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-02-21T12:21:37+00:00","article_modified_time":"2021-05-09T07:38:28+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/data-flow-in-apache-flume.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Flume Data Flow – Types & Failure Handling in Apache Flume","datePublished":"2018-02-21T12:21:37+00:00","dateModified":"2021-05-09T07:38:28+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/"},"wordCount":672,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/data-flow-in-apache-flume.jpg","keywords":["Data Flow in Apache Flume","data flow in Flume","Failure Handling","Fan-in Flow","Fan-out Flow","Flume Data Flow","Flume Data Flow diagram","Multi-hop Flow"],"articleSection":["Flume Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/flume-data-flow\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/","url":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/","name":"Flume Data Flow - Types & Failure Handling in Apache Flume - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/data-flow-in-apache-flume.jpg","datePublished":"2018-02-21T12:21:37+00:00","dateModified":"2021-05-09T07:38:28+00:00","description":"What is flume Data Flow-Types of data flow in Flume, Multi-hop Flow, Fan-out Flow, Fan-in Flow, Failure Handling with flume Data Flow model","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/flume-data-flow\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/data-flow-in-apache-flume.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/data-flow-in-apache-flume.jpg","width":802,"height":420,"caption":"data flow in apache flume"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/flume-data-flow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Flume Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flume\/"},{"@type":"ListItem","position":3,"name":"Flume Data Flow – Types & Failure Handling in Apache Flume"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8882","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=8882"}],"version-history":[{"count":3,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8882\/revisions"}],"predecessor-version":[{"id":92684,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8882\/revisions\/92684"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/78380"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=8882"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=8882"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=8882"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}