Flume Channel Selectors – Apache Flume

1. Objective

Basically, we use Channel Selectors to determine which channel we should select to transfer the data in case of multiple channels. However, there are several types of Flume Channel Selectors in Apache Flume and examples of Flume channel Selectors. So, in this article, we will learn all types of Flume channel selectors. Also, we will cover several examples to understand it well.

What is Flume Channel Selectors

What is Flume Channel Selectors

2. Introduction to Flume Channel Selectors

Basically, we use Flume Channel Selectors to determine which channel we should select to transfer the data in case of multiple channels in Flume. However, Channel selectors are categorized into several types −

  1. Default channel selectors
  2. Multiplexing channel selector
  3. Custom Channel Selector

Note: Since, the type is not specified, then defaults to “replicating”.
Read more about Apache Flume Architecture & Features of Flume

Hadoop Quiz
If these professionals can make a switch to Big Data, so can you:
Rahul Doddamani Story - DataFlair
Rahul Doddamani
Java → Big Data Consultant, JDA
Follow on
Mritunjay Singh Success Story - DataFlair
Mritunjay Singh
PeopleSoft → Big Data Architect, Hexaware
Follow on
Rahul Doddamani Success Story - DataFlair
Rahul Doddamani
Big Data Consultant, JDA
Follow on
I got placed, scored 100% hike, and transformed my career with DataFlair
Enroll now
Deepika Khadri Success Story - DataFlair
Deepika Khadri
SQL → Big Data Engineer, IBM
Follow on
DataFlair Web Services
You could be next!
Enroll now

3. Types of Apache Flume Channel Selectors

There are three types of Channel selectors in Apache Flume are:

a. Replicating Channel Selector (Default)

Property NameDefaultDescription
selector.typereplicatingThe component type name needs to be replicating
selector.optionalSet of channels to be marked as optional

So, let’s see an example of Replicating (Default) Flume channel selectors, for agent named a1 and it’s source called r1:
a1.sources = r1
a1.channels = c1 c2 c3
a1.sources.r1.selector.type = replicating
a1.sources.r1.channels = c1 c2 c3
a1.sources.r1.selector.optional = c3
Since, we can see c3 is an optional channel in the above configuration. However, Failure to write to c3 is simply ignored. Also, make sure that failure to write to those channels will cause the transaction to fail since c1 and c2 are not marked optional.
Read to know more about Apache Flume Source

b. Multiplexing Channel Selector

Property Name

Join DataFlair on Telegram
DefaultDescription
 selector.typereplicatingThe component type name, needs to be multiplexing
 selector.headerflume.selector.header
 selector.default

selector.mapping.*

Also, see an example of Multiplexing Flume channel selectors, for agent  a1 and it’s source r1:
a1.sources = r1
a1.channels = c1 c2 c3 c4
a1.sources.r1.selector.type = multiplexing
a1.sources.r1.selector.header = state
a1.sources.r1.selector.mapping.CZ = c1
a1.sources.r1.selector.mapping.US = c2 c3
a1.sources.r1.selector.default = c4
Let’s know about Data Transfer from Flume to HDFS in detail

c. Custom Channel Selector

Basically, it is own implementation of the ChannelSelector interface. While starting the Flume agent a custom channel selector’s class and its dependencies must be included in the agent’s classpath. Although,  its FQCN is the type of the custom channel selector in Flume.

Property NameDefaultDescription
selector.typeThe component type name, needs to be your FQCN

So, let’s see an example of Custom Flume channel selectors, for agent named a1 and its source called r1:
a1.sources = r1
a1.channels = c1
a1.sources.r1.selector.type = org.example.MyChannelSelector
Let’s know more about Apache Flume Sink Processors

4. Conclusion

As a result, we have seen what are Flume Channel Selectors, types of Channel Selectors in Flume, Custom Channel Selector, Multiplexing Channel Selector, Replicating Channel Selector. Also, have seen its examples to understand well. Still, if you have any query, feel free to ask through the comment section.
See Also- Apache Flume Installation
For reference 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.