Flume Channel Selectors – Apache Flume

Flume Channel Selectors is the component of the Flume agent responsible for choosing the channel which transfers an event in case of multiple channels.

In this article, we will explore the Flume Channel Selector in detail. It explains the different types of Channel selector. It also describe the properties associated with different channel selectors along with examples.

Flume Channel Selectors

Flume Channel Selector is the component of the Flume agent which determines which channel should be selected for transferring an event when a group of channels exists. They work in between source and channel. The mechanism used is the internal mechanism.

The Channel Selector are categorized into three types:
1. Replicating channel selectors
2. Multiplexing channel selector
3. Custom Channel Selector

For any channel, the selector is specified using the selector.type property.
If the type is not specified, then the Replicating channel selector is used by default.

Let us now explore each type in detail.

1. Replicating Channel Selector in Flume

It is the default Channel Selector. If nothing is configured respective to the channel selector, then the Replicating channel selector decides the channel through which the event gets transferred.

Replicating channel selector works by replicating events into each channel by assuming that there is more than one channel.

Properties for Replicating Channel Selector:

Replicating channel selectors:

<Agent_name>.sources = <Source-name>
<Agent_name>.channels = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.type = replicating
<Agent_name>.sources.<source-name>.channels = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.optional = <Optional channel-number>

Example for agent named agent1 and it’s source called src , and channel be ch1, ch2, ch3:

agent1.sources = src
agent1.channels = ch1 ch2 ch3
agt1.sources.src.selector.type = replicating
agent1.sources.src.channels = ch1 ch2 ch3
agent1.sources.src.selector.optional = ch2

In the above configuration, ch2 is an optional channel. Failure for writing to ch2 is simply ignored. Failure to write to channel ch1, and ch3 would cause the transaction to fail as they are not marked as optional.

2. Multiplexing Channel Selector in Flume

Multiplexing Channel Selector writes an event to different channels on the basis of the information in the header.

Properties for Multiplexing Channel Selector:

Multiplexing Channel Selectors:

<Agent_name>.sources = <Source-name>
<Agent_name>.channels = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.type = multiplexing
<Agent_name>.sources.<source-name>.selector.header = <header-name>
<Agent_name>.sources.<source-name>.selector.mapping.<header-category> = <Channel1> <Channel2>……<Channeln>
…
<Agent_name>.sources.<source-name>.selector.mapping.<header-category> = <Channel1> <Channel2>……<Channeln>
<Agent_name>.sources.<source-name>.selector.default = <Channel1> <Channel2>……<Channeln>

Example for an agent named agent1 and source called src, and channel be ch1, ch2, ch3:

agent1.sources = src
agent1.channels = ch1 ch2 ch3 ch4
agent1.sources.src.selector.type = multiplexing
agent1.sources.src.selector.header = subject
agent1.sources.src.selector.mapping.subject1 = ch1
agent1.sources.src.selector.mapping.subject2 = ch2
agent1.sources.src.selector.default = ch4

In the above configuration, channels were separated by the header subject. subject1 are routed to ch1, subject2 are routed to ch2.
ch4 is a default channel.

3. Custom Channel Selector

The custom channel selector is our implementation of the ChannelSelector interface.
When starting the Flume agent, the custom channel selector’s class and its dependencies must be included in the agent’s classpath.

The Custom channel selector type is FQCN.

Property for Custom Channel Selector:

Custom Channel Selector:

<Agent_name>.sources = <Source-name>
<Agent_name>.channels = <Channel1>
<Agent_name>.sources.<source-name>.selector.type = custom selector type

Example for agent named agent1 and its source called src, and channel be ch1:

agent1.sources = src
agent1.channels = ch1
agent1.sources.src.selector.type = org.example.MyChannelSelector

Summary

I hope after reading this article you clearly understand the Flume Channel Selector. Channel Selector determines the channel through which the event gets transferred. The work in between source and channel.

The article explained the three different types of Channel selector that are replicating channel selector, multiplexing channel selector, and custom channel selector. The article also explains how to set the particular type of channel selector by using the selector.type property.