{"id":9324,"date":"2018-02-26T12:21:20","date_gmt":"2018-02-26T12:21:20","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=9324"},"modified":"2021-05-09T13:08:23","modified_gmt":"2021-05-09T07:38:23","slug":"flume-sink","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/flume-sink\/","title":{"rendered":"Apache Flume Sink – Types of Sink in Flume"},"content":{"rendered":"

In this article, we will study different sinks supported by Flume. The article covers all the Flume sinks along with their configuration properties and examples.
\nLet us first see a short introduction to Flume sink.<\/p>\n

Introduction to Flume sink<\/h2>\n

Apache Flume sink is the component of flume agent. It is used for storing data into a centralized store such as HDFS, HBase, etc. Sink consumes events from the Flume channel and pushes them on to the central repository.<\/p>\n

In simple words, the component that removes events from a Flume agent and writes it to another flume agent or some other system or a data store is called a sink. Sink is transactional in nature.<\/p>\n

Let us now explore different types of sink in detail.<\/p>\n

Types of Flume sink<\/h2>\n

1. HDFS sink<\/h3>\n

The HDFS sink writes flume events into the HDFS. The file formats supported by the HDFS sink are text files and sequence files. It provides support for compression in both file types.<\/p>\n

Based on the elapsed time or number of events or size of data, files can be rolled which means closing the current file and creating a new one. We can also use Hive concepts like bucketing, partitioning data by attributes in it.<\/p>\n

The HDFS directory path may sometimes contain formatting escape sequences. These escape sequences will be replaced by this sink for generating a directory or file name that stores the events.<\/p>\n

For using the HDFS sink we require Hadoop to be installed. Flume uses Hadoop jars for communicating with the HDFS cluster.<\/p>\n

Note: The Hadoop version must support the sync() call.<\/strong><\/p>\n

Some of the configuration property for HDFS sink are:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be hdfs.<\/span><\/td>\n<\/tr>\n
hdfs.path<\/td>\n–<\/span><\/td>\nIt specifies the HDFS directory path. For eg, hdfs:\/\/namenode\/flume\/webdata\/<\/span><\/td>\n<\/tr>\n
hdfs.filePrefix<\/span><\/td>\nFlumeData<\/span><\/td>\nIt specifies the name prefixed to the files created by Apache Flume in the HDFS directory.<\/span><\/td>\n<\/tr>\n
hdfs.fileSuffix<\/span><\/td>\n–<\/span><\/td>\nIt specifies the suffix to append to file. For\u00a0 eg, .avro\u00a0<\/span><\/td>\n<\/tr>\n
hdfs.inUsePrefix<\/span><\/td>\n–<\/span><\/td>\nIt specifies the prefix that is used for the temporary files that flume actively writes into.<\/span><\/td>\n<\/tr>\n
hdfs.inUseSuffix<\/span><\/td>\n.tmp<\/span><\/td>\nIt specifies the suffix that is used for the temporary files that flume actively writes into.<\/span><\/td>\n<\/tr>\n
hdfs.rollInterval<\/span><\/td>\n30<\/span><\/td>\nIt specifies the number of seconds to wait before rolling the current file. If it is equal to 0 then it means never roll based on the time interval.<\/span><\/td>\n<\/tr>\n
hdfs.rollSize<\/span><\/td>\n1024<\/span><\/td>\nIt specifies the file size to trigger roll, in bytes. If it is equal to 0 then it means never roll based on file size.<\/span><\/td>\n<\/tr>\n
hdfs.rollCount<\/span><\/td>\n10<\/span><\/td>\nIt specifies the number of events written to the file before it rolled. If it is equal to 0 then it means never roll based on the number of events.<\/span><\/td>\n<\/tr>\n
hdfs.idleTimeout<\/span><\/td>\n0<\/span><\/td>\nIt specifies the timeout after which inactive files get closed. If it is equal to 0 then it means disabling automatic closing of idle files.\u00a0<\/span><\/td>\n<\/tr>\n
hdfs.batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the number of events written to file before it is flushed to HDFS.<\/span><\/td>\n<\/tr>\n
hdfs.fileType<\/span><\/td>\nSequenceFile<\/span><\/td>\nIt specifies the File format. File formats that are currently supported are SequenceFile, DataStream or CompressedStream.<\/span><\/p>\n

The DataStream will not compress the output file and please don\u2019t set codeC. The CompressedStream requires set hdfs.codeC with an available codeC<\/span><\/td>\n<\/tr>\n

hdfs.maxOpenFiles<\/span><\/td>\n5000<\/span><\/td>\nIt specifies the number of files allowed to be open. The oldest file is closed<\/span> if this number is exceeded.<\/span><\/td>\n<\/tr>\n
hdfs.round<\/span><\/td>\nfalse<\/span><\/td>\nIt specifies should the timestamp be rounded down. If it is set to true then it affects all time-based escape sequences except %t<\/span><\/td>\n<\/tr>\n
hdfs.roundValue<\/span><\/td>\n1<\/span><\/td>\nRounded down to the highest multiple of this (in the unit configured using hdfs.roundUnit), less than the current time.<\/span><\/td>\n<\/tr>\n
hdfs.roundUnit<\/span><\/td>\nsecond<\/span><\/td>\nIt specifies the unit of the round down value. It can be second, minute or hour<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n

agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = hdfs\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.hdfs.path = \/flume\/events\/%y-%m-%d\/%H%M\/%S\nagent1.sinks.sk1.hdfs.filePrefix = events-\nagent1.sinks.sk1.hdfs.round = true\nagent1.sinks.sk1.hdfs.roundValue = 10\nagent1.sinks.sk1.hdfs.roundUnit = minute\n<\/pre>\n
This above configuration rounds down the timestamp to the last 10th minute.<\/p>\n
For example, a flume event with the timestamp 11:44:34 AM, April 08, 2020, will cause the hdfs path to become \/flume\/events\/2020-04-08\/1140\/00.<\/p>\n
2. Hive sink<\/h3>\n
The Hive sink streams flume events containing delimited text or JSON data directly into the Hive table or partition. The Flume Events are written using Hive transactions.<\/p>\n
As soon as a set of flume events are committed to Hive, they become immediately visible to Hive queries. We can either pre-create or optionally Flume can create the partitions to which the Flume will stream to if they are missing.<\/p>\n
Fields from incoming flume event data are mapped to the corresponding columns in a table in Hive.<\/p>\n
Some of the configuration property of hive are:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be hive.<\/span><\/td>\n<\/tr>\n
hive.metastore<\/td>\n–<\/span><\/td>\nIt specifies the Hive metastore URI. For eg, thrift:\/\/a.b.com:9083\u00a0<\/span><\/td>\n<\/tr>\n
hive.database<\/td>\n–<\/span><\/td>\nIt specifies the Hive database name.<\/span><\/td>\n<\/tr>\n
hive.table<\/td>\n–<\/span><\/td>\nIt specifies the Hive table name.<\/span><\/td>\n<\/tr>\n
hive.partition<\/span><\/td>\n–<\/span><\/td>\nIt specifies the comma-separated list of partition values identifying the partition to write to. It may contain escape sequences. For E.g: If the table is partitioned by continent: string, country :string, and\u00a0 time : string, then \u2018Asia,India,2020-04-08-01-21\u2019 will indicate continent=Asia,country=India,time=2020-04-08-01-21<\/span><\/td>\n<\/tr>\n
batchSize<\/span><\/td>\n15000<\/span><\/td>\nIt specifies the maximum number of flume events written to Hive in a single Hive transaction<\/span><\/td>\n<\/tr>\n
serializer<\/td>\n–<\/span><\/td>\nSerializer is responsible for parsing out the fields from the event and mapping them to the columns in the table in Hive. The choice of serializer depends upon the format of the data in the flume event. The supported serializers are DELIMITED and JSON<\/span><\/td>\n<\/tr>\n
roundUnit<\/td>\nminute<\/span><\/td>\nIt specifies the unit of the round down value. It can be second, minute or hour<\/span><\/td>\n<\/tr>\n
roundValue<\/td>\n1<\/span><\/td>\nRounded down to the highest multiple of this (in the unit configured using hive.roundUnit), less than the current time<\/span><\/td>\n<\/tr>\n
useLocalTimeStamp<\/td>\nfalse<\/span><\/td>\nIt specifies whether to use the local time (instead of the timestamp from the event header) while replacing the escape sequences.<\/span><\/td>\n<\/tr>\n
serializer.delimiter<\/td>\n,<\/span><\/td>\n(Type: string). It specifies the field delimiter in the incoming data.<\/span> For using special characters, surround characters with double quotes like \u201c\\t\u201d<\/span><\/td>\n<\/tr>\n
serializer.fieldnames<\/td>\n–<\/span><\/td>\nThe mapping from input fields to columns in the table in Hive. It is Specified as a comma-separated list (no spaces) of the hive table columns name, identifying the input fields in order of their occurrence. For skipping fields leave the column name unspecified. For eg. \u2018time,,ip,message\u2019 indicates the 1st, 3rd and 4th fields in the input map to time, ip and message columns in the hive table.<\/span><\/td>\n<\/tr>\n
serializer.serdeSeparator<\/td>\nCtrl-A<\/span><\/td>\n(Type: character). It customizes the separators which are used by the underlying serde. We can increase efficiency if:<\/span><\/p>\n
    \n
  1. the fields in serializer.fieldnames are in the same order as table columns<\/span><\/li>\n
  2. the serializer.delimiter is same as the serializer.serdeSeparator<\/span><\/li>\n
  3. the number of fields in serializer.fieldnames is less than or equal to the number of table columns, as the fields in the incoming event body do not need to be reordered to match the order of table columns.<\/span><\/li>\n<\/ol>\n
    Use single quotes for special characters like \u2018\\t\u2019. Ensure that the input fields do not contain this character.\u00a0<\/span><\/p>\n
    NOTE: If serializer.delimiter is a single character, preferably set this to the same character<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for Hive table :<\/p>\n
    create table logs ( id int , msg string )\n    partitioned by (continent string, country string, time string)\n    clustered by (id) into 5 buckets\n    stored as orc;\n<\/pre>\n
    Example for agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.channels.ch1.type = memory\nagent1.sinks = sk1\nagent1.sinks.sk1.type = hive\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.hive.metastore = thrift:\/\/127.0.0.1:9083\nagent1.sinks.sk1.hive.database = logsdb\nagent1.sinks.sk1.hive.table = logs\nagent1.sinks.sk1.hive.partition = asia,%{country},%y-%m-%d-%H-%M\nagent1.sinks.sk1.useLocalTimeStamp = false\nagent1.sinks.sk1.round = true\nagent1.sinks.sk1.roundValue = 10\nagent1.sinks.sk1.roundUnit = minute\nagent1.sinks.sk1.serializer = DELIMITED\nagent1.sinks.sk1.serializer.delimiter = \"\\t\"\nagent1.sinks.sk1.serializer.serdeSeparator = '\\t'\nagent1.sinks.sk1.serializer.fieldnames =id,,msg\n<\/pre>\n
    This above configuration rounds down the timestamp to the last 10th minute.<\/p>\n
    For example, a flume event with the timestamp header set to 11:44:34 AM, April 08, 2020, and \u2018country\u2019 header set to \u2018India\u2019 will evaluate to the partition (continent=\u2019asia\u2019, country=\u2019India\u2019, time=\u20182020-04-08-11-40\u2019.<\/p>\n
    The serializer is configured in such a way that to accept tab-separated input containing three fields and skip the second field.<\/p>\n
    3. Logger sink<\/h3>\n
    The Logger sink logs event at the information level. It is used for testing or debugging purposes. Logger sink is the only exception that does not require the extra configuration.<\/p>\n
    The configuration property for logger sink are:<\/p>\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be logger.<\/span><\/td>\n<\/tr>\n
    maxBytesToLog<\/td>\n16<\/span><\/td>\nIt specifies the maximum number of bytes of the Event body to log.\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = logger\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    4. Avro sink<\/h3>\n
    The Avro sink turns Flume events into Avro events and sends them to the configured hostname\/port pair. The flume events are taken in batches of configured batch size from the configured Channel. The Avro sink forms one half of the Apache Flume\u2019s tiered collection support.<\/p>\n
    Some of the properties of the Avro sink are:<\/p>\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be avro.<\/span><\/td>\n<\/tr>\n
    hostname<\/td>\n–<\/span><\/td>\nIt specifies the hostname or IP address to bind to.<\/span><\/td>\n<\/tr>\n
    port<\/td>\n–<\/span><\/td>\nIt specifies the port # to listen on.<\/span><\/td>\n<\/tr>\n
    batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the number of events to batch together for sending.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = avro\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.hostname = 10.10.10.10\nagent1.sinks.sk1.port = 4545\n<\/pre>\n
    5. Thrift sink<\/h3>\n
    The Thrift sink turns the Flume events into the Thrift events and sends them to the configured hostname\/port pair. The flume events are taken in batches of configured batch size from the configured Channel.<\/p>\n
    We can configure the Thrift sink to start in secure mode by enabling the Kerberos authentication. For communicating with a Thrift source which is started in the secure mode we need the Thrift sink to be operated in a secure mode.<\/p>\n
    The client-principal and client-keytab are two properties used by the Thrift sink for authentication to the Kerberos KDC.<\/p>\n
    Some of the configuration property for Thrift source are:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be thrift.<\/span><\/td>\n<\/tr>\n
    hostname<\/td>\n–<\/span><\/td>\nIt specifies the hostname or IP address to bind to.<\/span><\/td>\n<\/tr>\n
    port<\/td>\n–<\/span><\/td>\nIt specifies the port # to listen on.<\/span><\/td>\n<\/tr>\n
    batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the number of events to batch together for sending.<\/span><\/td>\n<\/tr>\n
    kerberos<\/span><\/td>\n<\/td>\nIt specifies whether to enable Kerberos authentication or not. For enabling Kerberos authentication, it is set to true.\u00a0<\/span><\/p>\n
    In kerberos mode, client-principal, client-keytab, and server-principal are required for successful authentication and communication to a Kerberos enabled Thrift Source.<\/span><\/td>\n<\/tr>\n
    client-principal<\/span><\/td>\n<\/td>\nIt specifies the Kerberos principal used by the Thrift Sink for authenticating to the Kerberos KDC.<\/span><\/td>\n<\/tr>\n
    client-keytab<\/span><\/td>\n<\/td>\nIt specifies the keytab location used by the Thrift Sink in combination with the client-principal for authenticating to the Kerberos KDC.<\/span><\/td>\n<\/tr>\n
    server-principal<\/span><\/td>\n<\/td>\nThe Kerberos principal of the Thrift Source to which the Thrift Sink is configured to connect to.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = thrift\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.hostname = 10.10.10.10\nagent1.sinks.sk1.port = 4545\n<\/pre>\n
    6. IRC sink<\/h3>\n
    This Apache Flume sink takes messages from the attached flume channel and relays those to the configured IRC destinations.<\/p>\n
    Some of the configuration properties for IRC sink are:<\/p>\n\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be irc.<\/span><\/td>\n<\/tr>\n
    hostname<\/td>\n–<\/span><\/td>\nIt specifies the hostname or IP address to bind to.<\/span><\/td>\n<\/tr>\n
    port<\/td>\n6667<\/span><\/td>\nIt specifies the port number of the remote host to connect<\/span><\/td>\n<\/tr>\n
    nick<\/td>\n100<\/span><\/td>\nIt specifies the Nickname<\/span><\/td>\n<\/tr>\n
    chan<\/td>\n<\/td>\nIt specifies the channel.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = irc\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.hostname = irc.yourdomain.com\nagent1.sinks.sk1.nick = flume\nagent1.sinks.sk1.chan = #flume\n<\/pre>\n
    7. File Roll sink<\/h3>\n
    This sink stores event on the local filesystem.<\/p>\n
    Some of the properties for file roll sink are:<\/p>\n\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be file_roll.<\/span><\/td>\n<\/tr>\n
    sink.directory<\/td>\n–<\/span><\/td>\nIt specifies the directory where files will be stored<\/span><\/td>\n<\/tr>\n
    sink.pathManager<\/span><\/td>\nDEFAULT<\/span><\/td>\nIt specifies the PathManager implementation to use.<\/span><\/td>\n<\/tr>\n
    sink.pathManager.extension<\/span><\/td>\n–<\/span><\/td>\nIt specifies the file extension if the default PathManager is used<\/span><\/td>\n<\/tr>\n
    sink.batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the batch size.\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = file_roll\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.sink.directory = \/var\/log\/flume\n<\/pre>\n
    8. Null sink<\/h3>\n
    Null sink discards all flume events it receives from the channel.<\/p>\n
    Some of the property for the null sink are:<\/p>\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be null.<\/span><\/td>\n<\/tr>\n
    batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the batch size.\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = null\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    9. HBase Sink<\/h3>\n
    9.1. HBaseSink<\/h4>\n
    The HBaseSink writes data to the HBase.The Hbase configuration is taken up from the first hbase-site.xml found in the classpath.<\/p>\n
    A class that is implementing the HbaseEventSerializer specified by the configuration is used for converting the events into HBase puts and\/or increments.<\/p>\n
    These HBase puts and increments are then written to HBase. This Flume sink guarantees the same consistency as HBase, which is row-wise atomicity.<\/p>\n
    In the case of failure of writing certain events, this sink will replay all the events in that transaction.
    \nThis sink supports writing data to the secure HBase. For writing data to the secure HBase, the agent must have the write permissions to the table the sink is configured to write to.<\/p>\n
    We can specify the principal and keytab property use for authentication against the KDC in the configuration.<\/p>\n
    The hbase-site.xml in the Flume agent\u2019s classpath must have an authentication set to Kerberos.<\/p>\n
    Two serializers are provided with Apache Flume.<\/strong><\/p>\n
    a) SimpleHbaseEventSerializer<\/h5>\n
    (org.apache.flume.sink.hbase.SimpleHbaseEventSerializer)<\/p>\n
    This serializer writes the event body as it is to HBase, and optionally increments a column in Hbase.<\/p>\n
    b) RegexHbaseEventSerializer<\/h5>\n
    (org.apache.flume.sink.hbase.RegexHbaseEventSerializer)<\/p>\n
    This serializer breaks the event body on the basis of the given regex. Then it writes each part into different columns.<\/p>\n
    The type is the FQCN: org.apache.flume.sink.hbase.HBaseSink.<\/p>\n
    Some of the configuration property for HBaseSink are:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be hbase.<\/span><\/td>\n<\/tr>\n
    table<\/td>\n–<\/span><\/td>\nIt specifies the name of the table in Hbase to write to.<\/span><\/td>\n<\/tr>\n
    columnFamily<\/td>\n–<\/span><\/td>\nIt specifies the column family in Hbase to write to.<\/span><\/td>\n<\/tr>\n
    batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the number of flume events to be written per transaction.<\/span><\/td>\n<\/tr>\n
    kerberosPrincipal<\/span><\/td>\n–<\/span><\/td>\nIt specifies the Kerberos user principal for accessing secure HBase.<\/span><\/td>\n<\/tr>\n
    kerberosKeytab<\/span><\/td>\n–<\/span><\/td>\nIt specifies the Kerberos keytab for accessing secure HBase.<\/span><\/td>\n<\/tr>\n
    serializer<\/span><\/td>\norg.apache.flume.sink.hbase.SimpleHbaseEventSerializer<\/span><\/td>\nDefault increment column = \u201ciCol\u201d, payload column = \u201cpCol\u201d.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = hbase\nagent1.sinks.sk1.table = employee\nagent1.sinks.sk1.columnFamily = family1\nagent1.sinks.sk1.serializer = org.apache.flume.sink.hbase.RegexHbaseEventSerializer\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    9.2 HBase2Sink<\/h4>\n
    The sink is the equivalent of HBaseSink for the HBase version 2.<\/p>\n
    The functionality and the configuration parameters for HBase2Sink are the same as in the case of HBaseSink except for the type which is hbase2 and the package\/class names.<\/p>\n
    The type is the FQCN: org.apache.flume.sink.hbase2.HBase2Sink.<\/p>\n
    Some of the property for HBase2Sink are:<\/p>\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be hbase2.<\/span><\/td>\n<\/tr>\n
    table<\/td>\n–<\/span><\/td>\nIt specifies the name of the table in Hbase to write to.<\/span><\/td>\n<\/tr>\n
    columnFamily<\/td>\n–<\/span><\/td>\nIt specifies the column family in Hbase to write to.<\/span><\/td>\n<\/tr>\n
    serializer<\/td>\norg.apache.flume.sink.hbase2.SimpleHBase2EventSerializer<\/span><\/td>\nDefault increment column = \u201ciCol\u201d, payload column = \u201cpCol\u201d.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = hbase2\nagent1.sinks.sk1.table = employee\nagent1.sinks.sk1.columnFamily = family1\nagent1.sinks.sk1.serializer = org.apache.flume.sink.hbase2.RegexHBase2EventSerializer\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    9.3 AsyncHBaseSink<\/h4>\n
    AsyncHBaseSink writes data to HBase by using an asynchronous model.<\/p>\n
    It uses a class which implements AsyncHbaseEventSerializer specified by the configuration for converting the flume events into HBase puts and\/or increments. These HBase puts and increments are then written to HBase.<\/p>\n
    AsyncHBaseSink uses the Asynchbase API for writing to HBase. This flume sink guarantees the same consistency as HBase, which is currently row-wise atomicity.<\/p>\n
    In the case of failure of writing certain events, this sink will replay all the events in that transaction. We can use this sink only with HBase 1.x. For HBase 2, the async library used by this sink is not available.<\/p>\n
    The type is the FQCN: org.apache.flume.sink.hbase.AsyncHBaseSink.<\/p>\n
    Some of the properties for AsyncHBaseSink are:<\/p>\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be asynchbase.<\/span><\/td>\n<\/tr>\n
    table<\/td>\n–<\/span><\/td>\nIt specifies the name of the table in Hbase to write to.<\/span><\/td>\n<\/tr>\n
    columnFamily<\/td>\n–<\/span><\/td>\nIt specifies the column family in Hbase to write to.<\/span><\/td>\n<\/tr>\n
    serializer<\/td>\norg.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer<\/span><\/td>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = asynchbase\nagent1.sinks.sk1.table = employee\nagent1.sinks.sk1.columnFamily =family1\nagent1.sinks.sk1.serializer = org.apache.flume.sink.hbase.SimpleAsyncHbaseEventSerializer\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    10. MorphlineSolrSink<\/h3>\n
    MorphlineSolrSink extracts data from the Flume events. They transform it and load them in near-real-time into the Apache Solr servers. Apache Solr servers in turn cater queries to the end-users or search applications.<\/p>\n
    It is the best option for the use cases that stream raw data into HDFS through the HdfsSink and simultaneously extract, transform, and load the same raw data into Apache Solr servers through MorphlineSolrSink.<\/p>\n
    We can configure the ETL functionality by using a morphline configuration file. This file defines a chain of transformation commands that the pipe event records from one command to another.<\/p>\n
    A morphline command is a little bit like a Flume Interceptor. We can embed Morphlines into Hadoop components such as Apache Flume.<\/p>\n
    The type is the FQCN: org.apache.flume.sink.solr.morphline.MorphlineSolrSink<\/p>\n
    Some of the properties for MorphlineSolrSink are:<\/p>\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be org.apache.flume.sink.solr.morphline.MorphlineSolrSink.<\/span><\/td>\n<\/tr>\n
    morphlineFile<\/td>\n–<\/span><\/td>\nIt specifies the relative or absolute path on the local file system to the morphline configuration file. For eg: \/etc\/flume-ng\/conf\/morphline.conf<\/span><\/td>\n<\/tr>\n
    batchSize<\/span><\/td>\n1000<\/span><\/td>\nIt specifies the maximum number of events to take per flume transaction.<\/span><\/td>\n<\/tr>\n
    morphlineId<\/td>\nnull<\/span><\/td>\nIt specifies the optional name that can be\u00a0 used for identifying a morphline if there are multiple morphlines in a morphline config file.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.morphlineFile = \/etc\/flume-ng\/conf\/morphline.conf\n# agent1.sinks.sk1.morphlineId = morphline1\n# agent1.sinks.sk1.batchSize = 1000\n# agent1.sinks.sk1.batchDurationMillis = 1000\n<\/pre>\n
    11. ElasticSearchSink<\/h3>\n
    ElasticSearchSink writes data to an elastic search cluster. The events will be written by default so that the Kibana graphical interface displays them as if logstash wrote them.<\/p>\n
    Make sure that the elasticsearch and the Lucene-core jar which are required for our environment are placed in the lib directory of Flume installation.<\/p>\n
    The Elasticsearch requires a major version of the client JAR to match with that of the server and both are running the same minor version of the JVM. If this is not correct then SerializationExceptions will appear.<\/p>\n
    Every day, events will be written to a new index. The name will be <indexName>-yyyy-MM-dd where <indexName> is the indexName parameter. The sink will start writing to a new index at midnight UTC.<\/p>\n
    By default, Events are serialized for elasticsearch by the ElasticSearchLogStashEventSerializer. The Serializer parameter can override this behavior.<\/p>\n
    The type is the FQCN: org.apache.flume.sink.elasticsearch.ElasticSearchSink<\/p>\n
    Some of the properties for ElasticSearchSink are:<\/p>\n\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be org.apache.flume.sink.elasticsearch.ElasticSearchSink<\/span><\/td>\n<\/tr>\n
    hostNames<\/td>\n–<\/span><\/td>\nIt specifies the comma-separated list of the hostname:port. If in case the port is not present the default port \u20189300\u2019 will be used<\/span><\/td>\n<\/tr>\n
    clusterName<\/span><\/td>\nelasticsearch<\/span><\/td>\nIt specifies the name of the ElasticSearch cluster to connect to<\/span><\/td>\n<\/tr>\n
    batchSize<\/td>\n100<\/span><\/td>\nIt specifies the number of events to be written per transaction.\u00a0\u00a0<\/span><\/td>\n<\/tr>\n
    serializer<\/td>\norg.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer<\/span><\/td>\nThe ElasticSearchIndexRequestBuilderFactory or ElasticSearchEventSerializer to use. The implementations of either class are accepted but ElasticSearchIndexRequestBuilderFactory is preferred.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = elasticsearch\nagent1.sinks.sk1.hostNames = 127.0.0.1:9200,127.0.0.2:9300\nagent1.sinks.sk1.indexName = employee_index\nagent1.sinks.sk1.indexType = bar_type\nagent1.sinks.sk1.clusterName =employee_cluster\nagent1.sinks.sk1.batchSize = 500\nagent1.sinks.sk1.ttl = 5d\nagent1.sinks.sk1.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchDynamicSerializer\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    12. Kite Dataset Sink<\/h3>\n
    Kite Dataset Sink is an experimental sink. This sink writes events to a Kite Dataset.<\/p>\n
    Kite Dataset sink deserializes the body of each incoming flume event and stores the resulting record in the Kite Dataset. By loading a dataset by URI it determines the target Dataset.<\/p>\n
    It supports only Avro serialization. The record schema must be passed in the event headers by using any of the two:<\/p>\n
    a) flume.avro.schema.literal with the JSON schema representation<\/p>\n
    b) flume.avro.schema.url with a URL where the schema may be found.<\/p>\n
    Some of the properties for Kite Dataset Sink are:<\/p>\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be org.apache.flume.sink.kite.DatasetSink<\/span><\/td>\n<\/tr>\n
    kite.dataset.uri<\/td>\n–<\/span><\/td>\nIt specifies the URI of the dataset to open.<\/span><\/td>\n<\/tr>\n
    kite.batchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies the number of records to process in each batch.<\/span><\/td>\n<\/tr>\n
    kite.dataset.uri<\/td>\n–<\/span><\/td>\nIt specifies the URI of the repository to open.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    13. Kafka sink<\/p>\n
    Kafka sink is a Flume Sink implementation which publishes data to a Kafka topic. The main objective behind it is to integrate Apache Flume with Kafka. This makes pull-based processing systems to process the data coming from various Flume sources.<\/p>\n
    This sink currently supports Kafka server 0.10.1.0 release or higher releases.<\/p>\n
    Some of the properties for Kafka Sink are:<\/p>\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be org.apache.flume.sink.kafka.KafkaSink<\/span><\/td>\n<\/tr>\n
    kafka.bootstrap.servers<\/td>\n–<\/span><\/td>\nIt specifies the list of brokers to which the Kafka-Sink will connect to, to get the list of topic partitions.\u00a0 This list can be a partial list of brokers. It is recommended at least two for HA. The format used is comma-separated list of hostname:port<\/span><\/td>\n<\/tr>\n
    kafka.topic<\/span><\/td>\ndefault-flume-topic<\/span><\/td>\nIt specifies the topic in Kafka to which the messages will be published. If we configure this parameter then messages will be published to this topic. In case if the event header contains a \u201ctopic\u201d field then the event will be published to that topic overriding the topic configured here.<\/span><\/td>\n<\/tr>\n
    flumeBatchSize<\/span><\/td>\n100<\/span><\/td>\nIt specifies how many messages should be processed in one batch. Larger batches improve throughput while adding latency.<\/span><\/td>\n<\/tr>\n
    kafka.producer.acks<\/td>\n1<\/span><\/td>\nIt specifies how many replicas must acknowledge a message before it is considered successfully written. The values accepted are 0\u00a0 which means never wait for acknowledgment, 1\u00a0 which means wait for the leader only, and -1\u00a0 which means wait for all replicas. Set this to -1 to avoid data loss in some cases of leader failure.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.type = org.apache.flume.sink.kafka.KafkaSink\nagent1.sinks.sk1.kafka.topic = mytopic\nagent1.sinks.sk1.kafka.bootstrap.servers = localhost:9092\nagent1.sinks.sk1.kafka.flumeBatchSize = 20\nagent1.sinks.sk1.kafka.producer.acks = 1\nagent1.sinks.sk1.kafka.producer.linger.ms = 1\nagent1.sinks.sk1.kafka.producer.compression.type = snappy\n<\/pre>\n
    14. HTTP sink<\/h3>\n
    HTTP sink takes events from the channel and sends them to a remote service by using an HTTP POST request. The content of the event is sent as the POST body.<\/p>\n
    This sink handles errors depending on the HTTP response returned by a target server.<\/p>\n
    The HTTP sink back off or ready status is configurable, as is the transaction commit\/rollback result and whether the event contributes to the successful event drain count.<\/p>\n
    If a malformed HTTP response is returned by the server where the status code is not readable then it will result in a backoff signal. In such a case the event is not consumed from the channel.<\/p>\n
    Some of the properties for HTTP Sink are:<\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be http<\/span><\/td>\n<\/tr>\n
    endpoint<\/td>\n–<\/span><\/td>\nIt specifies the fully qualified URL endpoint to POST to<\/span><\/td>\n<\/tr>\n
    connectTimeout<\/span><\/td>\n5000<\/span><\/td>\nIt specifies the socket connection timeout in milliseconds<\/span><\/td>\n<\/tr>\n
    requestTimeout<\/td>\n5000<\/span><\/td>\nIt specifies the maximum request processing time in milliseconds<\/span><\/td>\n<\/tr>\n
    contentTypeHeader<\/td>\ntext\/plain<\/span><\/td>\nIt specifies the\u00a0 HTTP Content-Type header<\/span><\/td>\n<\/tr>\n
    acceptHeader<\/td>\ntext\/plain<\/span><\/td>\nIt specifies the HTTP Accept header value<\/span><\/td>\n<\/tr>\n
    defaultBackoff<\/td>\ntrue<\/span><\/td>\nIt specifies whether to backoff by default on receiving all HTTP status codes\u00a0<\/span><\/td>\n<\/tr>\n
    defaultRollback<\/td>\ntrue<\/span><\/td>\nIt specifies whether to rollback by default on receiving all HTTP status codes<\/span><\/td>\n<\/tr>\n
    defaultIncrementMetrics<\/td>\nfalse<\/span><\/td>\nIt specifies whether to increment metrics by default on receiving all HTTP status codes<\/span><\/td>\n<\/tr>\n
    backoff.CODE<\/td>\n–<\/span><\/td>\nIt configures a specific backoff for an individual code or a group code.\u00a0<\/span><\/td>\n<\/tr>\n
    rollback.CODE<\/td>\n–<\/span><\/td>\nIt configures a specific rollback for an individual code or a group code.\u00a0<\/span><\/td>\n<\/tr>\n
    incrementMetrics.CODE<\/td>\n–<\/span><\/td>\nIt configures a specific metrics increment for an individual or a group code.\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for the agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = http\nagent1.sinks.sk1.channel = ch1\nagent1.sinks.sk1.endpoint = http:\/\/localhost:8080\/someuri\nagent1.sinks.sk1.connectTimeout = 2000\nagent1.sinks.sk1.requestTimeout = 2000\nagent1.sinks.sk1.acceptHeader = application\/json\nagent1.sinks.sk1.contentTypeHeader = application\/json\nagent1.sinks.sk1.defaultBackoff = true\nagent1.sinks.sk1.defaultRollback = true\nagent1.sinks.sk1.defaultIncrementMetrics = false\nagent1.sinks.sk1.backoff.4XX = false\nagent1.sinks.sk1.rollback.4XX = false\nagent1.sinks.sk1.incrementMetrics.4XX = true\nagent1.sinks.sk1.backoff.200 = false\nagent1.sinks.sk1.rollback.200 = false\nagent1.sinks.sk1.incrementMetrics.200 = true\n<\/pre>\n
    15. Custom sink<\/h3>\n
    This sink is our own implementation for the Sink interface. We must include a custom sink\u2019s class and its dependencies in the agent\u2019s classpath while starting the Flume agent.<\/p>\n
    Some of the properties for the custom sink are:<\/p>\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Value<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    channel<\/td>\n–<\/span><\/td>\nSpecify the channels through which an event travels.<\/span><\/td>\n<\/tr>\n
    type<\/td>\n–<\/span><\/td>\nIt specifies the component type. It must be our FQCN<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for agent named agent1, sink sk1, and channel ch1:<\/p>\n
    agent1.channels = ch1\nagent1.sinks = sk1\nagent1.sinks.sk1.type = org.example.MySink\nagent1.sinks.sk1.channel = ch1\n<\/pre>\n
    Summary<\/h2>\n
    I hope the article clearly explained different sinks supported by Flume. The flume sink is the component of the flume agent that consumes data from the flume channel and pushes them on the central repository. The article had explained different flume sinks along with their properties and examples.<\/p>\n","protected":false},"excerpt":{"rendered":"
    In this article, we will study different sinks supported by Flume. The article covers all the Flume sinks along with their configuration properties and examples. Let us first see a short introduction to Flume...<\/p>\n","protected":false},"author":6,"featured_media":78373,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[1318,3229,4124,4685,4836,5503,5597,5786,6415,7307,7944,8039,8374,8871,14718],"class_list":["post-9324","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flume","tag-avro-sink","tag-custom-sink","tag-elasticsearchsink","tag-file-roll-sink","tag-flume-sinks","tag-hbasesink","tag-hdfs-sink","tag-hive-sink","tag-http-sink","tag-irc-sink","tag-kafka-sink","tag-kite-dataset-sink","tag-logger-sink","tag-morphlinesolrsink","tag-thrift-sink"],"yoast_head":"\nApache Flume Sink - Types of Sink in Flume - DataFlair<\/title>\n<meta name=\"description\" content=\"What is Flume Sink-Types of Sink in flume-HDFS Sink,Hive Sink,Logger Sink,Null Sink,Flume Kafka Sink,HTTP Sink,Custom Sink,thrift Sink,Flume Avro Sink with examples\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/flume-sink\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Flume Sink - Types of Sink in Flume - DataFlair\" \/>\n<meta property=\"og:description\" content=\"What is Flume Sink-Types of Sink in flume-HDFS Sink,Hive Sink,Logger Sink,Null Sink,Flume Kafka Sink,HTTP Sink,Custom Sink,thrift Sink,Flume Avro Sink with examples\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/flume-sink\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-26T12:21:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:38:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/types-of-flume-sinks.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"20 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Flume Sink - Types of Sink in Flume - DataFlair","description":"What is Flume Sink-Types of Sink in flume-HDFS Sink,Hive Sink,Logger Sink,Null Sink,Flume Kafka Sink,HTTP Sink,Custom Sink,thrift Sink,Flume Avro Sink with examples","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/flume-sink\/","og_locale":"en_US","og_type":"article","og_title":"Apache Flume Sink - Types of Sink in Flume - DataFlair","og_description":"What is Flume Sink-Types of Sink in flume-HDFS Sink,Hive Sink,Logger Sink,Null Sink,Flume Kafka Sink,HTTP Sink,Custom Sink,thrift Sink,Flume Avro Sink with examples","og_url":"https:\/\/data-flair.training\/blogs\/flume-sink\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-02-26T12:21:20+00:00","article_modified_time":"2021-05-09T07:38:23+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/types-of-flume-sinks.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"20 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Apache Flume Sink – Types of Sink in Flume","datePublished":"2018-02-26T12:21:20+00:00","dateModified":"2021-05-09T07:38:23+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/"},"wordCount":4165,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/types-of-flume-sinks.jpg","keywords":["Avro Sink","Custom Sink","ElasticSearchSink","File Roll Sink","Flume Sinks","HBaseSink","HDFS Sink","Hive Sink","HTTP Sink","IRC Sink","Kafka Sink","Kite Dataset Sink","Logger Sink","MorphlineSolrSink","Thrift Sink"],"articleSection":["Flume Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/flume-sink\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/","url":"https:\/\/data-flair.training\/blogs\/flume-sink\/","name":"Apache Flume Sink - Types of Sink in Flume - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/types-of-flume-sinks.jpg","datePublished":"2018-02-26T12:21:20+00:00","dateModified":"2021-05-09T07:38:23+00:00","description":"What is Flume Sink-Types of Sink in flume-HDFS Sink,Hive Sink,Logger Sink,Null Sink,Flume Kafka Sink,HTTP Sink,Custom Sink,thrift Sink,Flume Avro Sink with examples","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/flume-sink\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/types-of-flume-sinks.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/types-of-flume-sinks.jpg","width":802,"height":420,"caption":"types of flume sinks"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/flume-sink\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Flume Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flume\/"},{"@type":"ListItem","position":3,"name":"Apache Flume Sink – Types of Sink in Flume"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/9324","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=9324"}],"version-history":[{"count":1,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/9324\/revisions"}],"predecessor-version":[{"id":92654,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/9324\/revisions\/92654"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/78373"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=9324"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=9324"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=9324"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}