{"id":8846,"date":"2018-02-21T08:46:35","date_gmt":"2018-02-21T08:46:35","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=8846"},"modified":"2021-05-09T13:08:29","modified_gmt":"2021-05-09T07:38:29","slug":"flume-event-serializers","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/","title":{"rendered":"Flume Event Serializers – Apache Flume"},"content":{"rendered":"

Apache Flume is a distributed system for transferring data from external sources to the HDFS or HBase. The event serializer is a mechanism that is used for converting a flume event into another format for the output.<\/p>\n

In this article, you will explore what event serializer is. The article covers different EventSerializers that ship with Apache Flume. You will see different event serializers along with their configuration options and examples.<\/p>\n

Introduction to Event Serializers<\/h2>\n

Event serializers is a mechanism which is used for converting a flume event into another format for the output. The Event serializer is an interface that allows for the random serialization of an event. The hdfs sink and the file_roll sink both support the EventSerializer interface.<\/p>\n

Event Serializers are similar in function to the Layout class in log4j. There are different types of Flume event serializers. The text serializer outputs the flume event body. The avro_event serializer is used for creating an Avro representation of the event.<\/p>\n

Let us now explore the EventSerializers that ship with Flume in detail.<\/p>\n

1. Body Text Serializer<\/h3>\n

It is the default serializer that writes the body of the flume event without any modification or transformation to the output stream. It ignores the event headers. If the headers exist on the flume event, then they will be discarded<\/p>\n

The Configuration options for Body Text serializer are as follows:<\/p>\n\n\n\n\n
Property Name<\/strong><\/td>\nDefault Values<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
appendNewline<\/td>\ntrue<\/span><\/td>\nIt specifies whether a newline will be appended to each flume event at write time. The default value is true which assumes that flume events do not contain newlines, for legacy reasons.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Example for agent named agent1, sink name sk1, and channel ch1:<\/p>\n

agent1.sinks = sk1\r\nagent1.sinks.sk1.type = file_roll\r\nagent1.sinks.sk1.channel = ch1\r\nagent1.sinks.sk1.sink.directory = \/var\/log\/flume\r\nagent1.sinks.sk1.sink.serializer = text\r\nagent1.sinks.sk1.sink.serializer.appendNewline = false\r\n<\/pre>\n
2. \u201cFlume Event\u201d Avro Event Serializer<\/h3>\n
Alias: avro_event<\/p>\n
This Event Serializer serializes the Flume events into an Avro container file. It uses the same schema which is used for the Flume events in the Avro RPC mechanism. This event serializer inherits from the AbstractAvroEventSerializer class.<\/p>\n
The Configuration options for \u201cFlume Event\u201d Avro Event Serializer are as follows:<\/p>\n\n\n\n\n\n
Property Name<\/strong><\/td>\nDefault Values<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
syncIntervalBytes<\/td>\n2048000<\/span><\/td>\nIt specifies Avro sync interval, in approximate bytes.<\/span><\/td>\n<\/tr>\n
compressionCodec<\/td>\nnull<\/span><\/td>\nIt specifies the Avro compression codec. For the supported codecs, go through the Avro\u2019s CodecFactory docs.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Example for agent named agent1, sink name sk1, and channel ch1:<\/p>\n
agent1.sinks.sk1.type = hdfs\r\nagent1.sinks.sk1.channel = ch1\r\nagent1.sinks.sk1.hdfs.path = \/flume\/events\/%y-%m-%d\/%H%M\/%S\r\nagent1.sinks.sk1.serializer = avro_event\r\nagent1.sinks.sk1.serializer.compressionCodec = snappy\r\n<\/pre>\n
3. Avro Event Serializer<\/h3>\n
Alias<\/h4>\n
It doesn’t have an alias. It must be specified using the qualified class name.<\/p>\n
This event serializer also serializes Flume events into an Avro container file just like the \u201cFlume Event\u201d Avro Event Serializer. However, the record schema is configurable and can be specified either as an Apache Flume configuration property or can be passed in a flume event header.<\/p>\n
For passing the record schema as a part of the Apache Flume configuration property, use the property schemaURL.<\/p>\n
For passing the record schema in the flume event header you can choose any one of the following ways:<\/p>\n
    \n
  • Either specify the event header flume.avro.schema.literal containing the JSON-format representation of the schema<\/li>\n
  • Specify the flume.avro.schema.url with a URL where the schema may be found.<\/li>\n<\/ul>\n
    This event serializer inherits from the AbstractAvroEventSerializer class.<\/p>\n
    The Configuration options for Avro Event Serializer are as follows:<\/p>\n\n\n\n\n\n\n
    Property Name<\/strong><\/td>\nDefault Values<\/strong><\/td>\nDescription<\/strong><\/td>\n<\/tr>\n
    syncIntervalBytes<\/td>\n2048000<\/span><\/td>\nIt specifies Avro sync interval, in approximate bytes.<\/span><\/td>\n<\/tr>\n
    compressionCodec<\/td>\nnull<\/span><\/td>\nIt specifies the Avro compression codec. For the supported codecs, go through the Avro\u2019s CodecFactory docs.<\/span><\/td>\n<\/tr>\n
    schemaURL<\/td>\nnull<\/span><\/td>\nIt specifies the Avro schema URL. The Schemas specified in the header overrides this option.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Example for agent named agent1, sink name sk1, and channel ch1:<\/p>\n
    agent1.sinks.sk1.type = hdfs\r\nagent1.sinks.sk1.channel = ch1\r\nagent1.sinks.sk1.hdfs.path = \/flume\/events\/%y-%m-%d\/%H%M\/%S\r\nagent1.sinks.sk1.serializer = org.apache.flume.sink.hdfs.AvroEventSerializer$Builder\r\nagent1.sinks.sk1.serializer.compressionCodec = snappy\r\nagent1.sinks.sk1.serializer.schemaURL = hdfs:\/\/namenode\/path\/to\/schema.avsc\r\n<\/pre>\n
    Summary<\/h2>\n
    In short, we can say that the event serializer is a mechanism that converts flume events into other formats for the output. There are different types of EventSerializers that ship with Apache Flume.<\/p>\n
    The different Event Serializers are Body Text serializer, \u201cFlume Event\u201d Avro Event Serializer, and Avro Event Serializer. The article had explained all these in detail along with their configuration properties and examples.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Apache Flume is a distributed system for transferring data from external sources to the HDFS or HBase. The event serializer is a mechanism that is used for converting a flume event into another format...<\/p>\n","protected":false},"author":6,"featured_media":78403,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21],"tags":[1285,2106,4224,4226,4816],"class_list":["post-8846","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-flume","tag-avro-event-serializer","tag-body-text-serializer","tag-event-serializers-in-flume","tag-eventserializers","tag-flume-event-serializers"],"yoast_head":"\nFlume Event Serializers - Apache Flume - DataFlair<\/title>\n<meta name=\"description\" content=\"Flume Event Serializers,Types of Event Serializers in flume,Body Text Serializer,Flume Event Avro Event Serializer,Avro Event Serializer with example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Flume Event Serializers - Apache Flume - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Flume Event Serializers,Types of Event Serializers in flume,Body Text Serializer,Flume Event Avro Event Serializer,Avro Event Serializer with example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-21T08:46:35+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-09T07:38:29+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/flume-event-serializers.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Flume Event Serializers - Apache Flume - DataFlair","description":"Flume Event Serializers,Types of Event Serializers in flume,Body Text Serializer,Flume Event Avro Event Serializer,Avro Event Serializer with example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/","og_locale":"en_US","og_type":"article","og_title":"Flume Event Serializers - Apache Flume - DataFlair","og_description":"Flume Event Serializers,Types of Event Serializers in flume,Body Text Serializer,Flume Event Avro Event Serializer,Avro Event Serializer with example","og_url":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-02-21T08:46:35+00:00","article_modified_time":"2021-05-09T07:38:29+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/flume-event-serializers.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Flume Event Serializers – Apache Flume","datePublished":"2018-02-21T08:46:35+00:00","dateModified":"2021-05-09T07:38:29+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/"},"wordCount":664,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/flume-event-serializers.jpg","keywords":["Avro Event Serializer","Body Text Serializer","Event Serializers in Flume","EventSerializers","Flume Event Serializers"],"articleSection":["Flume Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/","url":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/","name":"Flume Event Serializers - Apache Flume - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/flume-event-serializers.jpg","datePublished":"2018-02-21T08:46:35+00:00","dateModified":"2021-05-09T07:38:29+00:00","description":"Flume Event Serializers,Types of Event Serializers in flume,Body Text Serializer,Flume Event Avro Event Serializer,Avro Event Serializer with example","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/flume-event-serializers\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/flume-event-serializers.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/02\/flume-event-serializers.jpg","width":802,"height":420,"caption":"flume event serializers"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/flume-event-serializers\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Flume Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/flume\/"},{"@type":"ListItem","position":3,"name":"Flume Event Serializers – Apache Flume"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8846","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=8846"}],"version-history":[{"count":3,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8846\/revisions"}],"predecessor-version":[{"id":92683,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/8846\/revisions\/92683"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/78403"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=8846"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=8846"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=8846"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}