{"id":22767,"date":"2018-08-11T07:25:31","date_gmt":"2018-08-11T07:25:31","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=22767"},"modified":"2018-08-11T07:25:31","modified_gmt":"2018-08-11T07:25:31","slug":"avro-serde","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/avro-serde\/","title":{"rendered":"Avro SerDe | Avro Serialization & Deserialization Using Parsers"},"content":{"rendered":"

In our previous Avro tutorial<\/strong>, we discussed Avro SerDe with code generation<\/strong>. Today, we will see Avro SerDe using Parsers. So, in this article, “Avro Serialization and deserialization” we will learn to read the schema by using the parsers library and also to serialize and deserialize the data using Avro. <\/span><\/p>\n

As we have also discussed in the previous article, we can read an Avro schema into a program by two ways, either with code generation or without code generation, that is by using the parsers library. <\/span><\/p>\n

However, must read our previous article, \u201cAvro SerDe by generating class\u201d because we are using the same example as in the previous article, yet without code generation.<\/span><\/p>\n

So, let’s start Serialization and Deserialization In Avro Using Parsers.<\/p>\n

Serialization and Deserialization in Avro<\/span><\/h2>\n

While it comes to transport the data over the network or to store on some persistent storage, we need to perform the process of translating data structures or objects state into the binary or textual form that process is what we call Serialization. <\/span><\/p>\n

However, a<\/span>fter the process of Serialization, we also need to deserialize it again.<\/span><\/p>\n

Basically, the data is always stored with its corresponding schema, in Avro. Hence, it is possible to read a schema even without code generation. Although there are several steps for it, they are:<\/span><\/p>\n

    \n
  • Creating users<\/span><\/li>\n
  • Serializing<\/span><\/li>\n
  • Deserializing<\/span><\/li>\n
  • Compiling and running the example code<\/span><\/li>\n<\/ul>\n

    Creating Users For Avro SerDe<\/span><\/h2>\n

    At very first, to read our schema definition and create a Schema object we are using a Parser.<\/span><\/p>\n

    Schema schema = new Schema.Parser().parse(new File(\"user.avsc\")); \u00a0 \u00a0\u00a0<\/span><\/pre>\n
    Now, let’s create some users, using this schema.<\/span><\/p>\n
    GenericRecord user1 = new GenericData.Record(schema);\nuser1.put(\"name\", \"Chandler\");\nuser1.put(\"favorite_number\", 256);\n\/\/ Leave favorite color null\nGenericRecord user2 = new GenericData.Record(schema);\nuser2.put(\"name\", \"Liza\");\nuser2.put(\"favorite_number\", 7);\nuser2.put(\"favorite_color\", \"red\")<\/pre>\n
    Basically, here we use GenericRecords to represent users, as we’re not using code generation. Moreover, to verify that we only specify valid fields, GenericRecord uses the schema. Further, we’ll get an AvroRuntimeException when we run the program, if we try to set a non-existent field (e.g., user1.put(“favorite_animal”, “cat”))<\/strong>.<\/span><\/p>\n
    Make sure that we do not set user1’s favorite color. We can either set it to a string or leave it null since that record is of type [“string”, “null”]; it is essentially optional.<\/span><\/p>\n
    Serializing in Apache Avro<\/span><\/h2>\n
    However, serializing and deserializing user objects is almost similar to the above example with code generation. But the main difference\u00a0between them is here we use generic\u00a0rather than specific readers and writers.<\/span><\/p>\n
    Now, we’ll serialize our users to a data file on disk, at first.<\/span><\/p>\n
    \/\/ Serialize user1 and user2 to disk\nFile file = new File(\"users.avro\");\nDatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema);\nDataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter);\ndataFileWriter.create(schema, file);\ndataFileWriter.append(user1);\ndataFileWriter.append(user2);\ndataFileWriter.close(); \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/pre>\n
    Similarly, here also we are creating a DatumWriter,\u00a0that use to convert Java objects into an in-memory serialized format, but here we create a GenericDatumWriter due to lack of code generation. <\/span><\/p>\n
    However, for both the process either to determine how to write the GenericRecords\u00a0or to verify that all non-nullable fields are present, it requires the Avro Schema<\/strong>.<\/span><\/p>\n
    In addition, here also we will create a DataFileWriter, basically, that writes the serialized record and the schema, to the file specified in the dataFileWriter.create call, as same as the code generation example. <\/span><\/p>\n
    Further, via calls to the dataFileWriter.append method, we write our users to the file. Also, we close the data file when we are done writing.<\/span><\/p>\n
    Deserializing in Avro SerDe Using Parser<\/span><\/h2>\n
    Now, after serializing we’ll deserialize the data file:<\/span><\/p>\n
    \/\/ Deserialize users from disk\nDatumReader<GenericRecord> datumReader = new GenericDatumReader<GenericRecord>(schema);\nDataFileReader<GenericRecord> dataFileReader = new DataFileReader<GenericRecord>(file, datumReader);\nGenericRecord user = null;\nwhile (dataFileReader.hasNext()) {\n\/\/ Reuse user object by passing it to next(). This saves us from\n\/\/ allocating and garbage collecting many objects for files with\n\/\/ many items.\nuser = dataFileReader.next(user);\nSystem.out.println(user);<\/pre>\n
    So, Output is:<\/span>
    \n{“name”: “Chandler”, “favorite_number”: 256, “favorite_color”: null}<\/b>
    \n{“name”: “Liza”, “favorite_number”: 7, “favorite_color”: “red”}<\/b>\u00a0 \u00a0 \u00a0 \u00a0<\/span><\/p>\n
    Analogous to the GenericDatumWriter we used in serialization here also we will create a GenericDatumReader, that converts in-memory serialized items into GenericRecords. <\/span><\/p>\n
    Furthermore, analogous to the DataFileWriter again we will pass the DatumReader and the previously created File to a DataFileReader,\u00a0that reads the data file on disk.<\/span><\/p>\n
    In addition, to iterate through the serialized users and print the deserialized object to stdout, we use the DataFileReader.<\/span><\/p>\n
    Now see the process of iteration we perform. As a process, we create a single GenericRecord object at first,\u00a0in which we store the current deserialized user. Further, we pass this record object to every call of dataFileReader.next. <\/span><\/p>\n
    In other words, we call it performance optimization process which permits the DataFileReader in order to reuse the same record object\u00a0instead of allocating a new GenericRecord for every iteration process. <\/span><\/p>\n
    Because if we deserialize a large data file, it can be very expensive in terms of object allocation and garbage collection. Though it’s also possible to use for (GenericRecord user: dataFileReader) if performance is not a concern while this technique is the standard way to iterate through a data file.<\/span><\/p>\n
    Compiling and Running the Example Code<\/span><\/h2>\n
    Now, to build and run the example, execute the following commands:<\/span><\/p>\n
    $ mvn compile\n$ mvn -q exec:java -Dexec.mainClass=example.GenericMain<\/pre>\n
    Hence, this was all in Serialization and Deserialization in Avro. Hope you like our explanation.
    \nSo, this was all in Avro SerDe using Parsers. Hope you like our explanation.<\/p>\n
    Conclusion: Avro SerDe Using Parsers<\/span><\/h2>\n
    Hence, we have seen the concept of Avro SerDe using Parsers in detail. Moreover, we discussed creating users for Avro SerDe. Though if any doubt occurs regarding Avro Serialization and Deserialization, feel free to ask in the comment section. Hope it helps!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
    In our previous Avro tutorial, we discussed Avro SerDe with code generation. Today, we will see Avro SerDe using Parsers. So, in this article, “Avro Serialization and deserialization” we will learn to read the...<\/p>\n","protected":false},"author":7,"featured_media":23050,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[1313,1315,12745],"class_list":["post-22767","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-avro","tag-avro-serde","tag-avro-serde-with-parser","tag-serializing-in-apache-avro"],"yoast_head":"\nAvro SerDe | Avro Serialization & Deserialization Using Parsers - DataFlair<\/title>\n<meta name=\"description\" content=\"Avro SerDe using Parsers,what is Avro SerDe,creating users,serializing & deserializing in Avro Serde.Avro Serialization & Deserialization,Compiling example\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/avro-serde\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Avro SerDe | Avro Serialization & Deserialization Using Parsers - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Avro SerDe using Parsers,what is Avro SerDe,creating users,serializing & deserializing in Avro Serde.Avro Serialization & Deserialization,Compiling example\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/avro-serde\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-08-11T07:25:31+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Avro-Serialization-Deserialization-Using-Parsers-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Avro SerDe | Avro Serialization & Deserialization Using Parsers - DataFlair","description":"Avro SerDe using Parsers,what is Avro SerDe,creating users,serializing & deserializing in Avro Serde.Avro Serialization & Deserialization,Compiling example","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/avro-serde\/","og_locale":"en_US","og_type":"article","og_title":"Avro SerDe | Avro Serialization & Deserialization Using Parsers - DataFlair","og_description":"Avro SerDe using Parsers,what is Avro SerDe,creating users,serializing & deserializing in Avro Serde.Avro Serialization & Deserialization,Compiling example","og_url":"https:\/\/data-flair.training\/blogs\/avro-serde\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-08-11T07:25:31+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Avro-Serialization-Deserialization-Using-Parsers-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"Avro SerDe | Avro Serialization & Deserialization Using Parsers","datePublished":"2018-08-11T07:25:31+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/"},"wordCount":840,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Avro-Serialization-Deserialization-Using-Parsers-01.jpg","keywords":["Avro SerDe","Avro SerDe with Parser","Serializing in Apache Avro"],"articleSection":["AVRO Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/avro-serde\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/","url":"https:\/\/data-flair.training\/blogs\/avro-serde\/","name":"Avro SerDe | Avro Serialization & Deserialization Using Parsers - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Avro-Serialization-Deserialization-Using-Parsers-01.jpg","datePublished":"2018-08-11T07:25:31+00:00","description":"Avro SerDe using Parsers,what is Avro SerDe,creating users,serializing & deserializing in Avro Serde.Avro Serialization & Deserialization,Compiling example","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/avro-serde\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Avro-Serialization-Deserialization-Using-Parsers-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Avro-Serialization-Deserialization-Using-Parsers-01.jpg","width":1200,"height":628,"caption":"Avro SerDe | Avro Serialization & Deserialization Using Parsers"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/avro-serde\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"AVRO Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/avro\/"},{"@type":"ListItem","position":3,"name":"Avro SerDe | Avro Serialization & Deserialization Using Parsers"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/22767","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=22767"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/22767\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/23050"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=22767"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=22767"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=22767"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}