{"id":22681,"date":"2018-08-10T08:05:08","date_gmt":"2018-08-10T08:05:08","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=22681"},"modified":"2018-08-10T08:05:08","modified_gmt":"2018-08-10T08:05:08","slug":"avro-serialization-and-deserialization","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/","title":{"rendered":"AVRO Serialization and Deserialization: With Code Generation"},"content":{"rendered":"

Today in this Avro Tutorial<\/strong>, we will learn Avro Serialization and Deserialization with Code Generations. Moreover, we will see defining and compiling Avro Schema. Also, we will see Serializing and Deserializing Avro. <\/span><\/p>\n

There are two possible ways to read an Avro schema into the program, one is by generating a class\/code generation corresponding to a schema or another one is by using the parsers library.”. Also, we will see to\u00a0Deserialize the data by using Avro in detail.<\/span><\/p>\n

So, let’s start Avro Serialization and Deserialization with Code Generation.<\/p>\n

Avro Serialization and Deserialization<\/span><\/h2>\n

In order to transport the data over the network or to store on some persistent storage. There is a process of translating data structures or objects state into binary or textual form, is what we call Serialization process. <\/span><\/p>\n

However, we need to deserialize the data as soon as the data is transported over the network or retrieved from the persistent storage.<\/span><\/p>\n

So, in order to serialize and deserialize the data by using Avro, the steps are \u2212<\/span><\/p>\n

    \n
  • Define an Avro schema.<\/span><\/li>\n
  • Compile the schema using Avro utility.<\/span><\/li>\n
  • Creating Users.<\/span><\/li>\n
  • Serialize it using Avro library.<\/span><\/li>\n
  • Deserializing with code generation.<\/li>\n
  • Compiling and running.<\/li>\n<\/ul>\n

    Now, let\u2019s learn Avro Serialization and Deserialization steps in detail.<\/span><\/p>\n

    Defining an Avro Schema<\/span><\/h2>\n

    Basically, by using JSON, Avro schemas are defined. Generally, these Schemas are composed of primitive types as well as complex types.<\/span><\/p>\n

    However, primitive types are of several types such as null, boolean, int, long, float, double, bytes, and string whereas complex types are of various types such as record, enum, array, map, union, and fixed. <\/span><\/p>\n

    So, let’s start with a simple schema example, user.avsc:<\/span><\/p>\n

    {\"namespace\": \"example1.avro\",\n\"type\": \"record\",\n\"name\": \"User\",\n\"fields\": [\n    {\"name\": \"name\", \"type\": \"string\"},\n    {\"name\": \"favorite_number\",  \"type\": [\"int\", \"null\"]},\n    {\"name\": \"favorite_color\", \"type\": [\"string\", \"null\"]}\n]\n} \u00a0 \u00a0<\/span><\/pre>\n
    However, by representing a hypothetical user, this schema defines a record. Although, make sure that a schema file can only contain a single schema definition. In general, it is must that a record definition contain its type (“type”: “record”), a name (“name”: “User”), and fields.<\/span><\/p>\n
     Though. In this case, the name is favorite_number, and favorite_color. In addition, we also define a namespace (“namespace”: “example1.avro”), along with the name attribute. So that attribute defines the “full name” of the schema (example1.avro.User in this case).<\/span><\/p>\n
    Furthermore, via an array of objects, fields are defined, each of which defines a name and type. Moreover, another schema object is the type attribute of a field, though it can be of any type either a primitive or a complex type. <\/span><\/p>\n
    As an example, our user schema name field is the primitive type string, and both the fields \u00a0favorite_number and favorite_color are unions, which are represented by JSON arrays. As we have seen earlier that unions are a complex type so it can be any of the types listed in the array. <\/span><\/p>\n
    That says favorite_number can be an int or null, essentially making it an optional field.<\/span><\/p>\n
    Compiling the Avro Schema<\/span><\/h2>\n
    In order to automatically create classes on the basis of our previously-defined schema, code generation permits us. Although, there is no need to use the schema directly in our programs, once we have defined the relevant classes. So, to generate code, we use the avro-tools jar as follows:<\/span><\/p>\n
    java -jar \/path\/to\/avro-tools-1.8.2.jar compile schema <schema file> <destination>\u00a0 \u00a0 \u00a0\u00a0<\/span><\/pre>\n
    On the basis of schema’s namespace in the provided destination folder, this will generate the appropriate source files in a package. For example<\/em>, we can generate a user class in package example1.avro from the schema defined above, run<\/span><\/p>\n
    java -jar \/path\/to\/avro-tools-1.8.2.jar compile schema user.avsc . \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/pre>\n
    Make sure there is no need to manually invoke the schema compiler if we are using the Avro Maven plugin; So on any .avsc files which are present in the configured source directory, the plugin automatically performs code generation.<\/span><\/p>\n
    Creating Users For Avro Serialization<\/span><\/h2>\n
    Further, create some Users and then serialize them to a data file on disk. Further, read back the file and then deserialize the User objects since we’ve completed the code generation.<\/span>
    \nAt very first let’s create some Users and also set their fields.<\/span><\/p>\n
    User user1 = new User();\nuser1.setName(\"Chandler\");\nuser1.setFavoriteNumber(256);\n\/\/ Leave favorite color null\n\/\/ Alternate constructor\nUser user2 = new User(\"Liza\", 7, \"red\");\n\/\/ Construct via builder\nUser user3 = User.newBuilder()\n            .setName(\"Ross\")\n            .setFavoriteColor(\"blue\")\n            .setFavoriteNumber(null)\n            .build(); \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/pre>\n
    As you can see, either by using a builder or by invoking a constructor directly, we can create Avro objects<\/em>. Dissimilar to constructors, builders will automatically set any default values specified in the schema. <\/span><\/p>\n
    In addition, objects constructed directly will not cause an error until the object is serialized since the builders validate the data as it set. Although, as builders create a copy of the data structure before it is written, using constructors directly generally offers better performance.<\/span><\/p>\n
    However, make sure that we do not set user1’s favorite color. As in that record is of type [“string”, “null”], hence either we can set it to a string or leave it null; though it is essentially optional. Similarly, by using a builder requires setting all fields, even if they are null, we set user3’s favorite number to null.<\/span><\/p>\n
    Serializing in Apache Avro<\/span><\/h2>\n
    Further, let’s serialize our Users to disk.<\/span><\/p>\n
    \/\/ Serialize user1, user2 and user3 to disk\nDatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class);\nDataFileWriter<User> dataFileWriter = new DataFileWriter<User>(userDatumWriter);\ndataFileWriter.create(user1.getSchema(), new File(\"users.avro\"));\ndataFileWriter.append(user1);\ndataFileWriter.append(user2);\ndataFileWriter.append(user3);\ndataFileWriter.close(); \u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/pre>\n
    To convert Java objects<\/strong> into an in-memory serialized format, here we create a DatumWriter. Though, with generated classes and extracts the schema from the specified generated type, we use the SpecificDatumWriter class.<\/span><\/p>\n
    Afterward, to write the serialized records, as well as the schema, to the file specified in the dataFileWriter, we create a DataFileWriter.create call. Also, via calls to the dataFileWriter.append method, we write our users to the file. Hence, we close the data file, when we are done writing.<\/span><\/p>\n
    Deserializing With Code Generation<\/span><\/h2>\n
    Now, after serialization, we will read the schema by generating a class and also learn to Deserialize the data by using Avro.<\/span><\/p>\n
    Now, let’s deserialize the data file:<\/span><\/p>\n
    \/\/ Deserialize Users from disk\nDatumReader<User> userDatumReader = new SpecificDatumReader<User>(User.class);\nDataFileReader<User> dataFileReader = new DataFileReader<User>(file, userDatumReader);\nUser user = null;\nwhile (dataFileReader.hasNext()) {\n\/\/ Reuse user object by passing it to next(). This saves us from\n\/\/ allocating and garbage collecting many objects for files with\n\/\/ many items.\nuser = dataFileReader.next(user);\nSystem.out.println(user);\n} \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span><\/pre>\n
    Output:<\/strong>
    \n{“name”: “Chandler”, “favorite_number”: 256, “favorite_color”: null}<\/b>
    \n{“name”: “Liza”, “favorite_number”: 7, “favorite_color”: “red”}<\/b>
    \n{“name”: “Ross”, “favorite_number”: null, “favorite_color”: “blue”}<\/b>\u00a0 \u00a0 \u00a0\u00a0<\/span><\/p>\n
    Although, we can say Avro Serialization and Deserialization are very similar. Now as same as SpecificDatumWriter in Serialization. Here we create a SpecificDatumReader, basically, that converts in-memory serialized items into instances of our generated class, in this case, User. We pass the DatumReader and the previously created File to a DataFileReader, as same as DataFileWriter in serialization and that reads the data file on disk.<\/span><\/p>\n
    Further, to iterate through the serialized Users and print the deserialized object to stdout, we use the DataFileReader. Now see the process of iteration we perform. As a process we create a single User object at first, further we store the current deserialized user in that object.\u00a0Afterward, we pass this record object to every call of dataFileReader.next.<\/span><\/p>\n
    Also,\u00a0we call it performance optimization. Basically, it permits the DataFileReader to reuse the same User object instead of allocating a new User for every iteration. It is because in terms of object allocation and garbage collection<\/em> that can be very expensive when we deserialize a large data file. Hence we can say it is the standard way to iterate through a data file so if performance is not a concern it’s also possible to use for (User user: dataFileReader).<\/span><\/p>\n
    Compiling and Running the Example Code<\/span><\/h2>\n
    Further, to build and run the example, execute the following commands:<\/span><\/p>\n
    $ mvn compile # includes code generation via Avro Maven plugin\n$ mvn -q exec:java -Dexec.mainClass=example1.SpecificMain<\/pre>\n
    So, this was all in Avro Serialization and Deserialization. Hope you like our explanation.<\/p>\n
    Conclusion: AVRO Serialization and Deserialization\u00a0<\/span><\/h2>\n
    Hence, we have seen the whole concept of AVRO Serialization and Deserialization with code generation. So if any doubt occurs regarding\u00a0AVRO Serialization and Deserialization, feel free to ask in the comment tab. <\/span><\/p>\n
    And, in our next tutorial, we will see how to perform AVRO Serialization and Deserialization without code generation. Keep visiting, keep learning!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
    Today in this Avro Tutorial, we will learn Avro Serialization and Deserialization with Code Generations. Moreover, we will see defining and compiling Avro Schema. Also, we will see Serializing and Deserializing Avro. There are...<\/p>\n","protected":false},"author":7,"featured_media":23023,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[1313,1314,2792,3728,3784,12745],"class_list":["post-22681","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-avro","tag-avro-serde","tag-avro-serde-with-code-generation","tag-compiling-the-avro-schema","tag-defining-an-avro-schema","tag-deserializing-with-code-generation","tag-serializing-in-apache-avro"],"yoast_head":"\nAVRO Serialization and Deserialization: With Code Generation - DataFlair<\/title>\n<meta name=\"description\" content=\"Avro Serialization and Deserialization with code generation.Serializing & Deserializing in Apache Avro,Compiling the Avro Schema,defining Avro Schema\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AVRO Serialization and Deserialization: With Code Generation - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Avro Serialization and Deserialization with code generation.Serializing & Deserializing in Apache Avro,Compiling the Avro Schema,defining Avro Schema\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-08-10T08:05:08+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/AVRO-Serialization-and-Deserialization-With-Code-Generation-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"AVRO Serialization and Deserialization: With Code Generation - DataFlair","description":"Avro Serialization and Deserialization with code generation.Serializing & Deserializing in Apache Avro,Compiling the Avro Schema,defining Avro Schema","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/","og_locale":"en_US","og_type":"article","og_title":"AVRO Serialization and Deserialization: With Code Generation - DataFlair","og_description":"Avro Serialization and Deserialization with code generation.Serializing & Deserializing in Apache Avro,Compiling the Avro Schema,defining Avro Schema","og_url":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-08-10T08:05:08+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/AVRO-Serialization-and-Deserialization-With-Code-Generation-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"AVRO Serialization and Deserialization: With Code Generation","datePublished":"2018-08-10T08:05:08+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/"},"wordCount":1248,"commentCount":2,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/AVRO-Serialization-and-Deserialization-With-Code-Generation-01.jpg","keywords":["Avro SerDe","Avro SerDe With Code Generation","Compiling the Avro Schema","Defining an Avro Schema","Deserializing With Code Generation","Serializing in Apache Avro"],"articleSection":["AVRO Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/","url":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/","name":"AVRO Serialization and Deserialization: With Code Generation - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/AVRO-Serialization-and-Deserialization-With-Code-Generation-01.jpg","datePublished":"2018-08-10T08:05:08+00:00","description":"Avro Serialization and Deserialization with code generation.Serializing & Deserializing in Apache Avro,Compiling the Avro Schema,defining Avro Schema","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/AVRO-Serialization-and-Deserialization-With-Code-Generation-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/AVRO-Serialization-and-Deserialization-With-Code-Generation-01.jpg","width":1200,"height":628,"caption":"AVRO Serialization and Deserialization: With Code Generation"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/avro-serialization-and-deserialization\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Site SAML","item":"https:\/\/data-flair.training\/blogs\/tag\/site-saml\/"},{"@type":"ListItem","position":3,"name":"AVRO Serialization and Deserialization: With Code Generation"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/22681","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=22681"}],"version-history":[{"count":0,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/22681\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/23023"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=22681"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=22681"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=22681"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}