{"id":21005,"date":"2018-07-23T03:00:12","date_gmt":"2018-07-23T03:00:12","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=21005"},"modified":"2021-05-19T18:24:06","modified_gmt":"2021-05-19T12:54:06","slug":"zookeeper-queues","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/","title":{"rendered":"Zookeeper Queues – Priority & Producer-Consumer Queue"},"content":{"rendered":"

In our last ZooKeeper tutorial<\/strong>, we discussed ZooKeeper CLI<\/strong>.\u00a0 Today, in this Zookeeper article, we will learn to implement queue (distributed and Priority) in ZooKeeper Queues. Also, we will show simple implementations of producer-consumer Zookeeper Queues in detail.<\/span><\/p>\n

So let’s begin Zookeeper Queues.<\/p>\n

What is Zookeeper Queues?<\/span><\/h2>\n

On defining Distributed queues, they are a common data structure. At very first designate a Znode to hold the queue, the queue node, in order to implement a distributed queue in ZooKeeper.<\/span><\/p>\n

However, by calling create() with a pathname ending in “queue-” the distributed clients put something into the queue, along with the sequence and ephemeral flags in the create() call set to true. <\/span><\/p>\n

Though it is only because the sequence flag is set. So, the form of the new pathnames will\u00a0be\u00a0 _path-to-queue-node_\/queue-X. Make sure, X\u00a0refers to monotonic increasing number here.<\/span><\/p>\n

However, if any client wants to be removed from the queue calls ZooKeeper’s getChildren( ) function, then\u00a0it set to true on the queue node, and further\u00a0starts processing nodes with the lowest number. <\/span><\/p>\n

Until\u00a0the client exhausts the list obtained from the first getChildren( ) call, it does not need to issue another getChildren( ). So, the ZooKeeper reader waits for a watch notification to check the queue again if there are are no children in the queue node.<\/span><\/p>\n

Priority Queues: Zookeeper Queues<\/span><\/h2>\n

Further, we only need to make two simple changes to the generic ZooKeeper queues recipe, to implement a priority queue. <\/span><\/p>\n

    \n
  • The first change is, the pathname must ends with “queue-YY” where YY is the priority of the element with lower numbers representing higher priority (as same as UNIX), in order to add to a queue.<\/span><\/li>\n
  • And the second change is, as a client uses an up-to-date children list meaning while removing from the queue,\u00a0 if a watch notification triggers for the queue node, that the client will invalidate previously obtained children lists.<\/span><\/li>\n<\/ul>\n

    Producer-Consumer Queues<\/span><\/h2>\n

    A\u00a0distributed data structure\u00a0which we generally use to generate and consume items is what we call producer-consumer queues in ZooKeeper. Basically, at first Producer processes create new elements and then further it adds them to the queue. <\/span><\/p>\n

    Whereas, the Consumer processes remove elements from the list and then further process them. However, the elements are simple integers, in this implementation.<\/span><\/p>\n

    Here, by a root node a queue is represented, and also a producer process creates a new node, a child of the root node, in order to add an element to the queue.<\/span><\/p>\n

    Below is an excerpt of code corresponds to the constructor of the object. Though, first, it calls the constructor of the parent class,!\u00a0SyncPrimitive. So, if one doesn’t exist, that creates a!ZooKeeper object.\u00a0Also, it checks if the root node of the queue exists, and then creates one if in case it doesn’t.<\/span>\u00a0 \u00a0<\/b><\/p>\n

    \/**\r\n******* Constructor of producer-consumer queue\r\n*******\r\n******* @param address\r\n******* @param name\r\n******* \/\r\n       Queue(String address, String name)\r\n       throws KeeperException, InterruptedException {\r\n           super(address);\r\n           this.root = name;\r\n           \/\/ Create ZK node name\r\n           if (zk != null) {\r\n                   Stat s = zk.exists(root, false);\r\n                   if (s == null) {\r\n                       zk.create(root, new byte[0], Ids.OPEN_ACL_UNSAFE, 0);\r\n                   }\r\n           }\r\n       }<\/pre>\n
    Furthermore, to add an element to the queue, a producer process calls “produce()”, and then also passes an integer as an argument. <\/span><\/p>\n
    And, the method creates a new node using “create()”, to add an element to the queue, and further to instruct!ZooKeeper to append the value of the sequencer counter associated to the root node, it uses the SEQUENCE flag. <\/span><\/p>\n
    Thus guaranteeing that the oldest element of the queue is the next one consumed, while, we impose a total order on the elements of the queue, in this way.<\/span><\/p>\n
       \/**\r\n******* Add element to the queue.\r\n*******\r\n******* @param i\r\n******* @return\r\n******* \/\r\n       boolean produce(int i) throws KeeperException, InterruptedException{\r\n           ByteBuffer b = ByteBuffer.allocate(4);\r\n           byte[] value;\r\n           \/\/ Add child with value i\r\n           b.putInt(i);\r\n           value = b.array();\r\n           zk.create(root + \"\/element\", value, Ids.OPEN_ACL_UNSAFE,\r\n                      CreateFlags.SEQUENCE);\r\n           return true;\r\n       }<\/pre>\n
    However, in order to consume an element, a consumer process first obtains the children of the root node, after that it reads the node with smallest counter value, and also returns the element.\u00a0<\/span><\/p>\n
    In addition, when we make a call to getChildren(). we get that list of children in lexicographic order. However, make sure the lexicographic order does not necessarily follow the numerical order of the counter values. <\/span><\/p>\n
    Hence we have to decide that which element is the smallest here. So, we traverse the list and remove the prefix “element” from each one, to decide which one has the smallest counter value.<\/span><\/p>\n
        \/**\r\n******* Remove first element from the queue.\r\n*******\r\n******* @return\r\n******* @throws KeeperException\r\n******* @throws InterruptedException\r\n******* \/\r\n       int consume() throws KeeperException, InterruptedException\r\n           int retvalue = -1;\r\n           Stat stat = null;\r\n           \/\/ Get the first element available\r\n           while (true) {\r\n               synchronized (mutex) {\r\n                   ArrayList<String> list = zk.getChildren(root, true);\r\n                   if (list.isEmpty()) {\r\n                       System.out.println(\"Going to wait\");\r\n                       mutex.wait();\r\n                   } else {\r\n                       Integer min = new Integer(list.get(0).substring(7));\r\n                       for(String s : list){\r\n                           Integer tempValue = new Integer(s.substring(7));\r\n                           if(tempValue < min) min = tempValue;\r\n                       }\r\n                       System.out.println(\"Temporary value: \" + root + \"\/element\" + min);\r\n                       byte[] b = zk.getData(root + \"\/element\" + min, false, stat);\r\n                       zk.delete(root + \"\/element\" + min, 0);\r\n                       ByteBuffer buffer = ByteBuffer.wrap(b);\r\n                       retvalue = buffer.getInt();\r\n                       return retvalue;\r\n                   }\r\n               }\r\n           }\r\n       }<\/pre>\n
    So, this was all in ZooKeeper Queues. Hope you like our explanation.<\/p>\n
    Conclusion<\/span><\/h2>\n
    Hence, we have learned Zookeeper Queues in detail. Moreover, we discussed Producer-consumer ZooKeeper Queues. Furthermore, if any doubt occurs regarding Queue in ZooKeeper, feel free to ask in the comment section. We are happy to help!<\/p>\n","protected":false},"excerpt":{"rendered":"
    In our last ZooKeeper tutorial, we discussed ZooKeeper CLI.\u00a0 Today, in this Zookeeper article, we will learn to implement queue (distributed and Priority) in ZooKeeper Queues. Also, we will show simple implementations of producer-consumer...<\/p>\n","protected":false},"author":7,"featured_media":21067,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[77],"tags":[3998,9973,10163,11119,16047,16408],"class_list":["post-21005","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-zookeeper","tag-distributed-queues","tag-priority-queues","tag-producer-consumer-queues","tag-queue-in-zookeeper","tag-what-is-zookeeper-queue","tag-zookeeper-queues"],"yoast_head":"\nZookeeper Queues - Priority & Producer-Consumer Queue - DataFlair<\/title>\n<meta name=\"description\" content=\"In this Zookeeper article, we will learn to implement Zookeeper Queues (distributed and Priority) along with implementations of producer-consumer queues.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Zookeeper Queues - Priority & Producer-Consumer Queue - DataFlair\" \/>\n<meta property=\"og:description\" content=\"In this Zookeeper article, we will learn to implement Zookeeper Queues (distributed and Priority) along with implementations of producer-consumer queues.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-23T03:00:12+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-19T12:54:06+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Zookeeper-Queues-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Zookeeper Queues - Priority & Producer-Consumer Queue - DataFlair","description":"In this Zookeeper article, we will learn to implement Zookeeper Queues (distributed and Priority) along with implementations of producer-consumer queues.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/","og_locale":"en_US","og_type":"article","og_title":"Zookeeper Queues - Priority & Producer-Consumer Queue - DataFlair","og_description":"In this Zookeeper article, we will learn to implement Zookeeper Queues (distributed and Priority) along with implementations of producer-consumer queues.","og_url":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-07-23T03:00:12+00:00","article_modified_time":"2021-05-19T12:54:06+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Zookeeper-Queues-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd"},"headline":"Zookeeper Queues – Priority & Producer-Consumer Queue","datePublished":"2018-07-23T03:00:12+00:00","dateModified":"2021-05-19T12:54:06+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/"},"wordCount":716,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Zookeeper-Queues-01.jpg","keywords":["Distributed queues","Priority Queues","Producer-Consumer Queues","Queue in ZooKeeper","what is ZooKeeper Queue","Zookeeper Queues"],"articleSection":["Zookeeper Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/","url":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/","name":"Zookeeper Queues - Priority & Producer-Consumer Queue - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Zookeeper-Queues-01.jpg","datePublished":"2018-07-23T03:00:12+00:00","dateModified":"2021-05-19T12:54:06+00:00","description":"In this Zookeeper article, we will learn to implement Zookeeper Queues (distributed and Priority) along with implementations of producer-consumer queues.","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/zookeeper-queues\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Zookeeper-Queues-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/Zookeeper-Queues-01.jpg","width":1200,"height":628,"caption":"Zookeeper Queues - Priority & Producer-Consumer Queue"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/zookeeper-queues\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Zookeeper Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/zookeeper\/"},{"@type":"ListItem","position":3,"name":"Zookeeper Queues – Priority & Producer-Consumer Queue"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/beb0cab24b7aa54423a3b50e669a9dcd","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c322416204232f4dd97ef3901b0a499a5d34d7ba7fe333f4bfe53a907873d293?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"DataFlair Team specializes in creating clear, actionable content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Backed by industry expertise, we make learning easy and career-oriented for beginners and pros alike.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam3\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/21005","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=21005"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/21005\/revisions"}],"predecessor-version":[{"id":94228,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/21005\/revisions\/94228"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/21067"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=21005"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=21005"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=21005"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}