{"id":14869,"date":"2018-05-23T05:28:20","date_gmt":"2018-05-23T05:28:20","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=14869"},"modified":"2021-05-14T11:00:17","modified_gmt":"2021-05-14T05:30:17","slug":"distributed-tensorflow","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/","title":{"rendered":"Distributed TensorFlow | TensorFlow Clustering"},"content":{"rendered":"

By now, you\u2019ve seen what TensorFlow<\/strong> is capable of and how to get it up and running in your system. Today, we\u2019ll be looking at how to make a cluster of TensorFlow servers and distributed TensorFlow in our computation (graph) over those clusters. <\/span><\/p>\n

Moreover, we will see how to define a cluster, assigning model for distributed TensorFlow. Along with this, we will discuss the training methods and training session for distributed TensorFlow.<\/span><\/p>\n

So, let\u2019s start Distributed TensorFlow and TensorFlow Clustering.<\/span><\/p>\n

Distributed TensorFlow and TensorFlow Clustering<\/span><\/h2>\n

TensorFlow Clusters are nothing but individual tasks that participate in the complete execution of a graph. <\/span><\/p>\n

A server contains a master that is used to create sessions and there is a worker that operates on the respective graphs and every task is associated with a server. TensorFlow cluster can also be fragmented into jobs where each job contains more than one task.<\/span><\/p>\n

\"Working<\/a>

Working Model of Distributed TensorFlow<\/p><\/div>\n

To see a simple example<\/strong>, you can start off by creating a single process cluster as shown below:<\/span><\/p>\n

# Start a TensorFlow server as a single-process \"cluster\".\r\n$ python\r\n>>> import tensorflow as tf\r\n>>> c = tf.constant(\"Hello, distributed TensorFlow!\")\r\n>>> server = tf.train.Server.create_local_server()\r\n>>> sess = tf.Session(server.target)  # Create a session on the server.\r\n>>> sess.run(c)\r\n'Hello, distributed TensorFlow!'<\/pre>\n
Define TensorFlow Cluster<\/span><\/h2>\n
You start a cluster in tensorFlow, by creating a server per task which runs on each machine but you can run multiple tasks on the same machine.<\/span><\/p>\n
Firstly, you create a TensorFlow cluster specification (master) using <\/span>tf.train.ClusterSpec<\/b>. Now, you create a server<\/span> tf.train.Server, <\/b>which passes the specification to the constructor, identifying the tasks with a job and an index.<\/span><\/p>\n
# In task 0:<\/strong>
\n<\/span><\/p>\n
cluster = tf.train.ClusterSpec({\"local\": [\"localhost:2222\", \"localhost:2223\"]})\r\nserver = tf.train.Server(cluster, job_name=\"local\", task_index=0)<\/pre>\n
# In task 1:<\/strong>
\n<\/span><\/p>\n
cluster = tf.train.ClusterSpec({\"local\": [\"localhost:2222\", \"localhost:2223\"]})\r\nserver = tf.train.Server(cluster, job_name=\"local\", task_index=1)<\/pre>\n
Assigning Model Variables and Ops the Worker<\/span><\/h2>\n
We do that using <\/span>with tf.device <\/b>that gets assigned to a specific task of a specific job. Nodes left out of this block are automatically assigned to a device by Tensorflow. The device function is also used to distinguish whether the operations run on a CPU or a GPU<\/span>.<\/span><\/p>\n
with tf.device(\"\/job:ps\/task:0\"):\r\n  weights_1 = tf.Variable(...)\r\n  biases_1 = tf.Variable(...)\r\nwith tf.device(\"\/job:ps\/task:1\"):\r\n  weights_2 = tf.Variable(...)\r\n  biases_2 = tf.Variable(...)\r\nwith tf.device(\"\/job:worker\/task:7\"):\r\n  input, labels = ...\r\n  layer_1 = tf.nn.relu(tf.matmul(input, weights_1) + biases_1)\r\n  logits = tf.nn.relu(tf.matmul(layer_1, weights_2) + biases_2)\r\n  # ...\r\n  train_op = ...\r\nwith tf.Session(\"grpc:\/\/worker7.example.com:2222\") as sess:\r\n  for _ in range(10000):\r\n    sess.run(train_op)<\/pre>\n
Training<\/span><\/h2>\n
Here, we use the concept of data parallelism wherein the entire graph is settled on a single machine called as the <\/span>parameter server <\/b>or <\/span>ps<\/b>,<\/span> while the training operation occurs on multiple machines called as <\/span>workers<\/b>. Now, we can specify the structure using one of the following<\/span> methods:<\/span><\/p>\n
\"Distributed<\/a>
Distributed TensorFlow- Training<\/p><\/div>\n
a. In-Graph Replication<\/span><\/h3>\n
Here, the client builds a single <\/span>tf.Graph <\/b>, containing one set of parameters (variables attached to \/job:ps)<\/span>. <\/span><\/p>\n
b. Between-Graph Replication<\/span><\/h3>\n
There is a separate client for each \/job:worker task, typically in the same process as the worker<\/span>. <\/span><\/p>\n
c. Asynchronous Training<\/span><\/h3>\n
Here, each copy of the graph has an exclusive training loop that executes without coordination and is compatible with both forms of replication given above<\/span>.<\/span><\/p>\n
d. Synchronous Training<\/span><\/h3>\n
Here, all the copies read the same values for the current parameters, compute gradients in parallel, and then apply them together<\/span>.<\/span><\/p>\n
Monitored Training Session<\/span><\/h2>\n
Similar to tf.Session, it handles session initialization and restoring from a checkpoint, saving and closing when completed or when an error occurs. It sets up a master node to initialize the graph, does model checkpointing, exports TensorBoard summaries and starts\/stops the sessions. For example<\/p>\n
with tf.train.MonitoredTrainingSession(master=server.target,is_chief=(FLAGS.task_index == 0),checkpoint_dir=\"\/tmp\/train_logs\",hooks=hooks) as mon_sess:<\/pre>\n
Training Steps For Distributed TensorFlow<\/span><\/h2>\n
Rather than the traditional method of calling a while loop for every <\/span>with tf.session <\/b>block, and running the sessions for every iteration, in the monitored training session, you terminate all the instances properly and sync it with saved checkpoints.<\/span><\/p>\n
Now, you can pass a hook with <\/span>tf.train.StopAtStepHook<\/b> to the session object. It defines the last step, after which the <\/span>ps <\/b>and<\/span> workers<\/b> will shut down. You can try it with<\/span>
\n# The StopAtStepHook handles stopping after running given steps.<\/strong>\u00a0<\/span><\/p>\n
hooks=[tf.train.StopAtStepHook(last_step=1000000)]<\/pre>\n
The big picture is similar to what\u2019s been shown belo<\/span>w.<\/span><\/p>\n
\"Distributed<\/a>
Distributed TensorFlow- Training Steps<\/p><\/div>\n
Joining the Dots<\/span><\/h2>\n
Next, we run a training step asynchronously and see <\/span>tf.train.SyncReplicasOptimizer<\/b> for additional details on how to perform the synchronous training. <\/span>mon_sess.run<\/b> handles AbortedError in case of preempted <\/span>ps<\/b>. We can use the following code:<\/span><\/p>\n
while not mon_sess.should_stop():\r\nmon_sess.run(train_op)\r\nif __name__ == \"__main__\":\r\n  parser = argparse.ArgumentParser()\r\n  parser.register(\"type\", \"bool\", lambda v: v.lower() == \"true\")\r\n  # Flags for defining the tf.train.ClusterSpec\r\n  parser.add_argument(\r\n      \"--ps_hosts\",\r\n      type=str,\r\n      default=\"\",\r\n      help=\"Comma-separated list of hostname:port pairs\"\r\n  )\r\n  parser.add_argument(\r\n      \"--worker_hosts\",\r\n      type=str,\r\n      default=\"\",\r\n      help=\"Comma-separated list of hostname:port pairs\"\r\n  )\r\n  parser.add_argument(\r\n      \"--job_name\",\r\n      type=str,\r\n      default=\"\",\r\n      help=\"One of 'ps', 'worker'\"\r\n  )\r\n  # Flags for defining the tf.train.Server\r\n  parser.add_argument(\r\n      \"--task_index\",\r\n      type=int,\r\n      default=0,\r\n      help=\"Index of task within the job\"\r\n  )\r\n  FLAGS, unparsed = parser.parse_known_args()\r\n  tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)<\/pre>\n
Thus, you have successfully built a TensorFlow distributed trainer program which implements between-graph replication and asynchronous training.<\/p>\n
So, this was all about Distributed TensorFlow. Hope you like our explanation of Cluster in TensorFlow.<\/p>\n
Conclusion<\/span><\/h2>\n
Hence, you saw how to run distributed TensorFlow by making clusters and assigning the job to different workers using a master. <\/span><\/p>\n
It can be very useful while implementing neural networks and other machine learning<\/strong> algorithms where you might encounter a huge number of parameters and there is no other workaround than to distribute the task among different systems and make them work in parallel to save the computational time. <\/span><\/p>\n
In the next tutorial, we will be learning TensorFlow APIs<\/strong>. Furthermore, if you have any query, feel free to ask through the comment section.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
By now, you\u2019ve seen what TensorFlow is capable of and how to get it up and running in your system. Today, we\u2019ll be looking at how to make a cluster of TensorFlow servers and...<\/p>\n","protected":false},"author":6,"featured_media":14874,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[73],"tags":[1211,1896,2594,4000,4001,6661,8860,14030,14519,14528,14881,14882],"class_list":["post-14869","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tensorflow","tag-asynchronous-training","tag-between-graph-training","tag-cluster","tag-distributed-tensorflow","tag-distributed-tensorflow-tutorial","tag-in-graph-training","tag-monitored-training-session","tag-synchronous-training","tag-tensorflow","tag-tensorflow-clustering","tag-training-in-tensorflow","tag-training-steps"],"yoast_head":"\nDistributed TensorFlow | TensorFlow Clustering - DataFlair<\/title>\n<meta name=\"description\" content=\"Distributed tensorflow,tensorflow clustreing,Define Cluster,Training:Ingraph,between graph replication,Asynchronous and synchronous Training,Training steps\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Distributed TensorFlow | TensorFlow Clustering - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Distributed tensorflow,tensorflow clustreing,Define Cluster,Training:Ingraph,between graph replication,Asynchronous and synchronous Training,Training steps\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-05-23T05:28:20+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-14T05:30:17+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/TensorFlow-Cluster-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Distributed TensorFlow | TensorFlow Clustering - DataFlair","description":"Distributed tensorflow,tensorflow clustreing,Define Cluster,Training:Ingraph,between graph replication,Asynchronous and synchronous Training,Training steps","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/","og_locale":"en_US","og_type":"article","og_title":"Distributed TensorFlow | TensorFlow Clustering - DataFlair","og_description":"Distributed tensorflow,tensorflow clustreing,Define Cluster,Training:Ingraph,between graph replication,Asynchronous and synchronous Training,Training steps","og_url":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-05-23T05:28:20+00:00","article_modified_time":"2021-05-14T05:30:17+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/TensorFlow-Cluster-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Distributed TensorFlow | TensorFlow Clustering","datePublished":"2018-05-23T05:28:20+00:00","dateModified":"2021-05-14T05:30:17+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/"},"wordCount":790,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/TensorFlow-Cluster-01.jpg","keywords":["Asynchronous Training","Between-graph training","cluster","Distributed TensorFlow","distributed TensorFlow Tutorial","In-graph training","Monitored Training Session","synchronous Training","TensorFlow","TensorFlow Clustering","Training in tensorflow","Training Steps"],"articleSection":["Tensorflow Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/","url":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/","name":"Distributed TensorFlow | TensorFlow Clustering - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/TensorFlow-Cluster-01.jpg","datePublished":"2018-05-23T05:28:20+00:00","dateModified":"2021-05-14T05:30:17+00:00","description":"Distributed tensorflow,tensorflow clustreing,Define Cluster,Training:Ingraph,between graph replication,Asynchronous and synchronous Training,Training steps","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/TensorFlow-Cluster-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/05\/TensorFlow-Cluster-01.jpg","width":1200,"height":628,"caption":"Distributed TensorFlow -TensorFlow Clustering"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/distributed-tensorflow\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Tensorflow Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/tensorflow\/"},{"@type":"ListItem","position":3,"name":"Distributed TensorFlow | TensorFlow Clustering"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14869","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=14869"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14869\/revisions"}],"predecessor-version":[{"id":94968,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/14869\/revisions\/94968"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/14874"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=14869"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=14869"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=14869"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}