{"id":5946,"date":"2018-01-17T09:16:25","date_gmt":"2018-01-17T09:16:25","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=5946"},"modified":"2019-11-08T18:04:50","modified_gmt":"2019-11-08T12:34:50","slug":"spark-executor","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/spark-executor\/","title":{"rendered":"Apache Spark Executor for Executing Spark Tasks"},"content":{"rendered":"

1. Objective<\/h2>\n

In Apache Spark<\/strong><\/a>, some distributed agent is responsible for executing tasks, this agent is what we call Spark Executor. This document aims the whole concept of Apache Spark Executor. Also, we will see the method to create executor instance in Spark. To learn in depth, we will also see the launch task method in Spark Executor.<\/p>\n

\"Apache<\/a>

Apache Spark Executor for Executing Spark Tasks<\/p><\/div>\n

You must test your Spark Learning<\/a><\/strong><\/p>\n

2. What is Spark Executor<\/h2>\n

Basically, we can say Executors in Spark are worker nodes<\/em>. Those help to process in charge of running individual tasks in a given Spark job. Moreover, we launch them at the\u00a0start of a Spark application. Then it typically runs for the entire lifetime of an application.\u00a0As soon as they have run the task, sends results to the driver. Executors also provide in-memory storage for Spark RDDs<\/a> that are cached by user programs through Block Manager. <\/span>
\nIn addition, for the complete lifespan of a spark application, it runs. That infers the static allocation of Spark executor. However, we can also prefer for dynamic allocation.<\/span>
\nMoreover, with the help of Heartbeat Sender Thread, it sends metrics and heartbeats. One of the advantage we can have as many executors in Spark as data nodes. Moreover also possible to have as many cores as you can get from the cluster. The other way to describe Apache Spark Executor is either by their id, hostname, environment (as SparkEnv), or classpath.
\nThe most important point to note is Executor backends exclusively manage Executor in Spark.<\/p>\n

Have a look at top Spark Certifications<\/a><\/strong><\/p>\n

\"heartbeat<\/a>

Heartbeat Receiver’s Heartbeat Message Handler – Spark Executor<\/p><\/div>\n

3. Conditions to Create Spark Executor<\/h2>\n

Some conditions in which we create Executor in Spark is:<\/p>\n

    \n
  1. When CoarseGrainedExecutorBackend receives RegisteredExecutor message. Only for Spark Standalone and YARN<\/a>.<\/li>\n
  2. While Mesos\u2019s MesosExecutorBackend registered on Spark.<\/li>\n
  3. When LocalEndpoint is created for local mode.<\/li>\n<\/ol>\n

    4. Creating Spark Executor Instance<\/h2>\n

    By using the following, we can create the Spark Executor:<\/span><\/p>\n

      \n
    1. From Executor ID.<\/span><\/li>\n
    2. By using SparkEnv we can access the local MetricsSystem as well as BlockManager. Moreover, we can also access the local serializer by it.<\/span><\/li>\n
    3. From Executor\u2019s hostname.<\/span><\/li>\n
    4. To add to tasks’ classpath, a collection of user-defined JARs. By default, it is empty.<\/span><\/li>\n
    5. By flag whether it runs in local or cluster mode (disabled by default, i.e. cluster is preferred)<\/span><\/li>\n<\/ol>\n

      You must read Spark SQL Features<\/a><\/strong><\/p>\n

      Moreover, when creation is successful, the one INFO messages pop up in the logs. That is:<\/span>
      \nINFO Executor: Starting executor ID [executorId] on host [executorHostname]<\/b><\/p>\n

      5. Heartbeater \u2014 Heartbeat Sender Thread<\/h2>\n

      Basically, with a single thread, heartbeater is a daemon ScheduledThreadPoolExecutor.
      \nWe call this thread pool a driver-heartbeater.<\/p>\n

      6. Launching Task \u2014 launchTask Method<\/h2>\n

      By using this method, we execute the input serializedTask task concurrently.<\/span><\/p>\n

      \"Spark<\/a>

      Spark Executor- Launching Tasks on Executor Using TaskRunners<\/p><\/div>\n

      Let’s revise the Spark Machine Learning Algorithm<\/a><\/strong><\/p>\n

      launchTask(\r\ncontext: ExecutorBackend,\r\ntaskId: Long,\r\nattemptNumber: Int,\r\ntaskName: String,\r\nserializedTask: ByteBuffer): Unit<\/pre>\n
      Moreover, by using launchTask we use to create a TaskRunner, internally. Then, with the help of taskId, we register it in the runningTasks internal registry. Afterwards, we execute it on “Executor task launch worker” thread pool.<\/p>\n
      7. “Executor Task Launch Worker” Thread Pool \u2014 ThreadPool Property<\/h2>\n
      Basically, To launch, by task launch worker id. It uses threadPool daemon cached thread pool. Moreover, at the same time of creation of Spark Executor, threadPool is created. Also, shuts it down when it stops.<\/span><\/p>\n
      You must read about Structured Streaming in SparkR<\/a><\/strong><\/p>\n
      8. Conclusion<\/span><\/h2>\n
      As a result, we have seen, the whole concept of Executors in Apache Spark<\/a>. Moreover, we have also learned how Spark Executors are helpful for executing tasks. The major advantage we have learned is, we can have as many executors we want. Therefore, Executors helps to enhance the Spark performance<\/a> of the system. We have covered each aspect about Apache Spark Executor above. However, if any query occurs feel free to ask in the comment section. \u00a0<\/span><\/p>\n
      See also –\u00a0<\/strong>
      \nList of       Best Apache Spark Books<\/a>
      \n      For Reference<\/a>
      1. Objective In Apache Spark, some distributed agent is responsible for executing tasks, this agent is what we call Spark Executor. This document aims the whole concept of Apache Spark Executor. Also, we will...<\/p>\n","protected":false},"author":6,"featured_media":10289,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[10],"tags":[918,16589,16587,16588],"class_list":["post-5946","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-spark","tag-apache-spark-executor","tag-creating-spark-executor-instance","tag-executer-in-apache-spark","tag-launching-task"],"yoast_head":"\nApache Spark Executor for Executing Spark Tasks - DataFlair<\/title>\n<meta name=\"description\" content=\"Apache Spark Executor, condition to create Spark executor, Creating Spark Executor, Instance,Heartbeat Sender Thread,launch Task Method, Threadpool Property\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/spark-executor\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Spark Executor for Executing Spark Tasks - DataFlair\" \/>\n<meta property=\"og:description\" content=\"Apache Spark Executor, condition to create Spark executor, Creating Spark Executor, Instance,Heartbeat Sender Thread,launch Task Method, Threadpool Property\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/spark-executor\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-01-17T09:16:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-11-08T12:34:50+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Apache-Spark-Executor-01.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Apache Spark Executor for Executing Spark Tasks - DataFlair","description":"Apache Spark Executor, condition to create Spark executor, Creating Spark Executor, Instance,Heartbeat Sender Thread,launch Task Method, Threadpool Property","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/spark-executor\/","og_locale":"en_US","og_type":"article","og_title":"Apache Spark Executor for Executing Spark Tasks - DataFlair","og_description":"Apache Spark Executor, condition to create Spark executor, Creating Spark Executor, Instance,Heartbeat Sender Thread,launch Task Method, Threadpool Property","og_url":"https:\/\/data-flair.training\/blogs\/spark-executor\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-01-17T09:16:25+00:00","article_modified_time":"2019-11-08T12:34:50+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Apache-Spark-Executor-01.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Apache Spark Executor for Executing Spark Tasks","datePublished":"2018-01-17T09:16:25+00:00","dateModified":"2019-11-08T12:34:50+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/"},"wordCount":674,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Apache-Spark-Executor-01.jpg","keywords":["Apache Spark Executor","Creating Spark Executor Instance","Executer in Apache Spark","Launching Task"],"articleSection":["Apache Spark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/spark-executor\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/","url":"https:\/\/data-flair.training\/blogs\/spark-executor\/","name":"Apache Spark Executor for Executing Spark Tasks - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Apache-Spark-Executor-01.jpg","datePublished":"2018-01-17T09:16:25+00:00","dateModified":"2019-11-08T12:34:50+00:00","description":"Apache Spark Executor, condition to create Spark executor, Creating Spark Executor, Instance,Heartbeat Sender Thread,launch Task Method, Threadpool Property","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/spark-executor\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Apache-Spark-Executor-01.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/03\/Apache-Spark-Executor-01.jpg","width":1200,"height":628,"caption":"Apache Spark Executor for Executing Spark Tasks"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/spark-executor\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"Apache Spark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/spark\/"},{"@type":"ListItem","position":3,"name":"Apache Spark Executor for Executing Spark Tasks"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5946","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=5946"}],"version-history":[{"count":7,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5946\/revisions"}],"predecessor-version":[{"id":72544,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/5946\/revisions\/72544"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/10289"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=5946"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=5946"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=5946"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}