{"id":19698,"date":"2018-07-01T04:27:48","date_gmt":"2018-07-01T04:27:48","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=19698"},"modified":"2021-05-12T11:09:05","modified_gmt":"2021-05-12T05:39:05","slug":"pyspark-statustracker","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/","title":{"rendered":"PySpark StatusTracker(jtracker) | Learn PySpark"},"content":{"rendered":"

In our last PySpark tutorial, we discussed Pyspark Profiler<\/strong>.\u00a0Today, in this PySpark Tutorial<\/strong>, “Introduction to PySpark StatusTracker” we will learn the concept of\u00a0PySpark StatusTracker(jtracker).
\nSo, let’s begin with PySpark StatusTracker(jtracker).<\/p>\n

What is PySpark StatusTracker(jtracker)?<\/span><\/h2>\n

Basically, for monitoring job and stage progress, low-level status reporting APIs are there.<\/span>
\nHowever, these APIs\u00a0offers very weak consistency semantics intentionally; So,\u00a0we can say consumers of these APIs\u00a0must be prepared to handle empty \/ missing information. <\/span><\/p>\n

Let’s understand PySpark StatusTracker with an example, here a job\u2019s stage ids may be known\u00a0yet there status API may not know about the information about the details of those stages. Hence, in that case, getStageInfo could potentially return None for a valid stage id.<\/span><\/p>\n

Make sure, these APIs only provide information on recent jobs\/stages, to limit memory usage. In addition,\u00a0for the last spark.ui.retainedStages stages and spark.ui.retainedJobs jobs, these APIs will provide information.<\/span><\/p>\n

def __init__(self, jtracker):\r\n       self._jtracker = jtracker<\/pre>\n
i. getActiveJobsIds()<\/span><\/h3>\n
Moreover, this class returns an array, which contains the ids of all active jobs.<\/span><\/p>\n
def getActiveJobsIds(self):\r\n               return sorted((list(self._jtracker.getActiveJobIds())))<\/pre>\n
ii. getActiveStageIds()<\/h3>\n
However, this returns an array, which contains the ids of all active stages.<\/span>
\nRead Pros and cons of<\/strong> PySpark\u00a0<\/strong><\/p>\n
def getActiveStageIds(self):\r\n       return sorted(list(self._jtracker.getActiveStageIds()))However<\/pre>\n
iii. getJobIdsForGroup(jobGroup=None)<\/span><\/h3>\n
It helps to get a list of all known jobs in a particular job group. However, it\u00a0returns all known jobs that are not associated with a job group if jobGroup is None.<\/p>\n
Basically, this returned list may contain several types of jobs, like running, failed, and completed jobs. However, this may vary across invocations of this method. Make sure, there is no guarantee of the order of the elements in its result.<\/span><\/p>\n
def getJobIdsForGroup(self, jobGroup=None):\r\n              return list(self._jtracker.getJobIdsForGroup(jobGroup))<\/pre>\n
iv. getJobInfo(jobId)<\/span><\/h3>\n
If not able to find job info or\u00a0due to garbage collection, it returns a SparkJobInfo object, or None.<\/span><\/p>\n
def getJobInfo(self, jobId):\r\n       job = self._jtracker.getJobInfo(jobId)\r\n       if job is not None:\r\n           return SparkJobInfo(jobId, job.stageIds(), str(job.status()))<\/pre>\n
v. getStageInfo(stageId)<\/span><\/h3>\n
Similarly, If not able to find job info or\u00a0due to garbage collection, it returns a SparkStageInfo object or None.<\/span><\/p>\n
def getStageInfo(self, stageId):\r\n         stage = self._jtracker.getStageInfo(stageId)\r\n       if stage is not None:\r\n           # TODO: fetch them in batch for better performance\r\n           attrs = [getattr(stage, f)() for f in SparkStageInfo._fields[1:]]\r\n           return SparkStageInfo(stageId, *attrs)<\/pre>\n
So, this is all about PySpark StatusTracker(jtracker). Hope you like our explanation.<\/p>\n
Conclusion<\/span><\/h2>\n
Hence, we have seen the whole concept of PySpark StatusTracker(jtracker) along with their code. Still, if any doubt regarding PySpark StatusTracker(jtracker), ask in the comment tab.<\/p>\n","protected":false},"excerpt":{"rendered":"
In our last PySpark tutorial, we discussed Pyspark Profiler.\u00a0Today, in this PySpark Tutorial, “Introduction to PySpark StatusTracker” we will learn the concept of\u00a0PySpark StatusTracker(jtracker). So, let’s begin with PySpark StatusTracker(jtracker). What is PySpark StatusTracker(jtracker)?...<\/p>\n","protected":false},"author":6,"featured_media":75107,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[10290,10325,10328],"class_list":["post-19698","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pyspark","tag-pyspark-activity-tracker","tag-pyspark-statustracker","tag-pyspark-tutorial"],"yoast_head":"\nPySpark StatusTracker(jtracker) | Learn PySpark - DataFlair<\/title>\n<meta name=\"description\" content=\"PySpark StatusTracker (jtracker), what is PySpark StatusTracker, how to use PySpark StatusTracker, PySpark activity Tracker\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark StatusTracker(jtracker) | Learn PySpark - DataFlair\" \/>\n<meta property=\"og:description\" content=\"PySpark StatusTracker (jtracker), what is PySpark StatusTracker, how to use PySpark StatusTracker, PySpark activity Tracker\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-01T04:27:48+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-12T05:39:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/pyspark-statustracker.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"802\" \/>\n\t<meta property=\"og:image:height\" content=\"420\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"PySpark StatusTracker(jtracker) | Learn PySpark - DataFlair","description":"PySpark StatusTracker (jtracker), what is PySpark StatusTracker, how to use PySpark StatusTracker, PySpark activity Tracker","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/","og_locale":"en_US","og_type":"article","og_title":"PySpark StatusTracker(jtracker) | Learn PySpark - DataFlair","og_description":"PySpark StatusTracker (jtracker), what is PySpark StatusTracker, how to use PySpark StatusTracker, PySpark activity Tracker","og_url":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-07-01T04:27:48+00:00","article_modified_time":"2021-05-12T05:39:05+00:00","og_image":[{"width":802,"height":420,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/pyspark-statustracker.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"PySpark StatusTracker(jtracker) | Learn PySpark","datePublished":"2018-07-01T04:27:48+00:00","dateModified":"2021-05-12T05:39:05+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/"},"wordCount":365,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/pyspark-statustracker.jpg","keywords":["PySpark activity Tracker","PySpark StatusTracker","PySpark tutorial"],"articleSection":["PySpark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/","url":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/","name":"PySpark StatusTracker(jtracker) | Learn PySpark - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/pyspark-statustracker.jpg","datePublished":"2018-07-01T04:27:48+00:00","dateModified":"2021-05-12T05:39:05+00:00","description":"PySpark StatusTracker (jtracker), what is PySpark StatusTracker, how to use PySpark StatusTracker, PySpark activity Tracker","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/pyspark-statustracker.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/07\/pyspark-statustracker.jpg","width":802,"height":420,"caption":"pyspark statustracker"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/pyspark-statustracker\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"PySpark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pyspark\/"},{"@type":"ListItem","position":3,"name":"PySpark StatusTracker(jtracker) | Learn PySpark"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19698","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=19698"}],"version-history":[{"count":8,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19698\/revisions"}],"predecessor-version":[{"id":94249,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19698\/revisions\/94249"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/75107"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=19698"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=19698"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=19698"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}