{"id":19391,"date":"2018-07-01T04:30:19","date_gmt":"2018-07-01T04:30:19","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=19391"},"modified":"2021-05-12T11:08:58","modified_gmt":"2021-05-12T05:38:58","slug":"pyspark-profiler","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/","title":{"rendered":"Pyspark Profiler – Methods and Functions"},"content":{"rendered":"

In our last article, we discussed PySpark MLlib – Algorithms and Parameters<\/strong>. Today, in this article, we will see PySpark Profiler. Moreover, we will discuss PySpark<\/strong> Profiler functions.<\/p>\n

Basically, to\u00a0ensure that the applications do not waste any resources,\u00a0we want to profile their threads to try\u00a0and spot any problematic code.<\/p>\n

So, let’s start PySpark Profiler.<\/p>\n

What is Pyspark Profiler?<\/span><\/h2>\n

In PySpark,\u00a0custom profilers are supported. The reason behind using custom profilers is to allow different profilers to be used.\u00a0 Also, to do outputting to different formats rather than what is\u00a0offered in the BasicProfiler.<\/span><\/p>\n

However, here are some methods which a custom profiler has to define or inherit:<\/span><\/p>\n

\"PySpark<\/a>

Methods and Functions in PySpark Profilers<\/p><\/div>\n

i. Profile<\/h3>\n

Basically, it produces a system profile of some sort.<\/span><\/p>\n

ii. Stats<\/h3>\n

This method returns the collected stats.<\/span><\/p>\n

iii. Dump<\/h3>\n

It dumps the profiles to a path<\/span><\/p>\n

iv. Add<\/h3>\n

Well, this method adds a profile to the existing accumulated profile.<\/span>
\nMake sure,\u00a0at the time of creating a SparkContext, the profiler class is chosen.<\/span><\/p>\n

>>> from pyspark import SparkConf, SparkContext\r\n>>> from pyspark import BasicProfiler\r\n>>> class MyCustomProfiler(BasicProfiler):\r\n...     def show(self, id):\r\n...         print(\"My custom profiles for RDD:%s\" % id)\r\n...\r\n>>> conf = SparkConf().set(\"spark.python.profile\", \"true\")\r\n>>> sc = SparkContext('local', 'test', conf=conf, profiler_cls=MyCustomProfiler)\r\n>>> sc.parallelize(range(1000)).map(lambda x: 2 * x).take(10)\r\n[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]\r\n>>> sc.parallelize(range(1000)).count()\r\n1000\r\n>>> sc.show_profiles()\r\nMy custom profiles for RDD:1\r\nMy custom profiles for RDD:3\r\n>>> sc.stop()<\/pre>\n
v. Dump(id, path)<\/h3>\n
This function d<\/span>ump the profile into the path, here also\u00a0id is the RDD id<\/span><\/p>\n
def dump(self, id, path):\r\n       if not os.path.exists(path):\r\n           os.makedirs(path)\r\n       stats = self.stats()\r\n       if stats:\r\n           p = os.path.join(path, \"rdd_%d.pstats\" % id)\r\n           stats.dump_stats(p)<\/pre>\n
vi. Profile(func)<\/span><\/h3>\n
It performs profiling on the function func.<\/span><\/p>\n
def profile(self, func):\r\n       raise NotImplemented<\/pre>\n
vii. Show(id)<\/h3>\n
Moreover, this function Prints the profile stats to stdout. And here id is the RDD id.<\/span><\/p>\n
def show(self, id):\r\n       stats = self.stats()\r\n       if stats:\r\n           print(\"=\" * 60)\r\n           print(\"Profile of RDD<id=%d>\" % id)\r\n           print(\"=\" * 60)\r\n           stats.sort_stats(\"time\", \"cumulative\").print_stats()<\/pre>\n
viii. Stats()<\/h3>\n
This function returns the collected profiling stats (pstats.Stats)<\/span><\/p>\n
def stats(self):\r\n       raise NotImplemented<\/pre>\n
class pyspark.BasicProfiler(ctx)<\/span><\/h2>\n
A default profiler, that is implemented on the basis of cProfile and Accumulator, is what we call a BasicProfiler.\u00a0<\/span><\/p>\n
[docs]    def profile(self, func):\r\n       pr = cProfile.Profile()\r\n       pr.runcall(func)\r\n       st = pstats.Stats(pr)\r\n       st.stream = None  # make it picklable\r\n       st.strip_dirs()\r\n       # It adds a new profile to the existing accumulated value\r\n       self._accumulator.add(st)<\/pre>\n
i. profile(func)<\/h3>\n
This function runs as well as profiles the method to_profile passed in. Moreover, here a profile object is returned.<\/span><\/p>\n
ii. stats()<\/h3>\n
def stats(self):\r\n       return self._accumulator.value<\/pre>\n
So, this was all about PySpark Profiler. Hope you like our explanation.<\/p>\n
Conclusion<\/span><\/h2>\n
Hence, we have seen whole about PySpark Profilers including their functions. Thus this article will definitely clear your concepts regarding PySpark Profilers. Still, if any doubt, ask in the comment tab.<\/p>\n","protected":false},"excerpt":{"rendered":"
In our last article, we discussed PySpark MLlib – Algorithms and Parameters. Today, in this article, we will see PySpark Profiler. Moreover, we will discuss PySpark Profiler functions. Basically, to\u00a0ensure that the applications do...<\/p>\n","protected":false},"author":6,"featured_media":19537,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[10172,10299,10313,10314],"class_list":["post-19391","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pyspark","tag-profiler-in-pyspark","tag-pyspark-data-profiling","tag-pyspark-profiler","tag-pyspark-profiler-functions"],"yoast_head":"\nPyspark Profiler - Methods and Functions - DataFlair<\/title>\n<meta name=\"description\" content=\"PySpark Profiler tutorial, what is PySpark profiler, functions of PySpark Profiler, PySpark Profiler example, PySpark Basic Profiler\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Pyspark Profiler - Methods and Functions - DataFlair\" \/>\n<meta property=\"og:description\" content=\"PySpark Profiler tutorial, what is PySpark profiler, functions of PySpark Profiler, PySpark Profiler example, PySpark Basic Profiler\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-01T04:30:19+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-12T05:38:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Pyspark-Profiler-01-1-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Pyspark Profiler - Methods and Functions - DataFlair","description":"PySpark Profiler tutorial, what is PySpark profiler, functions of PySpark Profiler, PySpark Profiler example, PySpark Basic Profiler","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/","og_locale":"en_US","og_type":"article","og_title":"Pyspark Profiler - Methods and Functions - DataFlair","og_description":"PySpark Profiler tutorial, what is PySpark profiler, functions of PySpark Profiler, PySpark Profiler example, PySpark Basic Profiler","og_url":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-07-01T04:30:19+00:00","article_modified_time":"2021-05-12T05:38:58+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Pyspark-Profiler-01-1-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"Pyspark Profiler – Methods and Functions","datePublished":"2018-07-01T04:30:19+00:00","dateModified":"2021-05-12T05:38:58+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/"},"wordCount":337,"commentCount":1,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Pyspark-Profiler-01-1-1.jpg","keywords":["profiler in PySpark","PySpark Data profiling","Pyspark Profiler","PySpark Profiler functions"],"articleSection":["PySpark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/","url":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/","name":"Pyspark Profiler - Methods and Functions - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Pyspark-Profiler-01-1-1.jpg","datePublished":"2018-07-01T04:30:19+00:00","dateModified":"2021-05-12T05:38:58+00:00","description":"PySpark Profiler tutorial, what is PySpark profiler, functions of PySpark Profiler, PySpark Profiler example, PySpark Basic Profiler","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/pyspark-profiler\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Pyspark-Profiler-01-1-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Pyspark-Profiler-01-1-1.jpg","width":1200,"height":628,"caption":"Pyspark Profiler - Methods and Functions"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/pyspark-profiler\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"PySpark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pyspark\/"},{"@type":"ListItem","position":3,"name":"Pyspark Profiler – Methods and Functions"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19391","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=19391"}],"version-history":[{"count":6,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19391\/revisions"}],"predecessor-version":[{"id":94247,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19391\/revisions\/94247"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/19537"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=19391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=19391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=19391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}