{"id":19091,"date":"2018-06-30T04:30:52","date_gmt":"2018-06-30T04:30:52","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=19091"},"modified":"2021-05-12T11:09:05","modified_gmt":"2021-05-12T05:39:05","slug":"pyspark-sparkfiles","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/","title":{"rendered":"PySpark SparkFiles and Its Class Methods"},"content":{"rendered":"

In this PySpark<\/strong> article, “PySpark SparkFiles and its Class Methods\u201d we will learn the whole concept of SparkFiles using PySpark(Spark<\/strong> with Python<\/strong>). Also, we will describe both of its Class Methods\u00a0along with their code to understand it well. <\/span><\/p>\n

So, let’s start PySpark SparkFiles.<\/span><\/p>\n

What is PySpark SparkFiles?<\/h2>\n

By using SparkFiles.get, we can upload our files in Apache Spark<\/strong>. However, sc\u00a0refers to our default SparkContext<\/strong> here. Moreover, we can also get the path on a worker using the command “SparkFiles.get”. <\/span><\/p>\n

Hence, in order to resolve the paths to files added through SparkContext.addFile(), we can use SparkFiles.<\/span><\/p>\n

There are following types of class methods in SparkFiles, such as\u00a0 \u2212<\/span><\/p>\n

    \n
  • get(filename)<\/span><\/li>\n
  • getrootdirectory()<\/span><\/li>\n<\/ul>\n

    Although make sure that SparkFiles\u00a0only contains class methods; users should not create SparkFiles instances.
    \nFurther, let\u2019s learn about both of the classmethods in depth.<\/span><\/p>\n

    Class Methods of PySpark SparkFiles<\/h2>\n

    So, let’s learn the two PySpark SparkFiles Class Methods in detail:<\/p>\n

    i. get(filename)<\/span><\/h3>\n

    Basically, the classmethod “get(filename)” specifies the path of the file which is added through SparkContext.addFile().<\/span><\/p>\n

    import os\r\n -class SparkFiles(object):\r\n \"\"\"\r\n Resolves paths to files added through\r\n L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.\r\n SparkFiles contains only classmethods; users should not create SparkFiles\r\n instances.\r\n \"\"\"\r\n _root_directory = None\r\n _is_running_on_worker = False\r\n _sc = None\r\n - def __init__(self):\r\n raise NotImplementedError(\"Do not construct SparkFiles objects\")\r\n @classmethod\r\n - def get(cls, filename):\r\n \"\"\"\r\n Get the absolute path of a file added through C{SparkContext.addFile()}.\r\n \"\"\"\r\n path = os.path.join(SparkFiles.getRootDirectory(), filename)\r\n return os.path.abspath(path)\r\n @classmethod\r\n + def getRootDirectory(cls):\r\n...\r\n<\/pre>\n
    ii. getrootdirectory()<\/h3>\n
    Whereas, this file, specifies the path to the root directory. Basically, it contains the whole file which is added through the SparkContext.addFile().<\/span><\/p>\n
     import os\r\n -class SparkFiles(object):\r\n \"\"\"\r\n Resolves paths to files added through\r\n L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.\r\n SparkFiles contains only classmethods; users should not create SparkFiles\r\n instances.\r\n \"\"\"\r\n _root_directory = None\r\n _is_running_on_worker = False\r\n _sc = None\r\n - def __init__(self):\r\n raise NotImplementedError(\"Do not construct SparkFiles objects\")\r\n @classmethod\r\n + def get(cls, filename):\r\n...\r\n @classmethod\r\n - def getRootDirectory(cls):\r\n \"\"\"\r\n Than Get the root directory which contains files added through\r\n C{SparkContext.addFile()}.\r\n \"\"\"\r\n if cls._is_running_on_worker:\r\n return cls._root_directory\r\n else:\r\n # This will have to change if we support multiple SparkContexts:\r\n return cls._sc._jvm.spark.SparkFiles.getRootDirectory()<\/pre>\n
    So, this is all about PySpark SparkFiles.<\/p>\n
    Conclusion<\/span><\/h2>\n
    Hence, we have seen the whole concept of PySpark SparkFiles in this article<\/span>. Also, we have included their class methods to get in-depth knowledge of the topic. So, if any doubt occurs regarding PySpark SparkFiles, feel free to ask through the comment section. we are happy to respond. Hope it helps!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
    In this PySpark article, “PySpark SparkFiles and its Class Methods\u201d we will learn the whole concept of SparkFiles using PySpark(Spark with Python). Also, we will describe both of its Class Methods\u00a0along with their code...<\/p>\n","protected":false},"author":6,"featured_media":19502,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[44],"tags":[2555,5086,5091,10300,13163,15490],"class_list":["post-19091","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pyspark","tag-classmethods-of-pyspark-sparkfiles","tag-getfilename","tag-getrootdirectory","tag-pyspark-documentation","tag-sparkfiles-get-examples","tag-what-are-pyspark-sparkfiles"],"yoast_head":"\nPySpark SparkFiles and Its Class Methods - DataFlair<\/title>\n<meta name=\"description\" content=\"PySpark Tutorial, PySpark SparkFiles, Class Methods in PySpark SparkFiles, get(filename), getrootdirectory method, what is PySpak SparkFiles\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark SparkFiles and Its Class Methods - DataFlair\" \/>\n<meta property=\"og:description\" content=\"PySpark Tutorial, PySpark SparkFiles, Class Methods in PySpark SparkFiles, get(filename), getrootdirectory method, what is PySpak SparkFiles\" \/>\n<meta property=\"og:url\" content=\"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/\" \/>\n<meta property=\"og:site_name\" content=\"DataFlair\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataFlairWS\/\" \/>\n<meta property=\"article:published_time\" content=\"2018-06-30T04:30:52+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-05-12T05:39:05+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Class-Methods-01-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"628\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"DataFlair Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:site\" content=\"@DataFlairWS\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DataFlair Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"PySpark SparkFiles and Its Class Methods - DataFlair","description":"PySpark Tutorial, PySpark SparkFiles, Class Methods in PySpark SparkFiles, get(filename), getrootdirectory method, what is PySpak SparkFiles","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/","og_locale":"en_US","og_type":"article","og_title":"PySpark SparkFiles and Its Class Methods - DataFlair","og_description":"PySpark Tutorial, PySpark SparkFiles, Class Methods in PySpark SparkFiles, get(filename), getrootdirectory method, what is PySpak SparkFiles","og_url":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/","og_site_name":"DataFlair","article_publisher":"https:\/\/www.facebook.com\/DataFlairWS\/","article_published_time":"2018-06-30T04:30:52+00:00","article_modified_time":"2021-05-12T05:39:05+00:00","og_image":[{"width":1200,"height":628,"url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Class-Methods-01-1.jpg","type":"image\/jpeg"}],"author":"DataFlair Team","twitter_card":"summary_large_image","twitter_creator":"@DataFlairWS","twitter_site":"@DataFlairWS","twitter_misc":{"Written by":"DataFlair Team","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#article","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/"},"author":{"name":"DataFlair Team","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89"},"headline":"PySpark SparkFiles and Its Class Methods","datePublished":"2018-06-30T04:30:52+00:00","dateModified":"2021-05-12T05:39:05+00:00","mainEntityOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/"},"wordCount":269,"commentCount":0,"publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Class-Methods-01-1.jpg","keywords":["Classmethods of PySpark SparkFiles","get(filename)","getrootdirectory()","PySpark Documentation","Sparkfiles.get examples","What are PySpark SparkFiles?"],"articleSection":["PySpark Tutorials"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/","url":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/","name":"PySpark SparkFiles and Its Class Methods - DataFlair","isPartOf":{"@id":"https:\/\/data-flair.training\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#primaryimage"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#primaryimage"},"thumbnailUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Class-Methods-01-1.jpg","datePublished":"2018-06-30T04:30:52+00:00","dateModified":"2021-05-12T05:39:05+00:00","description":"PySpark Tutorial, PySpark SparkFiles, Class Methods in PySpark SparkFiles, get(filename), getrootdirectory method, what is PySpak SparkFiles","breadcrumb":{"@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#primaryimage","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Class-Methods-01-1.jpg","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2018\/06\/Class-Methods-01-1.jpg","width":1200,"height":628,"caption":"PySpark SparkFiles and Its Class Methods"},{"@type":"BreadcrumbList","@id":"https:\/\/data-flair.training\/blogs\/pyspark-sparkfiles\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog Home","item":"https:\/\/data-flair.training\/blogs\/"},{"@type":"ListItem","position":2,"name":"PySpark Tutorials","item":"https:\/\/data-flair.training\/blogs\/category\/pyspark\/"},{"@type":"ListItem","position":3,"name":"PySpark SparkFiles and Its Class Methods"}]},{"@type":"WebSite","@id":"https:\/\/data-flair.training\/blogs\/#website","url":"https:\/\/data-flair.training\/blogs\/","name":"DataFlair","description":"Learn Today. Lead Tomorrow.","publisher":{"@id":"https:\/\/data-flair.training\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/data-flair.training\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/data-flair.training\/blogs\/#organization","name":"DataFlair","url":"https:\/\/data-flair.training\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","contentUrl":"https:\/\/data-flair.training\/blogs\/wp-content\/uploads\/sites\/2\/2016\/07\/Data-Flair.png","width":106,"height":48,"caption":"DataFlair"},"image":{"@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataFlairWS\/","https:\/\/x.com\/DataFlairWS","https:\/\/www.linkedin.com\/company\/dataflair-web-services-pvt-ltd\/","https:\/\/www.youtube.com\/user\/DataFlairWS"]},{"@type":"Person","@id":"https:\/\/data-flair.training\/blogs\/#\/schema\/person\/2c58ecb4f73a39f0ef993f1ddfcd7b89","name":"DataFlair Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1ce4a0e3e542444fc73bbebf83e89e8b73e2d95ccb1fcee64da9945f078b97c5?s=96&d=mm&r=g","caption":"DataFlair Team"},"description":"The DataFlair Team provides industry-driven content on programming, Java, Python, C++, DSA, AI, ML, data Science, Android, Flutter, MERN, Web Development, and technology. Our expert educators focus on delivering value-packed, easy-to-follow resources for tech enthusiasts and professionals.","url":"https:\/\/data-flair.training\/blogs\/author\/dfteam2\/"}]}},"amp_enabled":true,"_links":{"self":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19091","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/comments?post=19091"}],"version-history":[{"count":4,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19091\/revisions"}],"predecessor-version":[{"id":94251,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/posts\/19091\/revisions\/94251"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media\/19502"}],"wp:attachment":[{"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/media?parent=19091"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/categories?post=19091"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/data-flair.training\/blogs\/wp-json\/wp\/v2\/tags?post=19091"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}