PySpark SparkFiles and Its Class Methods

In this PySpark article, “PySpark SparkFiles and its Class Methods” we will learn the whole concept of SparkFiles using PySpark(Spark with Python). Also, we will describe both of its Class Methods along with their code to understand it well.

So, let’s start PySpark SparkFiles.

What is PySpark SparkFiles?

By using SparkFiles.get, we can upload our files in Apache Spark. However, sc refers to our default SparkContext here. Moreover, we can also get the path on a worker using the command “SparkFiles.get”.

Hence, in order to resolve the paths to files added through SparkContext.addFile(), we can use SparkFiles.

There are following types of class methods in SparkFiles, such as  −

• get(filename)
• getrootdirectory()

Although make sure that SparkFiles only contains class methods; users should not create SparkFiles instances.
Further, let’s learn about both of the classmethods in depth.

Class Methods of PySpark SparkFiles

So, let’s learn the two PySpark SparkFiles Class Methods in detail:

i. get(filename)

Basically, the classmethod “get(filename)” specifies the path of the file which is added through SparkContext.addFile().

import os
 -class SparkFiles(object):
 """
 Resolves paths to files added through
 L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.
 SparkFiles contains only classmethods; users should not create SparkFiles
 instances.
 """
 _root_directory = None
 _is_running_on_worker = False
 _sc = None
 - def __init__(self):
 raise NotImplementedError("Do not construct SparkFiles objects")
 @classmethod
 - def get(cls, filename):
 """
 Get the absolute path of a file added through C{SparkContext.addFile()}.
 """
 path = os.path.join(SparkFiles.getRootDirectory(), filename)
 return os.path.abspath(path)
 @classmethod
 + def getRootDirectory(cls):
...

ii. getrootdirectory()

Whereas, this file, specifies the path to the root directory. Basically, it contains the whole file which is added through the SparkContext.addFile().

 import os
 -class SparkFiles(object):
 """
 Resolves paths to files added through
 L{SparkContext.addFile()<pyspark.context.SparkContext.addFile>}.
 SparkFiles contains only classmethods; users should not create SparkFiles
 instances.
 """
 _root_directory = None
 _is_running_on_worker = False
 _sc = None
 - def __init__(self):
 raise NotImplementedError("Do not construct SparkFiles objects")
 @classmethod
 + def get(cls, filename):
...
 @classmethod
 - def getRootDirectory(cls):
 """
 Than Get the root directory which contains files added through
 C{SparkContext.addFile()}.
 """
 if cls._is_running_on_worker:
 return cls._root_directory
 else:
 # This will have to change if we support multiple SparkContexts:
 return cls._sc._jvm.spark.SparkFiles.getRootDirectory()

So, this is all about PySpark SparkFiles.

Conclusion

Hence, we have seen the whole concept of PySpark SparkFiles in this article. Also, we have included their class methods to get in-depth knowledge of the topic. So, if any doubt occurs regarding PySpark SparkFiles, feel free to ask through the comment section. we are happy to respond. Hope it helps!