{"id":19143,"date":"2018-06-30T04:04:23","date_gmt":"2018-06-30T04:04:23","guid":{"rendered":"https:\/\/data-flair.training\/blogs\/?p=19143"},"modified":"2021-05-12T11:09:08","modified_gmt":"2021-05-12T05:39:08","slug":"pyspark-storagelevel","status":"publish","type":"post","link":"https:\/\/data-flair.training\/blogs\/pyspark-storagelevel\/","title":{"rendered":"Learn PySpark StorageLevel With Example"},"content":{"rendered":"

Today, in this PySpark<\/strong> article, we will learn the whole concept of\u00a0PySpark StorageLevel in depth. Basically, while it comes to store RDD<\/strong>,\u00a0StorageLevel in Spark decides how\u00a0it should be stored.<\/p>\n

So, let’s learn about Storage levels using PySpark. Also, we will learn an\u00a0example of StorageLevel\u00a0in PySpark to understand it well.<\/p>\n

So, let’s start PySpark StorageLevel.<\/p>\n

What is PySpark StorageLevel?<\/span><\/h2>\n

Well, how RDD should be stored in Apache Spark<\/strong>, PySpark StorageLevel decides it. Also,\u00a0whether RDD should be stored in the memory or should it be stored over the disk, or both StorageLevel decides. <\/span><\/p>\n

In addition,\u00a0whether to serialize RDD and whether to replicate RDD partitions,\u00a0it also decides this.\u00a0Moreover, for some commonly used PySpark StorageLevels, it contains static constants,\u00a0like MEMORY_ONLY.<\/span><\/p>\n

So, here is the code which has the class definition of a PySpark\u00a0StorageLevel \u2212<\/span><\/p>\n

class pyspark.StorageLevel(useDisk, useMemory, useOffHeap, deserialized, replication = 1)<\/pre>\n
Class Variables<\/span><\/h2>\n
Hence, there are different PySpark\u00a0 StorageLevels, to decide the storage of RDD, such as:<\/p>\n