1. Objective
Today, in this Spark tutorial, we will learn the Data type mapping between R and Spark. Before, them we will also learn a brief introduction to SparkR.
So, let’s start Data Type Mapping Between R and Spark.
Data type mapping between R and Spark
2. What is SparkR?
Apache Spark 1.4 releases SparkR. One of the major components of SparkR is SparkR DataFrame. Basically, it is nothing but fundamental data structure for data processing in R. Moreover, DataFrames concept extends to other languages with libraries, for example, Pandas etc.
In addition, R offers several software facilities for data manipulation, calculation, and graphical display. Therefore, the key concept behind SparkR was to explore different techniques to integrate the usability of R with the scalability of Spark. Basically, it is the R package. Also gives light-weight frontend to use Apache Spark from R.
Moreover, Using SparkR is beneficial in the following ways:
a. SparkR Data Sources API
Basically, API SparkR can read in data from a variety of sources. It is possible by tying into Spark SQL’s data sources. For example, Hive tables, JSON files, Parquet files etc.
b. SparkR Data Frame Optimizations
Moreover, it inherits all the optimizations made to the computation engine. That is in terms of code generation, memory management.
c. SparkR Scalability to Many Cores and Machines
Although, those operations which execute on SparkR DataFrames get distributed across all the cores and machines over theSpark cluster. Therefore, SparkR DataFrames can run on terabytes of data and clusters with thousands of machines.
3. Data type mapping between R and Spark
| R | Spark |
| byte | byte |
| integer | integer |
| float | float |
| double | double |
| numeric | double |
| character | string |
| string | string |
| binary | binary |
| raw | binary |
| logical | boolean |
| POSIXct | timestamp |
| POSIXlt | timestamp |
| Date | date |
| array | array |
| list | array |
| env | map |
So, this was all in Spark and R data type mapping. Hope you like our explanation.
4. Conclusion
Hence, we have learned about Data type mapping between R and Spark. Also, learned about SparkR. However, if any query occurs, feel free to ask in the comment section. I assure you that we will get back to you.
Best Books for learning Spark.
For reference