- This topic has 2 replies, 1 voice, and was last updated 5 years, 7 months ago by
.
-
Posts
-
-
-
In Hadoop HDFS, block size is specified in conf. file – hdfs-site.xml. To change the block size, parameter, dfs.block.size can be changed to required value(default 64mb/128mb) in hdfs-site.xml file. Once this is changed, cluster restart is required for the change to effect, for which will be applied only to the new files.
Existing files’ block size doesn’t change. In order to change the existing files’ block size, ‘distcp’ utility can be used.DistCp (distributed copy) is a tool used for large inter/intra-cluster copying. It uses MapReduce to effect its
distribution, error handling and recovery and reporting.It will copy the files with the new block size to the new location. However, we have to manually delete the old files with the older block size.
Command: hadoop distcp –
Ddfs.block.size=XX /path/to/inputdata(Source) /path/to/inputdata-with-largeblocks(Destination). -
To configure the data block at Cluster level we need to specify in the hdfs.site.xml, eg. For 128 mb,value will be 128*1024*1024
<property>
<name>dfs.block.size</name>
<value>134217728</value>
<description>Block size</description>
</property>
For multinode, in a Cluster. we need to update the same in the node(Name Node and Data Node) and restart the daemons.This change doesn’t affect the existing files in Hadoop HDFS.
To change a block size for specific file in a cluster:
hadoop fs -Ddfs.blocksize=134217728 -put /home/hdadmin/mydata/test.text /input
-
- You must be logged in to reply to this topic.
