How to change the size of data chunk in Hadoop?

Question

I'm using the new Hadoop API and as there is no way to explicitly specify the number of mappers (unlike the old API), I need to change the size of the data chunks so that I can control the number of mappers. How to change the default size of the data split in the code?

http://stackoverflow.com/questions/2669800/changing-the-block-size-of-a-dfs-file-in-hadoop might help — michaeltang, Feb 27 '14 at 03:59
I don't think that you could do that in the old API, either. — vefthym, Feb 27 '14 at 13:07

score 0 · Answer 1 · answered Feb 27 '14 at 04:28

0

Number of Mappers is determined by (File) Splits
The split size is determined by the InputFormat being used.
mapred.max.split.size parameter will let you define the split size.

answered Feb 27 '14 at 04:28

Jasper

7,942
27
90
131

How to change the size of data chunk in Hadoop?

1 Answers1