0

I want to import data from Mysql using sqoop import but my requirement is i want to use 4 mappers but it should create only one file in hdfs target directory is there is any way to do this ?

Rajendra B
  • 11
  • 5
  • Possible duplicate of [merge output files after reduce phase](http://stackoverflow.com/questions/5700068/merge-output-files-after-reduce-phase) – Rahul Sharma Oct 25 '16 at 17:07

3 Answers3

2

No. there is no option in sqoop to re-partition files into 1 file.

I don't think this should be a headache of sqoop.

You can do it easily using getmerge feature of hadoop. Example:

hadoop fs -getmerge /sqoop/target-dir/ /desired/local/output/file.txt

Here

  • /sqoop/target-dir is the target-dir of your sqoop command (directory containing all the part files).

  • desired/local/output/file.txt is the combined single file.

Dev
  • 12,854
  • 16
  • 67
  • 159
0

you can use below sqoop command..!!

Suppose database name is prateekDB and table name is Emp...!!

sqoop import  --connect "jdbc:mysql://localhost:3306/prateekDB"  --username=root \
--password=data  --table Emp  --target-dir /SqoopImport  --split-by empno
Piyush Patel
  • 1,456
  • 1
  • 13
  • 24
Prateek Jaiswal
  • 1
-1

Add this option to sqoop 

--num-mappers 1

the sqoop log shows:

Job Counters 
    Launched map tasks=1
    Other local map tasks=1

and finally on hdfs ONE file is created.

ozw1z5rd
  • 2,764
  • 3
  • 26
  • 47