46

trying to run MR program version(2.7) in windows 7 64 bit in eclipse while running the above exception occurring . I verified that using 64 bit 1.8 java version and observed that all the hadoop daemons are running.

Any suggestions highly appreciated

blackbishop
  • 26,760
  • 8
  • 50
  • 69
sukumar konduru
  • 459
  • 1
  • 4
  • 4
  • It seems that java unable to find any native library. May be any dll file is corrupted or not accessible by java program. – Anil Agrawal Jan 25 '17 at 12:32
  • There is a similar issue , you can check here http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows – Anil Agrawal Jan 25 '17 at 12:32
  • in that case while starting daemons this problem occurring .but in my case no issues while starting but while executing map reduce program that exception occurring – sukumar konduru Jan 25 '17 at 13:05
  • It seems any native library that is required for map reduce program is not accessible. Please share stack trace for more information. – Anil Agrawal Jan 25 '17 at 19:22

15 Answers15

44

In addition to other solutions, Please download winutil.exe and hadoop.dll and add to $HADOOP_HOME/bin. It works for me.

https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin

Note: I'm using hadoop-2.7.3 version enter image description here

enter image description here enter image description here

Thirupathi Chavati
  • 1,491
  • 11
  • 10
  • I have encountered the same problem while starting Namenode. Iam unable to start Namenode. Please help – Jon Andrews Jun 04 '20 at 06:53
  • Iam using Hadoop-3.0.0. I had downloaded hadoop.dll and winutils.exe from the Github link you had provided. But still no luck. It throws the same error while starting Namenode – Jon Andrews Jun 04 '20 at 06:55
  • 2
    The link above doesn't have 3.1.1. (yet?). Found this one that seems to be working for the newest version: https://github.com/kontext-tech/winutils – Dan Mar 15 '21 at 19:35
  • @JonAndrews - were you able to sort it out at the end? Having the same issue, the same hadoop version... Thanks! – ialekseev Mar 29 '21 at 11:55
35

After putting haddop.dll and winutils in hadoop/bin folder and adding the folder of hadoop to PATH, we also need to put hadoop.dll into the C:\Windows\System32 folder

JasonWayne
  • 1,466
  • 1
  • 16
  • 16
  • 1
    Have done this but the error still pop up. Any suggestions? – Jaison Apr 13 '20 at 06:20
  • I didn't needed the `C:\Windows\System32` step. `WinUntils.exe` and `hadoop.dll` in the Hadoop bin folders did it for me. The environment variable and path setting are critical though. – Dan Mar 15 '21 at 19:32
  • 4
    @Dan Dunno why, but putting hadoop.dll into the C:\Windows\System32 folder was critical for me as well. – diman82 Jul 10 '21 at 22:36
  • @diman82 Did you have the environment variable `HADOOP_HOME` and path set-up and restarted? (Not critical, things are running for you; I'm simply curious what's causing the difference). – Dan Jul 13 '21 at 00:49
22

This issue occurred to me and the cause was I forgot to append %HADOOP_HOME%/bin to PATH in my environment variables.

Julius Delfino
  • 881
  • 7
  • 25
  • It's worth noting that the ``HADOOP_HOME`` user or system variable has to be created first before the above can be added to the ``PATH``. Typically, ``HADOOP_HOME`` would point to where your hadoop is installed such as ``C:\hadoop-2.8.1``. And as always with changing the ``PATH`` variable, a restart is usually required. – Sal Jul 23 '19 at 20:22
5

After trying all the above, things worked after putting hadoop.dll to windows/System32

mohit bhindwal
  • 61
  • 1
  • 1
5

In my case I was having this issue when running unit tests on local machine after upgrading dependencies to CDH6. I already had HADOOP_HOME and PATH variables configured properly but I had to copy the hadoop.ddl to C:\Windows\System32 as suggested in the other answer.

Gass
  • 4,098
  • 2
  • 13
  • 27
OsvaldoP
  • 51
  • 1
  • 1
4

For me this issue was resolved by downloading the winutils.exe & hadoop.dll from https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin and putting those in hadoop/bin folder

Aman Tandon
  • 1,241
  • 1
  • 13
  • 24
3

I already had %HADOOP_HOME%/bin in my PATH and my code had previously run without errors. Restarting my machine made it work again.

Ben Watson
  • 4,984
  • 4
  • 36
  • 62
2

The version mismacth is main cause for this issue. Follow complete hadoop version with java library will solve the issue and if you still face issue and working on hadoop 3.1.x version use this library to download bin

https://github.com/s911415/apache-hadoop-3.1.0-winutils/tree/master/bin

Vik_Technologist
  • 51
  • 4
2

Adding hadoop.dll and WinUntils.exe fixed the error, support for latest versions can be found here

MansourAlawi
  • 21
  • 3
  • 1
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jan 20 '22 at 09:44
  • this one work for me. – trungducng Jan 25 '22 at 16:06
1

I already had %HADOOP_HOME%/bin in my PATH. Adding hadoop.dll in Hadoop/bin directory made it work again.

ahmedshahriar
  • 778
  • 5
  • 19
0

In Intellij under Run/Debug Configurations, open the application you are trying to run, Under configurations tab, specify the exact working Directory.having the variable to represent the working directory also creates this problem. When I changed the Working Directory under configurations, it started working again.

hemanth
  • 1
  • 2
0

Yes this issues arose when I was using the PIGUNITS for automation of PIGSCRIPTS. Two things in sequence need to be done:

  1. Copy both the files as mentioned about in a location and add it to the environment variables under PATH.

  2. To reflect the change what you have just done, you have to restart your machine to load the file.

Under JUNIT I was getting this error which would help others as well:

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias XXXXX. Backend error : java.lang.IllegalStateException: Job in state DEFINE instead of RUNNING at org.apache.pig.PigServer.openIterator(PigServer.java:925)

kenlukas
  • 3,221
  • 9
  • 23
  • 35
ravi
  • 15
  • 3
0

This might be old but if its still not working for someone, Step 1- Double click on winutils.exe. If it shows some dll file is missing, download that .dll file and place that at appropriate place.

In my case, msvcr100.dll wasmissing and I had to install Microsoft Visual C++ 2010 Service Pack 1 Redistributable Package to make it work. All the best

Shubham Chourasia
  • 31
  • 2
0

I had everything configured correctly, but while using pyspark.SparkContext.wholeTextFiles I specified the path as "/directory/" instead of "/directory/*" which gave me this issue.

Sai Kishore
  • 326
  • 1
  • 9
  • 16
0

This is what worked for me: Download the latest winutils https://github.com/kontext-tech/winutils Or check your spark Release text, it shows the cer of Hadoop it is using

Steps

  1. Dowload repo

  2. Create a folder named hadoop anywhere (e.g. desktop/hadoop)

  3. Paste the bin into that folder (you will then have hadoop/bin)

  4. copy hadoop.dll to windows/system32

  5. Set system environment:

    set HADOOP_HOME=c:/desktop/hadoop 
set PATH=%PATH%;%HADOOP_HOME%/bin;
Ardan
  • 1
  • 2