Dangling hash node ref: too many open files

Question

I'm running two supernodes with the geth client. At peak times they have 500-800 connections and handle around 16GB traffic per hour.

This is how I run the node:

geth --networkid "1" --identity "5chdn-supernode-deneb" --maxpeers "1024" --rpc console 2>>/tmp/geth.log

Now it keeps crashing every few days with a too many open files error on the chaindata:

E0405 10:04:40.631505   17793 trie.go:309] Dangling hash node ref 3c356161363764616266326264373164613363306632313861373037393930376533633962616537313338333732383161623537656534333063396630363636313e20: open /home/user/.ethereum/chaindata/100673.ldb: too many open files
E0405 10:04:40.632653   17793 trie.go:309] Dangling hash node ref 3c386639363738343930626135663333623135656434616339663135373938623530313564393533323664626639333865383237623633383164616232666166333e20: open /home/user/.ethereum/chaindata/100596.ldb: too many open files
E0405 10:04:40.634643   17793 trie.go:309] Dangling hash node ref 3c353531373463643738393431336166666338313331633038313736656531663039383735326130353034306533326162323563663762363137376664633662643e20: open /home/user/.ethereum/chaindata/100662.ldb: too many open files
E0405 10:04:40.634690   17793 trie.go:309] Dangling hash node ref 3c353531373463643738393431336166666338313331633038313736656531663039383735326130353034306533326162323563663762363137376664633662643e20: open /home/user/.ethereum/chaindata/100662.ldb: too many open files
E0405 10:04:40.634817   17793 trie.go:309] Dangling hash node ref 3c353531373463643738393431336166666338313331633038313736656531663039383735326130353034306533326162323563663762363137376664633662643e20: open /home/user/.ethereum/chaindata/100662.ldb: too many open files
E0405 10:04:40.635057   17793 trie.go:309] Dangling hash node ref 3c643533653331616464343066653864633666363766373530646462626435636565306233383337643539643639626562636463356138613662366465346432633e20: open /home/user/.ethereum/chaindata/100627.ldb: too many open files
E0405 10:04:40.639013   17793 trie.go:309] Dangling hash node ref 3c633838383230656236353763303762663366366335343133386635366664656337323030623166333361363464353862306464393639636137643964383738373e20: open /home/user/.ethereum/chaindata/100622.ldb: too many open files
E0405 10:04:40.639076   17793 trie.go:309] Dangling hash node ref 3c633838383230656236353763303762663366366335343133386635366664656337323030623166333361363464353862306464393639636137643964383738373e20: open /home/user/.ethereum/chaindata/100622.ldb: too many open files
E0405 10:04:40.639281   17793 trie.go:309] Dangling hash node ref 3c633838383230656236353763303762663366366335343133386635366664656337323030623166333361363464353862306464393639636137643964383738373e20: open /home/user/.ethereum/chaindata/100622.ldb: too many open files
E0405 10:04:40.639367   17793 trie.go:309] Dangling hash node ref 3c323762626362616166306236663131386230333534313862303831373539653935616637333532383564323337656263623262643734363638643836333733363e20: open /home/user/.ethereum/chaindata/100219.ldb: too many open files
E0405 10:04:40.640494   17793 trie.go:309] Dangling hash node ref 3c326166613631323661333034656265356538643265633165623130316135366561303932373566373139353736306234373965313266663065323930643864643e20: open /home/user/.ethereum/chaindata/100544.ldb: too many open files
E0405 10:04:40.640977   17793 trie.go:309] Dangling hash node ref 3c613362623765633164313033316464633163366536316438313534396265613639633536336632653431353233653739633833303232303238343263373163633e20: open /home/user/.ethereum/chaindata/100605.ldb: too many open files
E0405 10:04:40.644731   17793 trie.go:309] Dangling hash node ref 3c316139653230373531643062346136383163323863616631353461376633396334643165623037396464646163343834393465343532393961396437663531363e20: open /home/user/.ethereum/chaindata/100065.ldb: too many open files
E0405 10:04:40.644986   17793 trie.go:309] Dangling hash node ref 3c663633343166326138393366313636356461336235353732666137386535373839626161386237373633643534653063623865303336396233343035666665333e20: open /home/user/.ethereum/chaindata/100638.ldb: too many open files
E0405 10:04:40.645818   17793 trie.go:309] Dangling hash node ref 3c323361343762373937356438626363636638643431616336666130336332653939353539646434396436336463333330656138353638323466663763656337383e20: open /home/user/.ethereum/chaindata/100211.ldb: too many open files
E0405 10:04:40.645878   17793 trie.go:309] Dangling hash node ref 3c323361343762373937356438626363636638643431616336666130336332653939353539646434396436336463333330656138353638323466663763656337383e20: open /home/user/.ethereum/chaindata/100211.ldb: too many open files
E0405 10:04:40.646031   17793 trie.go:309] Dangling hash node ref 3c323361343762373937356438626363636638643431616336666130336332653939353539646434396436336463333330656138353638323466663763656337383e20: open /home/user/.ethereum/chaindata/100211.ldb: too many open files
E0405 10:04:40.649087   17793 trie.go:309] Dangling hash node ref 3c356231383235393062666662663436643061653261303530653430313738333435616437646630303334643062666538333231343831396238343730663464613e20: open /home/user/.ethereum/chaindata/100675.ldb: too many open files
E0405 10:04:40.649151   17793 trie.go:309] Dangling hash node ref 3c356231383235393062666662663436643061653261303530653430313738333435616437646630303334643062666538333231343831396238343730663464613e20: open /home/user/.ethereum/chaindata/100675.ldb: too many open files
E0405 10:04:40.649349   17793 trie.go:309] Dangling hash node ref 3c356231383235393062666662663436643061653261303530653430313738333435616437646630303334643062666538333231343831396238343730663464613e20: open /home/user/.ethereum/chaindata/100675.ldb: too many open files
E0405 10:04:40.650758   17793 trie.go:309] Dangling hash node ref 3c393434663365613033363436313739663734326635636365366139323430343466383139663334363838386239643034323936336339656666626361396231643e20: open /home/user/.ethereum/chaindata/100598.ldb: too many open files
E0405 10:04:40.651084   17793 trie.go:309] Dangling hash node ref 3c353864343237306166363438373062643739626639363465653139363331633132376336376538356161326531316333636264383037343730333464306663363e20: open /home/user/.ethereum/chaindata/100669.ldb: too many open files
E0405 10:04:40.651348   17793 trie.go:309] Dangling hash node ref 3c633231643033396639393134353864316430336366666134663063653366636161363466383361616434646436346139343635333731356431663764666266373e20: open /home/user/.ethereum/chaindata/100620.ldb: too many open files
E0405 10:04:40.663574   17793 trie.go:309] Dangling hash node ref 3c663633343166326138393366313636356461336235353732666137386535373839626161386237373633643534653063623865303336396233343035666665333e20: open /home/user/.ethereum/chaindata/100638.ldb: too many open files
E0405 10:04:40.665738   17793 trie.go:309] Dangling hash node ref 3c346461643532333134616566613162313039613339626235373532376531326635653333366232303438333635333233623565663536636334383464613862323e20: open /home/user/.ethereum/chaindata/100643.ldb: too many open files
E0405 10:04:40.667364   17793 trie.go:309] Dangling hash node ref 3c343336383939613631616334336633396333366561636334353339376562643436396538303763356464356264623561653736373431663061306364346565333e20: open /home/user/.ethereum/chaindata/100511.ldb: too many open files
E0405 10:04:40.669370   17793 trie.go:309] Dangling hash node ref 3c356539623665376230376532643838643535626336343765646562653931303638613531646466363332333761323863303162623865323332373637666261313e20: open /home/user/.ethereum/chaindata/100683.ldb: too many open files
E0405 10:04:40.669724   17793 trie.go:309] Dangling hash node ref 3c383761313431313131303938646665613533393165353036653239383163373135663835633165336434366633353137396633363134333437623733383761323e20: open /home/user/.ethereum/chaindata/100593.ldb: too many open files
E0405 10:04:40.671738   17793 trie.go:309] Dangling hash node ref 3c366538653865633965616466356437313832663431333065666362623566346330663734326232653336383038306364643231646239363734356339663433383e20: open /home/user/.ethereum/chaindata/100574.ldb: too many open files

Why is that? How to fix this? I want to run a stable node.

Answer 1

Check Open File Descriptor Limits

You may need to increase the Linux file limit to handle the number of connections your nodes are serving. Here are some articles on checking and setting the limits. There are hard and soft, system wide and per user limits.

To check the maximum number of file descriptors configured on your Linux system:

user@Kumquat:~$ cat /proc/sys/fs/file-max
793317

To check a user's maximum number of open file descriptors (hard limit):

user@Kumquat:~$ ulimit -Hn
4096

To check a user's maximum number of open file descriptors (soft limit):

user@Kumquat:~$ ulimit -Sn
1024

To check the limits for a geth instance:

user@Kumquat:~$ ps -ef | grep geth
user      6492  6479 20 09:53 pts/6    00:00:01 geth console
user      6511 30948  0 09:53 pts/2    00:00:00 grep --color=auto geth
user@Kumquat:~$ cat /proc/6492/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             30995                30995                processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       30995                30995                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

To check the number of open file descriptors used by the geth instance:

user@Kumquat:~$ lsof -p 6492 | wc -l
78
user@Kumquat:~$ lsof -p 6492
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
geth    6492  user cwd    DIR              252,1     4096 18748035 /home/user/ESE/WhoisBug
geth    6492  user rtd    DIR              252,1     4096        2 /
geth    6492  user txt    REG              252,1 17848784  7734154 /usr/bin/geth
geth    6492  user mem    REG              252,1    47712 21894327 /lib/x86_64-linux-gnu/libnss_files-2.19.so
geth    6492  user mem    REG              252,1    47760 21894319 /lib/x86_64-linux-gnu/libnss_nis-2.19.so
geth    6492  user mem    REG              252,1    97296 21894315 /lib/x86_64-linux-gnu/libnsl-2.19.so
geth    6492  user mem    REG              252,1    39824 21890648 /lib/x86_64-linux-gnu/libnss_compat-2.19.so
geth    6492  user mem    REG              252,1  1840928 21894324 /lib/x86_64-linux-gnu/libc-2.19.so
geth    6492  user mem    REG              252,1   141574 21894316 /lib/x86_64-linux-gnu/libpthread-2.19.so
geth    6492  user mem    REG              252,1  1071552 21890641 /lib/x86_64-linux-gnu/libm-2.19.so
geth    6492  user mem    REG              252,1   149120 21894317 /lib/x86_64-linux-gnu/ld-2.19.so
geth    6492  user   0u   CHR              136,6      0t0        9 /dev/pts/6
geth    6492  user   1u   CHR              136,6      0t0        9 /dev/pts/6
geth    6492  user   2u   CHR              136,6      0t0        9 /dev/pts/6
geth    6492  user   3uW  REG              252,1        0 17305945 /home/user/.ethereum/chaindata/LOCK
geth    6492  user   4w   REG              252,1   445185 17305532 /home/user/.ethereum/chaindata/LOG
geth    6492  user   6u  0000               0,10        0     8681 anon_inode
geth    6492  user   7u  IPv4             301021      0t0      TCP 192-168-1-14.tpgi.com.au:36388->77.70.125.246:30303 (SYN_SENT)
geth    6492  user   8uW  REG              252,1        0 17175139 /home/user/.ethereum/nodes/LOCK
geth    6492  user   9w   REG              252,1     1288 17171201 /home/user/.ethereum/nodes/LOG
geth    6492  user  10w   REG              252,1   899579 17313769 /home/user/.ethereum/chaindata/MANIFEST-656380
geth    6492  user  12u  IPv4             301019      0t0      TCP 192-168-1-14.tpgi.com.au:51259->pool-98-113-173-46.nycmny.fios.verizon.net:30303 (ESTABLISHED)
geth    6492  user  13r   REG              252,1  1752712 17313448 /home/user/.ethereum/chaindata/656104.ldb
geth    6492  user  14u  IPv4             300637      0t0      TCP 192-168-1-14.tpgi.com.au:57807->cpc78189-warw17-2-0-cust572.3-2.cable.virginm.net:30303 (ESTABLISHED)
geth    6492  user  15u  IPv4             301126      0t0      TCP 192-168-1-14.tpgi.com.au:47302->195-154-165-137.rev.poneytelecom.eu:30303 (ESTABLISHED)
geth    6492  user  16r   REG              252,1    22345 17313871 /home/user/.ethereum/chaindata/656914.ldb
geth    6492  user  17uW  REG              252,1        0 17047070 /home/user/.ethereum/dapp/LOCK
geth    6492  user  18w   REG              252,1      396 17039394 /home/user/.ethereum/dapp/LOG
geth    6492  user  19w   REG              252,1        0 17040771 /home/user/.ethereum/dapp/000420.log
geth    6492  user  20w   REG              252,1       43 17040772 /home/user/.ethereum/dapp/MANIFEST-000421
geth    6492  user  21u  IPv4             302150      0t0      TCP 192-168-1-14.tpgi.com.au:58595->adsl-203.91.140.34.tellas.gr:30303 (ESTABLISHED)
geth    6492  user  23r   REG              252,1  2109837 17305672 /home/user/.ethereum/chaindata/481514.ldb
geth    6492  user  24u  IPv4             298824      0t0      TCP 192-168-1-14.tpgi.com.au:36823->user-24-96-159-205.knology.net:30303 (ESTABLISHED)
geth    6492  user  26u  IPv6             299586      0t0      UDP *:30303 
geth    6492  user  27w   REG              252,1   581824 17171210 /home/user/.ethereum/nodes/001081.log
geth    6492  user  28w   REG              252,1     2049 17171217 /home/user/.ethereum/nodes/MANIFEST-001082
geth    6492  user  29r   REG              252,1  2128363 17171214 /home/user/.ethereum/nodes/001083.ldb
geth    6492  user  30u  IPv6             297760      0t0      TCP *:30303 (LISTEN)
geth    6492  user  31u  unix 0x0000000000000000      0t0   297761 /home/user/.ethereum/geth.ipc
geth    6492  user  32r   REG              252,1  2124899 17171216 /home/user/.ethereum/nodes/001084.ldb
geth    6492  user  33r   REG              252,1  2125504 17171225 /home/user/.ethereum/nodes/001085.ldb
geth    6492  user  34r   REG              252,1  2121011 17171229 /home/user/.ethereum/nodes/001086.ldb
geth    6492  user  35r   REG              252,1  2124190 17313829 /home/user/.ethereum/chaindata/658913.ldb
geth    6492  user  37u  IPv4             298768      0t0      TCP 192-168-1-14.tpgi.com.au:53786->sky.loxal.net:30303 (ESTABLISHED)
geth    6492  user  38u  IPv4             299790      0t0      TCP 192-168-1-14.tpgi.com.au:40468->static.177.39.9.176.clients.your-server.de:30303 (ESTABLISHED)
geth    6492  user  39r   REG              252,1    18099 17313873 /home/user/.ethereum/chaindata/656916.ldb
geth    6492  user  40r   REG              252,1  1709208 17313834 /home/user/.ethereum/chaindata/658914.ldb
geth    6492  user  42u  IPv4             297767      0t0      TCP 192-168-1-14.tpgi.com.au:40764->63.224.55.73:30303 (ESTABLISHED)
geth    6492  user  45r   REG              252,1  2129807 17310863 /home/user/.ethereum/chaindata/656209.ldb
geth    6492  user  46r   REG              252,1  2129957 17306835 /home/user/.ethereum/chaindata/656245.ldb
geth    6492  user  48u  IPv4             301020      0t0      TCP 192-168-1-14.tpgi.com.au:41864->217.96.253.11.ipv4.supernova.orange.pl:30303 (SYN_SENT)
geth    6492  user  49u  IPv4             300960      0t0      TCP 192-168-1-14.tpgi.com.au:52510->41.142.88.193:30303 (ESTABLISHED)
geth    6492  user  51r   REG              252,1  2145665 17309081 /home/user/.ethereum/chaindata/656168.ldb
geth    6492  user  52r   REG              252,1    12749 17313877 /home/user/.ethereum/chaindata/656920.ldb
geth    6492  user  53w   REG              252,1  3007664 17313949 /home/user/.ethereum/chaindata/658851.log
geth    6492  user  54r   REG              252,1  2130155 17312570 /home/user/.ethereum/chaindata/655955.ldb
geth    6492  user  55r   REG              252,1  2131971 17313153 /home/user/.ethereum/chaindata/655988.ldb
geth    6492  user  57r   REG              252,1  2130863 17313239 /home/user/.ethereum/chaindata/656279.ldb
geth    6492  user  58r   REG              252,1    28822 17313865 /home/user/.ethereum/chaindata/656908.ldb
geth    6492  user  59r   REG              252,1    25844 17313876 /home/user/.ethereum/chaindata/656919.ldb
geth    6492  user  64r   REG              252,1  2130876 17313198 /home/user/.ethereum/chaindata/656361.ldb
geth    6492  user  65r   REG              252,1    42947 17313875 /home/user/.ethereum/chaindata/656918.ldb
geth    6492  user  67r   REG              252,1  2129128 17312345 /home/user/.ethereum/chaindata/656240.ldb
geth    6492  user  68u  IPv4             300629      0t0      TCP 192-168-1-14.tpgi.com.au:39338->c-76-101-62-156.hsd1.fl.comcast.net:30303 (ESTABLISHED)
geth    6492  user  69r   REG              252,1     8376 17313861 /home/user/.ethereum/chaindata/656904.ldb

A faster way to check the number of open file descriptors used by the geth instance:

user@Kumquat:~$ ls /proc/6491/fd
0  10  12  14  16  18  2   24  26  28  3   31  33  35  37  39  40  43  45  47  50  53  55  6   63  65  68  7  9
1  11  13  15  17  19  20  25  27  29  30  32  34  36  38  4   41  44  46  5   51  54  56  60  64  67  69  8
user@Kumquat:~$ ls /proc/6491/fd | wc -l
66

Change Soft Limit For Geth Instance

You can change the soft limit for geth by running the ulimit command prior to starting geth. The maximum soft limit number is the hard limit number.

user@Kumquat:~$ ulimit -n 5000
bash: ulimit: open files: cannot modify limit: Operation not permitted
# Can only change soft limit up to the hard limit
user@Kumquat:~$ ulimit -n 4000
user@Kumquat:~$ geth console
# In a separate window
user@Kumquat:~$ $ ps -ef | grep geth
user      6649  6479 14 10:14 pts/6    00:00:01 geth console
user      6667 30948  0 10:14 pts/2    00:00:00 grep --color=auto geth
user@Kumquat:~$ cat /proc/6649/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             30995                30995                processes 
Max open files            4000                 4000                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       30995                30995                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

If you only require a change to the soft limit below the default 4096 hard limit, create a bash script file to run geth and add the ulimit command just before starting geth.

Change Hard Limit For User

In the previous section, you cannot change the soft limit above the hard limit. To change the hard limit, add the following to /etc/security/limits.conf:

* soft nofile 14096
* hard nofile 14096

Note that * will change the limits for all non-root users. To change the limit for the root user, substitute * with 'root'. For example:

* soft nofile 14096
* hard nofile 14096
root soft nofile 24096
root hard nofile 24096

For the above to take effect, you will have to edit /etc/pam.d/common-session* (more than one file) and add:

session required pam_limits.so

A logout / login, or a reboot may be required for the changes to work.

Decrease Number Of Simultaneous Network Connections

You could alternatively decrease the number of peer connections that each of your nodes handle with the --maxpeers parameter and your open file descriptor limits may not be exceeded.

Network connections also fall under the file limits. See:

• https://unix.stackexchange.com/questions/157351/why-are-tcp-ip-sockets-considered-open-files

Potentially Corrupted Files

There is a possibility that you may have some corrupted files due to the file limits being exceeded. This depends on how the trie.go code handles the error condition when it needs to open the files and fails, and whether the file was opened for reads and/or writes.

Potentially A Bug

The issue you are experiencing may be caused by too many files descriptors being opened at the one time to serve the current traffic, or it may be due to the file descriptors not being closed properly. If it is the latter, then increasing the limits will just delay the problem occurring.

To test this, you can set up a cron job to periodically record the date, count of the opened file descriptors and the count of network connections and you should see an increasing trend in opened file descriptors if there is a potential bug.

Some references

• http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
• https://underyx.me/2015/05/18/raising-the-maximum-number-of-file-descriptors
• https://askubuntu.com/questions/162229/how-do-i-increase-the-open-files-limit-for-a-non-root-user