5

I have 4 nodes (each on a separate droplet on digital ocean) in a private network:

  • bootnode
  • miner1
  • miner2
  • jsonrpc

Each node is initialized with geth --datadir ./data init ./genesis.json using the exact same genesis.json

{
    "config": {
      "chainId": 55055,
      "homesteadBlock": 0,
      "eip150Block": 0,
      "eip155Block": 0,
      "eip158Block": 0,
      "byzantiumBlock": 0,
      "constantinopleBlock": 0,
      "petersburgBlock": 0,
      "ethash": {}
    },
    "difficulty": "1",
    "gasLimit": "12000000",
    "alloc": {}
}

I generate the bootnode.key with bootnode -genkey bootnode.key and start the bootnode with

geth --datadir ./data --nodekey bootnode.key --nodiscover --ipcdisable 
    \ --networkid 55055 --identity bootnode console

The jsonrpc node with

geth --datadir ./data --bootnodes $ENODE_ADDRESS --allow-insecure-unlock --http 
    \ --http.addr="0.0.0.0" --http.api="eth,web3,net,admin,personal" 
    \ --http.corsdomain="*" --identity jsonrpc --networkid 55055 console

where the $ENODE_ADDRESS is the enode address of the bootnode, e.g. enode://pubkey@ip:30303

The miner nodes with

geth --datadir ./data --bootnodes $ENODE_ADDRESS --mine --miner.threads 1 
    \ --miner.etherbase $ACCOUNT --networkid 55055 --identity miner console

where $ACCOUNT is an existing ETH wallet address. p.s. the identity is "miner2" on the other miner node.

With this setup everything starts correctly, the jsonrpc and the miners connect to the bootnode. I can make HTTP requests (e.g. via PostMan) to the public IP of the jsonrpc node and I get the expected HTTP response. I can even connect MetaMask to the chain and the wallet balance of those accounts specified to the miners increase (with each mined block).

The only issue I have is that the nodes do NOT connect to each other. net.listening is true on each. But net.peerCount is 1 on the jsonrpc and the miner nodes (it is 3 on the bootnode as expected).

I can add peer manually via admin.addPeer. I tried to add miner2 to miner1 and it worked, net.peerCount now shows 2 on each of the miner nodes (and admin.peers shows the correct peers = [bootnode, miner1] on miner2 and [bootnode, miner2] on miner1).

But ... despite the jsonrpc and miner nodes being connected to the bootnode, they just don't find each other. The console says "Looking for peers" on regular basis but finds/adds nothing.

The console on the jsonrpc server does show the "Snapshot extension registration failed ... peer connected on snap without compatible eth support" error from time to time (as well as the "Server parity_netPeers" warning)

Any ideas how I can get the nodes to find (and connected to) each other after they connect to the bootnode?

p.s. I've read the following questions/answers (which have not helped)

Edit #1: I can see the following console output on regular basis

INFO [02-11|10:03:43.180] Looking for peers     peercount=2 tried=25 static=0
> net.peerCount
1
> ERROR[02-11|10:03:50.094] Snapshot extension registration failed
   peer=84bfdb1d err="peer connected on snap without compatible eth support"

i.e. the INFO output gives peercount=2 but the net.peerCount is actualy 1

Darth Thanatus
  • 76
  • 6
  • 1
    Interesting, I see you have read pretty much all "duplicates" but I believe that based on what I've read here, it is not possible to have peer discovery (even through a bootnode) on a private network – scibuff Feb 12 '22 at 20:32
  • Wow, that seems like a serious limitation for a private network ... but I'm actually planning on changing the client source code so that could be a solution :) – Darth Thanatus Feb 12 '22 at 20:33
  • 1
    Hey did you get more info about that issue? I'm facing the same challenge and your info could really help me. And what do you mean by changing the client, you mean changing ethereum source code or opting for a complete different blockchain? Thank you very much!!! – Kevin Wad Mar 01 '22 at 11:58
  • I haven't yet tried the --nat exip:<external-ip> suggest below but the only way I could get the nodes to connect was to add peers manually (e.g. admin.addPeer or via static-nodes.json). And yes, I'll be changing the ethereum source for our private network so I guess I can just change the hardcoded enodes – Darth Thanatus Mar 07 '22 at 22:22
  • I also noticed this very same behavior with my test network (1 bootnode, 1 signer, and 1 rpc). Unfortunately, the --nat exip:<external-ip> option did not resolve this issue. – weeix Mar 27 '22 at 10:54

2 Answers2

0

Try using --nat exip:<external-ip>, where the external IP is specified so that the nodes find reach other.

Rilke Petrosky
  • 101
  • 1
0

Had similar setup and the same problem but have found some clues to this. Same experience with peering, i.e. bootnode had peers but the worker nodes did not. Initially ran a cluster in docker locally without issues but didn't look too closely as my transactions were being processed.

Deployed the setup to a kubernetes cluster and started having issues with peering. Got ethstats running in the cluster for observability and noticed my transactions were not propagating from my rpc node to the miner nodes due to peering was my best guess although as far as I could tell the txpool was receiving the transactions so don't know why the nodes couldn't pick them up.

Finally got peering to work by turning off --nodiscover on the bootnode and adding a netrestrict policy to all nodes + service origins cidr's to the worker nodes to protect them. Result was all my expected nodes peered but I have a bunch of noise for invalid neighbours which I'm guessing is related to geth having some logic to do some discovery of it's own when --nodiscover flag is turned off.

So my best guess on this from observation is that --nodiscover behaviour is propagated to connected peers, causing them to ignore the peer information container on the bootnode. Turning this off with and adding the --netrestrict cidr is giving me the desired behaviour but I don't know yet if there are any negative consequences for this. Perhaps an expert on this could comment?

Colm Ryan
  • 1