3

I know that the state of the txpool of a geth node can be queried using the txpool API. However, I was wondering if it is somehow possible to subscribe to new transactions entering a geth node's txpool, possibly on a websockets or local socket rpc connection? I'd like to write the program that subscribes to new txpool transactions in Go as well.

I discovered this subscription in go-ethereum internally, but it doesn't look like it's exposed, unfortunately.

Note that I'm not only talking about local pending transactions (answered here), but all transactions arriving to the txpool.

Background: I'd like to do research on frontrunning. The authors of the Flash Boys 2.0 paper released their geth fork on github, which they used to observe gas auctions in the txpool. However, using a fork is not as reusable and maintainable than having the possibility to extend geth's API so that external applications can subscribe to txpool udpates.

sebastian
  • 151
  • 1
  • 5

4 Answers4

6

We have been running ethviewer.live since late 2017 with a modified geth client. It has certainly been painful to keep up with latest versions of geth. Also, we have had outages when we were unable to keep up with forks. As suggested by @Eugene I tried with web3.eth.subscribe('pendingTransactions') and it seems to be working fine. One caveat though -- Subscription API on Geth is best-effort, which means it may occasionally miss out a TX or block. If you need all data, you need to do state management and pull any missing data using Web3.eth functions. As per GitHub discussions, Geth may not fix this, while Parity seems to fix it.

As you are interested in frontrunning, be aware that a node gets large fraction of transactions after the block containing them is received. For e.g., with --maxpeers = 50 on ethviewer.live about 40% of transactions arrived after the block containing them arrived (i.e., either block timestamp or arrival time < transaction arrival time). I also checked by dumping blockchair.com transaction pool and realized the fraction is about 55% (don't know what they used as --maxpeers). These measurements were done for several days. So you may want to play with high --- maxpeers values in addition to having good bandwidth.

Dilum Bandara
  • 111
  • 1
  • 6
  • How did you go about testing your claim that 40% of txs arrived after the block containing them arrived? – samlaf Mar 03 '22 at 04:06
  • We locally log the time that block and TX announcements are received by our node. For a ~2 months trace, ~40% of the TXs were received by our node after the block that contain them were received, i.e., block had an earlier local timestamp than TXs. I tried with --maxpeers = 25 and = 5. Didn't observe a significant improvement. – Dilum Bandara Mar 05 '22 at 05:33
  • @DilumBandara I have experienced the same phenomen. About 35-40% come in to late, even by more than 10 seconds! Have you experimented with nodes in different locations around the world? Is there any reason to explain this? Looking at https://ethereum.stackexchange.com/a/28679 it should not be an issue to receive all but a few outliers in time. – Plus Ultra Aug 08 '22 at 11:50
  • @PlusUltra We haven't tried it across geographies. I think it comes down to a factors such as bandwidth, peers connected to, and transaction originating peer. We have seen sometimes 200+ new transactions arrive to the pool within a 1 sec. Given transaction arrival is bursty, network is massive, we are connected only to 25 nodes (out of 5,000+), and inter-block time for some blocks is 2-5 sec, there's no way a gossip protocol is going to disseminate them before the next block arrives. There's an error in my previous comment. It should be "I tried with --maxpeers = 25 and = 50" – Dilum Bandara Aug 09 '22 at 21:06
4

It's a bit counterintuitive, but while web3.eth.getPendingTransactions() returns only local pending transactions (from transaction field value matching accounts within the node), web3.eth.subscribe('pendingTransactions') will return a stream of global pending transactions.

Please find usage examples here.

Eugene Aseev
  • 320
  • 1
  • 7
  • 1
    Thanks, didn't know that subscribe('pendingTransactions') does cover all incoming txs in the pool. Unfortunately, this doesn't seem to be exposed in go-ethereums ethclient. – sebastian Nov 11 '20 at 16:43
  • Eugene, I tried your examples from the link. I am connected directly on a node via icp. The problem is, I get only a stream of tx which are already mined. Not pending anymore. I have to say I do not use the geth client but opera of fantom. Do you have an idea if this is a general setting maybe somewhere which can be changed? – flo Oct 07 '21 at 14:12
  • How does the node get access to global pending txs without them also becoming local? Is it because subscribe(pendingTransactions) queries neighboring nodes' txpools, which might have larger sizes and not fit in the current node's local mempool? – samlaf Mar 03 '22 at 04:04
3

Supplement to Eugene's answer, I have this fiddle testing the speed of two nodes catching pending transactions. It was built with web3.subscription. Hope it helps.

Explanation of the code is also here.

Wuzhong - Chainstack DevEx
  • 1,424
  • 2
  • 10
1

You can create an account with Infura and subscribe to their pending pool as easy as this:

const { time } = require("console");
const Web3 = require("web3");
let fs = require('fs');

const web = new Web3();


const rpcUrl = 'INFURA NODE RPC'


const web3 = new Web3(rpcUrl)


const message = fs.createWriteStream("./pendingPool.txt");


message.write("tx-hash" + ' ' + "Arrival" + "\n");


var subscription = web3.eth.subscribe('pendingTransactions', function (error, result) {
    if (error)
        console.log(error)
})
    .on("data", function (result) {
        message.write(result + ' ' + Date.now() + "\n");
    });

This code listens to the pending pool and writes the tx and arrival time to a file.

Emrah
  • 1,654
  • 2
  • 9
  • 24