ETH Full Node and Pathfinder
Based on the optimization methods, our node processing speed is significantly faster than Alchemy's: we can handle 58.20 requests per second, while Alchemy only manages 37.88 requests per second.
When working on projects, we frequently come across situations where we need to input the ETH RPC or to retrieve information about a particular token locally. For instance, in the code snippet below, our objective is to obtain the decimal information of a contract:
package main
import (
"fmt"
"github.com/ethereum/go-ethereum/common"
"github.com/ethereum/go-ethereum/ethclient"
"github.com/reddio-com/standard-contracts/erc20"
)
func main() {
client, err := ethclient.Dial("<https://goerli.infura.io/v3/SOME_TOKEN_HERE>")
if err != nil {
panic(err)
}
contractAddress := common.HexToAddress("0x07865c6E87B9F70255377e024ace6630C1Eaa37F")
// Query baseuri
erc20, err := erc20.NewErc20(contractAddress, client)
if err != nil {
panic(err)
}
// Query BaseURI
decimals, err := erc20.Decimals(nil)
if err != nil {
panic(err)
}
fmt.Println("Decimals: ", decimals)
totalSupply, err := erc20.TotalSupply(nil)
if err != nil {
panic(err)
}
fmt.Println("TotalSupply: ", totalSupply)
}
The ethclient.Dial("<https://goerli.infura.io/v3/SOME_TOKEN_HERE>") is an ETH RPC Endpoint. It allows us to interact with events on the Ethereum chain through HTTP.
Infura, developed by Consensys, is an infrastructure service that simplifies Ethereum blockchain interaction for developers. By using Infura, developers can avoid the complexity and cost of setting up and maintaining their own Ethereum full node. Infura offers a range of APIs that enable developers to access Ethereum's data and functionality via remote nodes. This means they don't have to run their own Ethereum node but can rely on Infura's nodes for communication with the blockchain. As a result, developers can focus on building core functionalities for their applications without investing significant time and effort in node operations and maintenance. However, Infura is a paid service with the following pricing plans:
If your service requires a lot of interaction with the blockchain or it needs to analyze historical data, using Infura can be very expensive. In this case, you can consider setting up your own ETH full node so that you can retrieve the required data from a local node.
Setting up your own Ethereum full node has several advantages. Firstly, it provides higher data privacy and security as developers have complete control over the node's operating environment and access permissions. Secondly, self-built nodes allow developers to customize the configuration of the node to meet specific requirements, such as adjusting synchronization speed or storage capacity. Additionally, in some cases, self-built nodes may be more reliable as developers do not rely on the stability of third-party services.
However, there are also challenges associated with self-built nodes including hardware and network requirements, time and resources needed for node synchronization, as well as operational and maintenance costs. In comparison, Infura offers a more convenient and flexible option that allows developers to quickly launch their projects and focus on core development tasks without having to worry too much about managing underlying nodes. Taking all factors into consideration, developers can choose between using Infura or setting up their own nodes based on their specific needs and priorities when interacting with the Ethereum blockchain.
Geth is an execution client. Historically, having just one execution client was sufficient to run a full Ethereum node. However, since Ethereum switched from Proof of Work (PoW) to Proof of Stake (PoS) consensus mechanism, Geth needs to work in conjunction with another software called "consensus client".
From https://geth.ethereum.org/docs/getting-started/consensus-clients, we can find that the official recommendation for consensus clients is as follows:
- Lighthouse: written in Rust
- Nimbus: written in Nim
- Prysm: written in Go
- Teku: written in Java
- Lodestar: written in Typescript
The launch of the consensus client requires the execution client to support. In this case, we have chosen Prysm as the consensus client because we are using Geth, which is an official execution client.
At Reddio, all of our services are containerized, which reduces the hassle of dealing with configuration environments. Therefore, in this article, all examples will be written using docker-compose.
Firstly, prepare a machine with a sufficiently large disk (>900G SSD) and enough memory (>32G). Create an empty directory and then create a docker-compose.yml file with the following content:
version: "3"
services:
geth:
image: ethereum/client-go:v1.12.2
restart: unless-stopped
ports:
- 30303:30303
- 30303:30303/udp
- 127.0.0.1:8545:8545
- 127.0.0.1:8546:8546
- 127.0.0.1:8551:8551
volumes:
- ./data:/root/.ethereum
healthcheck:
test: [ "CMD-SHELL", "geth attach --exec eth.blockNumber" ]
interval: 10s
timeout: 5s
retries: 5
command:
- --http
- --cache=8192
- --http.api=eth,net,web3,engine,admin
- --http.addr=0.0.0.0
- --http.vhosts=*
- --http.corsdomain=*
- --maxpeers=200
- --ws
- --ws.origins=*
- --ws.addr=0.0.0.0
- --ws.api=eth,net,web3
- --graphql
- --graphql.corsdomain=*
- --graphql.vhosts=*
- --authrpc.addr=0.0.0.0
- --authrpc.jwtsecret=/root/.ethereum/jwt.hex
- --authrpc.vhosts=*
- --authrpc.port=8551
- --txlookuplimit=0
You can start the service using:
docker-compose up -d
At this point, Geth should have started up successfully.
INFO [08-17|06:43:02.867] Starting Geth on Ethereum mainnet...
INFO [08-17|06:43:02.870] Maximum peer count ETH=200 LES=0 total=200
INFO [08-17|06:43:02.872] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [08-17|06:43:02.874] Sanitizing cache to Go's GC limits provided=8192 updated=2579
INFO [08-17|06:43:02.875] Set global gas cap cap=50,000,000
INFO [08-17|06:43:02.877] Initializing the KZG library backend=gokzg
INFO [08-17|06:43:03.005] Allocated trie memory caches clean=386.00MiB dirty=644.00MiB
INFO [08-17|06:43:03.006] Defaulting to pebble as the backing database
INFO [08-17|06:43:03.006] Allocated cache and file handles database=/root/.ethereum/geth/chaindata cache=1.26GiB handles=524,288
INFO [08-17|06:43:03.022] Opened ancient database database=/root/.ethereum/geth/chaindata/ancient/chain readonly=false
INFO [08-17|06:43:03.023] Initialising Ethereum protocol network=1 dbversion=<nil>
INFO [08-17|06:43:03.024] Writing default main-net genesis block
INFO [08-17|06:43:03.664] Persisted trie from memory database nodes=12356 size=1.79MiB time=95.551853ms gcnodes=0 gcsize=0.00B gctime=0s livenodes=0 livesize=0.00B
INFO [08-17|06:43:03.718]
INFO [08-17|06:43:03.718] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-17|06:43:03.719] Chain ID: 1 (mainnet)
INFO [08-17|06:43:03.719] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [08-17|06:43:03.719]
INFO [08-17|06:43:03.719] Pre-Merge hard forks (block based):
INFO [08-17|06:43:03.719] - Homestead: #1150000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md>)
INFO [08-17|06:43:03.719] - DAO Fork: #1920000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/dao-fork.md>)
INFO [08-17|06:43:03.719] - Tangerine Whistle (EIP 150): #2463000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md>)
INFO [08-17|06:43:03.719] - Spurious Dragon/1 (EIP 155): #2675000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md>)
INFO [08-17|06:43:03.719] - Spurious Dragon/2 (EIP 158): #2675000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md>)
INFO [08-17|06:43:03.719] - Byzantium: #4370000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md>)
INFO [08-17|06:43:03.719] - Constantinople: #7280000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md>)
INFO [08-17|06:43:03.719] - Petersburg: #7280000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md>)
INFO [08-17|06:43:03.719] - Istanbul: #9069000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md>)
INFO [08-17|06:43:03.719] - Muir Glacier: #9200000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md>)
INFO [08-17|06:43:03.719] - Berlin: #12244000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md>)
INFO [08-17|06:43:03.719] - London: #12965000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md>)
INFO [08-17|06:43:03.719] - Arrow Glacier: #13773000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md>)
INFO [08-17|06:43:03.719] - Gray Glacier: #15050000 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md>)
INFO [08-17|06:43:03.719]
INFO [08-17|06:43:03.719] Merge configured:
INFO [08-17|06:43:03.719] - Hard-fork specification: <https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md>
INFO [08-17|06:43:03.719] - Network known to be merged: true
INFO [08-17|06:43:03.719] - Total terminal difficulty: 58750000000000000000000
INFO [08-17|06:43:03.719]
INFO [08-17|06:43:03.719] Post-Merge hard forks (timestamp based):
INFO [08-17|06:43:03.719] - Shanghai: @1681338455 (<https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md>)
INFO [08-17|06:43:03.719]
INFO [08-17|06:43:03.719] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [08-17|06:43:03.719]
INFO [08-17|06:43:03.719] Loaded most recent local block number=0 hash=d4e567..cb8fa3 td=17,179,869,184 age=54y4mo3w
WARN [08-17|06:43:03.719] Failed to load snapshot err="missing or corrupted snapshot"
INFO [08-17|06:43:03.721] Rebuilding state snapshot
INFO [08-17|06:43:03.722] Resuming state snapshot generation root=d7f897..0f0544 accounts=0 slots=0 storage=0.00B dangling=0 elapsed=1.221ms
INFO [08-17|06:43:03.722] Regenerated local transaction journal transactions=0 accounts=0
INFO [08-17|06:43:03.759] Chain post-merge, sync via beacon client
INFO [08-17|06:43:03.760] Gasprice oracle is ignoring threshold set threshold=2
WARN [08-17|06:43:03.761] Error reading unclean shutdown markers error="pebble: not found"
WARN [08-17|06:43:03.768] Engine API enabled protocol=eth
INFO [08-17|06:43:03.768] Starting peer-to-peer node instance=Geth/v1.12.2-stable-bed84606/linux-arm64/go1.20.7
INFO [08-17|06:43:03.782] New local node record seq=1,692,254,583,780 id=23d7280689379076 ip=127.0.0.1 udp=30303 tcp=30303
INFO [08-17|06:43:03.790] Started P2P networking self=enode://7b35cab1bcc1258a96814b204223ce90ee71ff7df44c3bd314e0cda0677e93a80bb5fd0efc3f166c28e4dde41771d0195eecd910f532fe3fb98ee14728046e0f@127.0.0.1:30303
INFO [08-17|06:43:03.793] IPC endpoint opened url=/root/.ethereum/geth.ipc
INFO [08-17|06:43:03.795] Generated JWT secret path=/root/.ethereum/jwt.hex
INFO [08-17|06:43:03.797] HTTP server started endpoint=[::]:8545 auth=false prefix= cors=* vhosts=*
INFO [08-17|06:43:03.797] GraphQL enabled url=http://[::]:8545/graphql
INFO [08-17|06:43:03.798] GraphQL UI enabled url=http://[::]:8545/graphql/ui
INFO [08-17|06:43:03.798] WebSocket enabled url=ws://[::]:8546
INFO [08-17|06:43:03.798] WebSocket enabled url=ws://[::]:8551
INFO [08-17|06:43:03.798] HTTP server started endpoint=[::]:8551 auth=true prefix= cors=localhost vhosts=*
INFO [08-17|06:43:03.835] Generated state snapshot accounts=8893 slots=0 storage=409.64KiB dangling=0 elapsed=114.105ms
INFO [08-17|06:43:09.120] New local node record seq=1,692,254,583,781 id=23d7280689379076 ip=49.12.77.197 udp=30303 tcp=30303
INFO [08-17|06:43:14.797] Looking for peers peercount=1 tried=19 static=0
INFO [08-17|06:43:25.466] Looking for peers peercount=1 tried=34 static=0
INFO [08-17|06:43:35.466] Looking for peers peercount=1 tried=33 static=0
WARN [08-17|06:43:38.770] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
The following entry can be found in the log:
WARN [08-17|06:43:38.770] Post-merge network, but no beacon client seen. Please launch one to follow the chain!
This is as mentioned at the beginning of the article, we need to start a "consensus client", and here we choose Prysm. Prysm needs to specify Geth's Endpoint for startup, so let's modify the docker-compose.yml file accordingly with the following content:
version: "3"
services:
geth:
image: ethereum/client-go:v1.12.2
restart: unless-stopped
ports:
- 30303:30303
- 30303:30303/udp
- 127.0.0.1:8545:8545
- 127.0.0.1:8546:8546
- 127.0.0.1:8551:8551
volumes:
- ./data:/root/.ethereum
healthcheck:
test: [ "CMD-SHELL", "geth attach --exec eth.blockNumber" ]
interval: 10s
timeout: 5s
retries: 5
command:
- --http
- --cache=8192
- --http.api=eth,net,web3,engine,admin
- --http.addr=0.0.0.0
- --http.vhosts=*
- --http.corsdomain=*
- --maxpeers=200
- --ws
- --ws.origins=*
- --ws.addr=0.0.0.0
- --ws.api=eth,net,web3
- --graphql
- --graphql.corsdomain=*
- --graphql.vhosts=*
- --authrpc.addr=0.0.0.0
- --authrpc.jwtsecret=/root/.ethereum/jwt.hex
- --authrpc.vhosts=*
- --authrpc.port=8551
- --txlookuplimit=0
prysm:
image: gcr.io/prysmaticlabs/prysm/beacon-chain
pull_policy: always
container_name: beacon
restart: unless-stopped
stop_grace_period: 2m
volumes:
- ./prysm_data:/data
- ./data:/geth
depends_on:
geth:
condition: service_healthy
ports:
- 127.0.0.1:4000:4000
- 127.0.0.1:3500:3500
command:
- --accept-terms-of-use
- --datadir=/data
- --disable-monitoring
- --rpc-host=0.0.0.0
- --execution-endpoint=http://geth:8551
- --jwt-secret=/geth/jwt.hex
- --rpc-host=0.0.0.0
- --rpc-port=4000
- --grpc-gateway-corsdomain=*
- --grpc-gateway-host=0.0.0.0
- --grpc-gateway-port=3500
- --checkpoint-sync-url=https://mainnet-checkpoint-sync.attestant.io/
- --genesis-beacon-api-url=https://mainnet-checkpoint-sync.attestant.io/
--checkpoint-sync-url=https://mainnet-checkpoint-sync.attestant.io/ are part of a mechanism called checkpoint-sync. You can find detailed information about it here: https://docs.prylabs.network/docs/prysm-usage/checkpoint-sync. In simple terms, it allows syncing information from a node that is already synchronized, reducing the time required for syncing from scratch.
At this point, we can already see from Geth's logs that we are successfully synchronizing data. Next, let's take a look at Pathfinder.
Pathfinder
https://github.com/eqlabs/pathfinder
Pathfinder is a Starknet full node written in Rust. Follow the “Running with Docker” instructions on GitHub:
# Ensure the directory has been created before invoking docker
mkdir -p $HOME/pathfinder
# Start the pathfinder container instance running in the background
sudo docker run \\
--name pathfinder \\
--restart unless-stopped \\
--detach \\
-p 9545:9545 \\
--user "$(id -u):$(id -g)" \\
-e RUST_LOG=info \\
-e PATHFINDER_ETHEREUM_API_URL="<https://goerli.infura.io/v3/><project-id>" \\
-v $HOME/pathfinder:/usr/share/pathfinder/data \\
eqlabs/pathfinder
For PATHFINDER_ETHEREUM_API_URL , since we already have our own full node, we should directly use our Geth's Endpoint to interact with Pathfinder and the chain. Let's continue integrating the docker-compose.yml file above, with the following content:
version: "3"
services:
geth:
image: ethereum/client-go:v1.12.2
restart: unless-stopped
ports:
- 30303:30303
- 30303:30303/udp
- 127.0.0.1:8545:8545
- 127.0.0.1:8546:8546
- 127.0.0.1:8551:8551
volumes:
- ./data:/root/.ethereum
healthcheck:
test: [ "CMD-SHELL", "geth attach --exec eth.blockNumber" ]
interval: 10s
timeout: 5s
retries: 5
command:
- --http
- --cache=8192
- --http.api=eth,net,web3,engine,admin
- --http.addr=0.0.0.0
- --http.vhosts=*
- --http.corsdomain=*
- --maxpeers=200
- --ws
- --ws.origins=*
- --ws.addr=0.0.0.0
- --ws.api=eth,net,web3
- --graphql
- --graphql.corsdomain=*
- --graphql.vhosts=*
- --authrpc.addr=0.0.0.0
- --authrpc.jwtsecret=/root/.ethereum/jwt.hex
- --authrpc.vhosts=*
- --authrpc.port=8551
- --txlookuplimit=0
pathfinder:
image: eqlabs/pathfinder:latest
restart: always
ports:
- '127.0.0.1:9545:9545'
environment:
PATHFINDER_ETHEREUM_API_URL: '<http://geth:8545>'
volumes:
- ./pathfinder:/usr/share/pathfinder/data
prysm:
image: gcr.io/prysmaticlabs/prysm/beacon-chain
container_name: beacon
restart: unless-stopped
stop_grace_period: 2m
volumes:
- ./prysm_data:/data
- ./data:/geth
depends_on:
geth:
condition: service_healthy
ports:
- 127.0.0.1:4000:4000
- 127.0.0.1:3500:3500
command:
- --accept-terms-of-use
- --datadir=/data
- --disable-monitoring
- --rpc-host=0.0.0.0
- --execution-endpoint=http://geth:8551
- --jwt-secret=/geth/jwt.hex
- --rpc-host=0.0.0.0
- --rpc-port=4000
- --grpc-gateway-corsdomain=*
- --grpc-gateway-host=0.0.0.0
- --grpc-gateway-port=3500
- --checkpoint-sync-url=https://mainnet-checkpoint-sync.attestant.io/
- --genesis-beacon-api-url=https://mainnet-checkpoint-sync.attestant.io/
Within the same docker-compose, all containers can interact directly with each other using container names. Therefore, our syntax for environment variables is:
PATHFINDER_ETHEREUM_API_URL: '<http://geth:8545>'
After the startup is complete, we can interact using the API interface specification found here: https://github.com/eqlabs/pathfinder/blob/main/doc/rpc/pathfinder_rpc_api.json
After setting up our own full node, we can use various tools to test and compare it with nodes from other service providers. One interesting tool for this purpose is: https://github.com/shazow/ethspam
- ethspam generates an infinite stream of realistic read-only Ethereum JSONRPC queries, anchored around the latest block with some amount of random jitter. The latest state is updated every 15 seconds, so it can run continuously without becoming stale.
In our own practice, the test results of our node are as follows:
./ethspam | ./versus --stop-after=1000 --concurrency=5 "<https://geth.reddio.network/>"
Endpoints:
0. "<https://geth.reddio.network/>"
Requests: 58.20 per second
Timing: 0.0859s avg, 0.0239s min, 4.0492s max
0.2834s standard deviation
Percentiles:
25% in 0.0374s
50% in 0.0408s
75% in 0.0467s
90% in 0.0612s
95% in 0.1085s
99% in 1.7751s
Errors: 0.00%
** Summary for 1 endpoints:
Completed: 1000 results with 1000 total requests
Timing: 85.904856ms request avg, 17.86287743s total run time
Errors: 0 (0.00%)
Mismatched: 0
As a comparison, we compared our results with nodes provided by Alchemy:
/ethspam | ./versus --stop-after=1000 --concurrency=5 "<https://eth-goerli.g.alchemy.com/v2/xxxxxxxxxx>"
Endpoints:
0. "<https://eth-goerli.g.alchemy.com/v2/xxxxxxxxxx>"
Requests: 37.88 per second, 8.34 per second for errors
Timing: 0.1320s avg, 0.1049s min, 0.5299s max
0.0332s standard deviation
Percentiles:
25% in 0.1192s
50% in 0.1255s
75% in 0.1361s
90% in 0.1466s
95% in 0.1555s
99% in 0.3645s
Errors: 9.70%
97 × "bad status code: 429"
** Summary for 1 endpoints:
Completed: 1000 results with 1000 total requests
Timing: 131.98702ms request avg, 27.179966564s total run time
Errors: 97 (9.70%)
Mismatched: 0
Based on the results, our node processing speed is significantly faster than Alchemy's: we can handle 58.20 requests per second, while Alchemy only manages 37.88 requests per second.