Home/Nodes/Node Operation/Node Setup

Setting Up a Flow Node

Updated: 2023/2/9 bypeterarguepeterargue
5 min read

How to run a Collection, Consensus, Verification and Execution node

This guide is for running a Collection, Consensus, Verification and Execution node. If you are planning to run an Access node then refer to access node setup.

First you'll need to provision a machine or virtual machine to run your node software. Please see follow the node-provisioning guide for it.

Pull the Flow Images

The flow-go binaries are distributed as container images, and need to be pulled down to your host with your image management tool of choice.

Replace $ROLE with the node type you are planning to run. Valid options are:

  • collection
  • consensus
  • execution
  • verification
  • access
1
# Docker
2
  docker pull gcr.io/flow-container-registry/${ROLE}:alpha-v0.0.1
3


4
  # Containerd
5
  ctr images pull gcr.io/flow-container-registry/${ROLE}:alpha-v0.0.1",

Prepare Your Node to Start

Your nodes will need to boot at startup, and restart if they crash.

If you are running systemd you can use the service files provided by flow-go. Find them in the Flow Go.

If you are using some other system besides Systemd, you need to ensure that the Flow container is started, the appropriate key directories are mounted into the container, and that the container will automatically restart following a crash.

The systemd files pull runtime settings from /etc/flow/runtime-config.env and any .env files under /etc/flow/conf.d. Examples of these files are also available in the github repo. You will need to modify the runtime config file later.

Systemd

If you are not using Systemd, you can skip this step

  1. Ensure that you pulled the latest changes from flow-go repository via git
1
## Clone the repo if you haven't already done so
2
git clone https://github.com/onflow/flow-go
3


4
## Get latest changes
5
cd flow-go
6
git pull origin master
  1. Copy your respective systemd unit file to: /etc/systemd/system
  2. Create directory sudo mkdir /etc/flow
  3. Copy the runtime-conf.env file to: /etc/flow/
  4. Enable your service sudo systemctl enable flow-$ROLE.service (replace $ROLE with your node role - eg. collection)

Docker Configuration

If you are not using Systemd, sample commands for running each Docker container are below. Be sure to replace /path/to/data and /path/to/bootstrap with the appropriate paths you are using.

Do not run your node using docker run command directly without a mechanism for the node to automatically restart following a crash.

The actual Docker image tag can be found here for appropriate spork.

System Configuration

Flow nodes create connections to other nodes on the network, which are represented as file descriptors by the OS. Depending on the default limits for your machine, you may need to increase the soft limit available to the node software.

Make sure the soft limit is at least 8192.

You can configure the ulimit for the node's docker container. See the Docker documentation for more details.

Access

1
docker run --rm \
2
	-v /path/to/bootstrap:/bootstrap:ro \
3
	-v /path/to/data:/data:rw \
4
	--name flow-go \
5
	--network host \
6
	--ulimit nofile=8192 \
7
	gcr.io/flow-container-registry/access:<applicable docker tag> \
8
	--nodeid=${FLOW_GO_NODE_ID} \
9
	--bootstrapdir=/bootstrap \
10
	--datadir=/data/protocol \
11
	--secretsdir=/data/secrets \
12
	--rpc-addr=0.0.0.0:9000 \
13
	--http-addr=0.0.0.0:8000 \
14
	--collection-ingress-port=9000 \
15
	--script-addr=${FLOW_NETWORK_EXECUTION_NODE} \
16
	--bind 0.0.0.0:3569 \
17
	--loglevel=error

Collection

1
docker run --rm \
2
	-v /path/to/bootstrap:/bootstrap:ro \
3
	-v /path/to/data:/data:rw \
4
	--name flow-go \
5
	--network host \
6
	--ulimit nofile=8192 \
7
	gcr.io/flow-container-registry/collection:<applicable docker tag> \
8
	--nodeid=${FLOW_GO_NODE_ID} \
9
	--bootstrapdir=/bootstrap \
10
	--datadir=/data/protocol \
11
	--secretsdir=/data/secrets \
12
	--ingress-addr=0.0.0.0:9000 \
13
	--bind 0.0.0.0:3569 \
14
	--loglevel=error

Consensus

1
docker run --rm \
2
	-v /path/to/bootstrap:/bootstrap:ro \
3
	-v /path/to/data:/data:rw \
4
	--name flow-go \
5
	--network host \
6
	--ulimit nofile=8192 \
7
	gcr.io/flow-container-registry/consensus:<applicable docker tag> \
8
	--nodeid=${FLOW_GO_NODE_ID} \
9
	--bootstrapdir=/bootstrap \
10
	--datadir=/data/protocol \
11
	--secretsdir=/data/secrets \
12
	--bind 0.0.0.0:3569 \
13
	--loglevel=error

Execution

1
docker run --rm \
2
	-v /path/to/bootstrap:/bootstrap:ro \
3
	-v /path/to/data:/data:rw \
4
	--name flow-go \
5
	--network host \
6
	--ulimit nofile=8192 \
7
	gcr.io/flow-container-registry/execution:<applicable docker tag> \
8
	--nodeid=${FLOW_GO_NODE_ID} \
9
	--bootstrapdir=/bootstrap \
10
	--datadir=/data/protocol \
11
	--secretsdir=/data/secrets \
12
	--triedir=/data/execution \
13
	--rpc-addr=0.0.0.0:9000 \
14
	--bind 0.0.0.0:3569 \
15
	--loglevel=error

Verification

1
docker run --rm \
2
	-v /path/to/bootstrap:/bootstrap:ro \
3
	-v /path/to/data:/data:rw \
4
	--name flow-go \
5
	--network host \
6
	--ulimit nofile=8192 \
7
	gcr.io/flow-container-registry/verification:<applicable docker tag> \
8
	--nodeid=${FLOW_GO_NODE_ID} \
9
	--bootstrapdir=/bootstrap \
10
	--datadir=/data/protocol \
11
	--secretsdir=/data/secrets \
12
	--bind 0.0.0.0:3569 \
13
	--loglevel=error

Archive

1
docker run --rm \
2
	-v /path/to/data:/data:rw \
3
	--network host \
4
	--label=project=flow \
5
	--label=network=mainnet
6
	--label=app=dps
7
	--label=version=v0.26
8
	--name flow-dps gcr.io/flow-container-registry/flow-dps-live:v0.26.1
9
	--address 0.0.0.0:5005
10
	--index /data/index
11
	--bootstrap /data/bootstrap
12
	--checkpoint /data/bootstrap/root.checkpoint
13
	--data /data/protocol
14
	--level debug
15
	--metrics 0.0.0.0:8080
16
	--bucket bucket_name
17
	--seed-address=access_node_address
18
	--seed-key=seed_key

Start the Node

Now that your node is provisioned and configured, it can be started.

Before starting your node, ensure it is registered and authorized.

Ensure you start your node at the appropriate time. See Spork Process for when to start up a node following a spork. See Node Bootstrap for when to start up a newly registered node.

Systemd

  1. Check that your runtime-conf.env is at /etc/flow/runtime-conf.env
  2. Update your environment variables: source /etc/flow/runtime-conf.env
  3. Start your service: sudo systemctl start flow

Verify your Node is Running

Here are a few handy commands that you can use to check if your Flow node is up and running

Systemd

  • To get Flow logs: sudo journalctl -u flow-YOUR_ROLE
  • To get the status: sudo systemctl status flow
1
● flow-verification.service - Flow Access Node running with Docker
2
Loaded: loaded (/etc/systemd/system/flow-verification.service; enabled; vendor preset: enabled)
3
Active: active (running) since Wed 2020-05-20 18:18:13 UTC; 1 day 6h ago
4
Process: 3207 ExecStartPre=/usr/bin/docker pull gcr.io/flow-container-registry/verification:${FLOW_GO_NODE_VERSION} (code=exited, status=0/SUCCESS)
5
Main PID: 3228 (docker)
6
Tasks: 10 (limit: 4915)
7
Memory: 33.0M
8
CGroup: /system.slice/flow-verification.service
9
        └─3228 /usr/bin/docker run --rm -v /var/flow/bootstrap:/bootstrap:ro -v /var/flow/data:/data:rw --rm --name flow-go --network host gcr.io/flow-container-registry/verification:candidate8 --nodeid=489f8a4513d5bd8b8b093108fec00327b683db545b37b4ea9153f61b2c0c49dc --bootstrapdir=/bootstrap --datadir=/data/protocol --alpha=1 --bind 0.0.0.0:3569 --loglevel=error

Docker

  • To get Flow logs: sudo docker logs flow-go
  • To get the status: sudo docker ps
1
$ sudo docker ps
2
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3
1dc5d43385b6 gcr.io/flow-container-registry/verification:candidate8 \"/bin/app --nodeid=4…\" 30 hours ago Up 30 hours flow-go

Monitoring and Metrics

This is intended for operators who would like to see what their Flow nodes are currently doing. Head over to Monitoring Node Health to get setup.

Node Status

The metrics for the node should be able to provide a good overview of the status of the node. If we want to get a quick snapshot of the status of the node, and if it's properly participating in the network, you can check the consensus_compliance_finalized_height or consensus_compliance_sealed_height metric, and ensure that it is not zero and strictly increasing.

1
curl localhost:8080/metrics | grep consensus_compliance_sealed_height
2


3
# HELP consensus_compliance_sealed_height the last sealed height
4
# TYPE consensus_compliance_sealed_height gauge
5
consensus_compliance_sealed_height 1.132054e+06
Prev
Access Node Setup
Next
Node Roles