Build EOSIO local network

Some experience on building EOSIO local network

Basic info

EOSIO is an application based on proof of stake. I am going to discuss my experience on setting up the local test environment for EOSIO. The test version for EOS is 1.7 and test environment is Ubuntu 18.04.
The whole system is made up of three components in order to build local environment:

• EOS - Main net. Basic communication

• EOS.CDT - Developer package. API for application

• EOS.CONTRACT - super user account setup. It store information about different user and send call to system, like create an voter account.

Following is the basic network structure of EOSIO

• Only one super account. It produces blocks based on gensis.json. It can also interact with system to create accounts, register producers.

• Multiple(21) block generators. They will produce(confirm) block once get enough vote

• Voter. It could vote producer.

gensis.json is a json file contains the information for the system to produce gensis block. Its structure is as follows:

{
  "initial_timestamp": "2018-12-05T08:55:11.000", 
  "initial_key": "EOS_PUB_DEV_KEY", 'Only the super account who has the private key relate to this key can produce block'
  "initial_configuration": {
    ... 
    "target_block_cpu_usage_pct": 500, 'It doesn't like bitcoind, you could set up everything of the genesis block by yourself.'
    "max_transaction_cpu_usage": 50000,
    "min_transaction_cpu_usage": 100,
    "max_transaction_lifetime": 3600,
    "deferred_trx_expiration_window": 600,
    "max_transaction_delay": 3888000,
    "max_inline_action_size": 4096,
    "max_inline_action_depth": 4,
    "max_authority_depth": 6
  },
  "initial_chain_id": "0000000000000000000000000000000000000000000000000000000000000000"
}

For most things in the file is easy to understand. The server is clever, for the string in the json, the server will choose characters he thought it is correct. Confound server thought will be ignored.

Build & install

In order to setup a local test environment, EOS.CDT and EOS.CONTRACT have to be compiled on the local machine. Official said EOS is not required to compile locally. But during my test and some hint from EOS.CONTRACT. It seems it is required. Original official words as follows:

1. First, ensure that your eosio is compiled to the core symbol for the EOSIO blockchain that intend to deploy to.

2. Second, make sure that you have sudo make installed eosio.

3. Then just run the build.sh in the top directory to build all the contracts and the unit tests for these contracts.

Before compile and run those stuff, make sure the computer has at least 40GB in your home directory and 8GB of memory.

To compile EOS locally, do the following:

git clone https://github.com/EOSIO/eos --recursive
cd eos/
./scripts/eosio_build.sh

It is also required to install this one into root. The build script is at scripts directory right now. And it may take up to 3 hours to build. It could be faster if you are using a multi-core computer.

It is also recommended to install it. So that it will be easier to use in the future. Just do ./scripts/eosio_install.sh. During my test on ubuntu 18.04. The script installed successfully. But I can not run it in the shell.

Then build eosio.cdt.

git clone --recursive https://github.com/eosio/eosio.cdt
cd eosio.cdt
./build.sh
sudo ./install.sh

There should exist at least 10GB to compile this one.

Then build eosio.contracts.

git clone https://github.com/EOSIO/eosio.contracts.git
cd eosio.contracts/
./build.sh

The building process of eosio.cdt and eosio.contracts is nothing special but time consuming. It might take up to 2 hours to build. And it is impossible to build using multiple cores.

Some common error and solutions could be found at here.

Initialize

You need to have an wallet in order to get start. You will have one by doing the following:

cleos wallet create --to-console

There is no way to recover password, do remember it.

When you have the password, you could play with cleos. You could use the following command to see full list of subcommand.

cleos wallet help

Some useful subcommands are open, unlock, create_key, private_keys. Whenever you want to visit sensitive data, you need to provide the password. To prevent prompted by system, you could use --password to pass in your password.

Typically, you need to open wallet at first and then unlock to interact with wallet. After all interactions, it is good to lock them.

Batch initialize

The following are scripts might be helpful to initialize for the appropriate data folder and public key-private key pair for multiple(21) nodes.

To use it, you have to pass in the password as the first attribute

cleos wallet open

if [ $1 ]; then
DATADIR="./blockchain"
if [ ! -d $DATADIR ]; then
  mkdir $DATADIR;
fi

for i in `seq 1 21 `
do
DATADIR="./blockchain"$i
if [ ! -d $DATADIR ]; then
  mkdir $DATADIR;
fi
done

cleos wallet unlock  --password $1

for i in $(seq 1 21)
do
cleos wallet create_key
done

cleos wallet private_keys  --password $1|grep "\"">list

python3 a.py #a.py will be discussed below
cleos wallet lock
fi

a.py is a python script which will export public-key and private key information into file in case it need to be used in the future. It is always required to have both information of public key and private key when starting a node.

file = open("list", "r") 
file1 = open("li_", "w")
count = 0
li = []
for line in file: 
	li.append(line.replace('\"', '').replace(',','').replace(' ',''))
	count+=1
for i in range(count-1,-1,-1):
	file1.write(li[i])
file.close()
file1.close()

The key start with EOS and longer is public key, like EOS5CZwEpiweHHZpEdHYsU9Q1MEk5zTtqfrcg3TefBzTYG9xdw2gC

The key which is shorter is private key, like this: 5Jarc4qfsXqHcivfGc8qWYTGdQJ5CPy8NeyRjfceXopn9dy3hgR

Batch clean

Those are some script that will remove everything. It will make the workspace(including wallet, work folder) be initialized.

#!/bin/bash

DATADIR="./blockchain"
rm -rf $DATADIR*;

cleos wallet open

if [ $1 ]; then
	cleos wallet unlock --password $1
	while read a; do
	read a;
	cleos wallet remove_key $a --password $1
	done < li_
	rm li_
	rm list
fi
cleos wallet lock

It is good to clean every time after experiment, otherwise blocks generated will occupy too many space. It is 0.5s/blocks.

Procedure

This is the basic procedure. The voter account and block generator account are produced by one super account. Generally speaking, the basic idea is voter will vote the producer they like, block producer are connected together. During the runtime, the procedure is:

• Firstly, start producer nodes.

• Next vote

• When the block producer has enough votes, block producer will go to confirm blocks.

After doing appropriate settings it should start to confirm blocks. I did’t test this part yet.

If you want or in some situation, you could link super account together like this, but only the one with appropriate public and private key could generate the block. More specifically, only the one has the private key of the public key in the gensis.json can produce block. Other super accounts nodes could only sync.

The whole structure like this:

You could use the following command to start multiple super account nodes from gensis status and form the structure shown above:

import sys
import os
import json
import time

password = ""
try:
	password = sys.argv[1]
except IndexError:
	print("Please include password")
	sys.exit(1)

'''
Generate gensis.json
'''
file1 = open("li_", "r")
temp = file1.readline()
initial_key = file1.readline()
os.system('rm -f genesis.json')
with open("genesis.json", "w") as outfile:
    json.dump({'initial_timestamp':"2018-12-05T08:55:11.000", 'initial_key': initial_key[:-1], 'initial_configuration':{"max_block_net_usage": 1048576, "target_block_net_usage_pct": 1000, "max_transaction_net_usage": 524288, "base_per_transaction_net_usage": 12, "net_usage_leeway": 500, "context_free_discount_net_usage_num": 20, "context_free_discount_net_usage_den": 100, "max_block_cpu_usage": 100000, "target_block_cpu_usage_pct": 500, "max_transaction_cpu_usage": 50000, "min_transaction_cpu_usage": 100, "max_transaction_lifetime": 3600, "deferred_trx_expiration_window": 600, "max_transaction_delay": 3888000, "max_inline_action_size": 4096, "max_inline_action_depth": 4, "max_authority_depth": 6 },  "initial_chain_id": "0000000000000000000000000000000000000000000000000000000000000000"}, outfile, indent=4)

'''
Start gensis
'''
ini_port=9000
currentport=ini_port
sentstring='./genesis_start.sh '+'./blockchain'+' '+initial_key[:-1]+' '+temp[:-1]+' '+str(currentport)+' '+str(currentport+100)+' '+str(currentport+101)
os.system('echo Start gensis')
os.system(sentstring)
currentport+=1

time.sleep(30)
'''
start nodes
'''
for i in range(1,22):
	private_key = file1.readline()
	public_key = file1.readline()
	tosend = './genesis_start.sh '+'./blockchain'+str(i)+' '+public_key[:-1]+' '+private_key[:-1]+' '+str(currentport)+' '+str(currentport+100)+' '+str(currentport+99)
	os.system('echo start node '+ str(i))
	os.system(tosend)
	currentport+=1

For this python code, it first generates the gensis.json. And then it starts nodes in some order(First gensis node, then other super accounts). All public key and private key used is based on the file generated previously. To use it, you have to put the password of the wallet as the first attribute pass in when you run the program. The script did so, in case it needs to deal with sensitive information.

The follow script is in a file called genesis_start.sh, which is called by the previous python program to start each node:

#!/bin/bash
#$1--datadir $2--Public_key $3--Private_key $4--httpserver $5--p2plisten $6--peer_address
if [ $1 ] && [ $2 ] && [ $3 ] && [ $4 ] && [ $5 ] && [ $6 ]; then
DATADIR=$1

nodeos \
--genesis-json $DATADIR"/../genesis.json" \
--signature-provider $2=KEY:$3 \
--plugin eosio::producer_plugin \
--plugin eosio::chain_api_plugin \
--plugin eosio::http_plugin \
--plugin eosio::history_api_plugin \
--data-dir $DATADIR"/data" \
--blocks-dir $DATADIR"/blocks" \
--config-dir $DATADIR"/config" \
--producer-name eosio \
--http-server-address 127.0.0.1:$4 \
--p2p-listen-endpoint 127.0.0.1:$5 \
--access-control-allow-origin=* \
--contracts-console \
--http-validate-host=false \
--verbose-http-errors \
--enable-stale-production \
--p2p-peer-address localhost:$6 \
>> $DATADIR"/nodeos.log" 2>&1 & \
echo $! > $DATADIR"/eosd.pid"
fi

This script will start a new node by using information given by user and save the running result to file at the datadir given… To use it manually, just use ./genesis_start.sh [datadir] [Public_key] [Private_key] [httpserver] [p2plisten] [peer_address]

If you get an information of “This is not an executable file”. Just do chmod 777 genesis_start.sh

I think those scripts are still useful for the following reason:

• The producer has option about the super account node it reports. It could make the main node not too stressful when lots of submission happens(I guess).
• By modifying them a little bit, they could also be used to generate multiple block producer and voters.

And I find out only the linked structure will be accepted. It means multiple connections to one super account node is not permitted. More detailed, as the graph show below:

To make the system recognize an account is block producer, it is required to use regproducer like the following. Otherwise it is just a voter.

Build local test network

Need to use the EOS (cleos and nodes) compiled from the previous steps. Maybe this is not required. But it seems that eosio.contract is related on eosio main program. The network can not be built when using the software from ubuntu package management on my machine.

Basically, I follow up the guidance from Guide to EOS voting simulation and the official guidance BIOS Boot Sequence. Some places has been changed to fit the latest version of EOSIO.

Before start, you need to initialize wallet by taking the previous steps. And then create some new public and private key pairs. Just do cleos wallet create_key. If you want to see all key pairs, do cleos wallet private_keys.

Then, create genesis.json and the script used to start producing genesis block.

• Create genesis.json at a property directory.

{
  "initial_timestamp": "2018-12-05T08:55:11.000",
  "initial_key": "EOS_PUB_DEV_KEY",
  "initial_configuration": {
    "max_block_net_usage": 1048576,
    "target_block_net_usage_pct": 1000,
    "max_transaction_net_usage": 524288,
    "base_per_transaction_net_usage": 12,
    "net_usage_leeway": 500,
    "context_free_discount_net_usage_num": 20,
    "context_free_discount_net_usage_den": 100,
    "max_block_cpu_usage": 100000,
    "target_block_cpu_usage_pct": 500,
    "max_transaction_cpu_usage": 50000,
    "min_transaction_cpu_usage": 100,
    "max_transaction_lifetime": 3600,
    "deferred_trx_expiration_window": 600,
    "max_transaction_delay": 3888000,
    "max_inline_action_size": 4096,
    "max_inline_action_depth": 4,
    "max_authority_depth": 6
  },
  "initial_chain_id": "0000000000000000000000000000000000000000000000000000000000000000"
}

• Then, create the script used to start the first node and save it as genesis_start.sh.

#!/bin/bash
DATADIR="./blockchain"

if [ ! -d $DATADIR ]; then
  mkdir -p $DATADIR;
fi

nodeos \
--genesis-json $DATADIR"/../genesis.json" \
--signature-provider EOS_PUB_DEV_KEY=KEY:EOS_PRIV_DEV_KEY \
--plugin eosio::producer_plugin \
--plugin eosio::chain_api_plugin \
--plugin eosio::http_plugin \
--plugin eosio::history_api_plugin \
--plugin eosio::history_plugin \
--data-dir $DATADIR"/data" \
--blocks-dir $DATADIR"/blocks" \
--config-dir $DATADIR"/config" \
--producer-name eosio \
--http-server-address 127.0.0.1:8888 \
--p2p-listen-endpoint 127.0.0.1:9010 \
--access-control-allow-origin=* \
--contracts-console \
--http-validate-host=false \
--verbose-http-errors \
--enable-stale-production \
--p2p-peer-address localhost:9011 \
--p2p-peer-address localhost:9012 \
--p2p-peer-address localhost:9013 \
>> $DATADIR"/nodeos.log" 2>&1 & \
echo $! > $DATADIR"/eosd.pid"

Note: The EOS_PUB_DEV_KEY in signature-provider in genesis_start.sh should be the same as the EOS_PUB_DEV_KEY in genesis.json. EOS_PRIV_DEV_KEY is the private key of this public key. Otherwise, you will get the error Not producing block because I don’t have the private key for EOS_PUB_DEV_KEY

1. Start the first node

This tutorial is based on v4.2.0, with one node for bios and two nodes as block producers. First start the bios node:

./gensis_start.sh

2. Deploy eosio.bios contract to eosio account.

cleos set contract eosio build/contracts/eosio.bios

3. **Create necessary accounts.**

The list of accounts required is provided by official:

> eosio.bpay
> eosio.msig
> eosio.names
> eosio.ram
> eosio.ramfee
> eosio.saving
> eosio.stake
> eosio.token
> eosio.vpay
> eosio.rex
>

Use create account command to create each of them:

cleos create account eosio eosio.*** EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo

4. Set the eosio.token contract

cleos set contract eosio.token eos/build/contracts/eosio.token

The eosio.token used should be at the build directory which contains file eosio.token.wasm, otherwise wrong message will be invoked.

5. set the eosio.msig contract

cleos create account eosio eosio.msig EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo
cleos set contract eosio.msig eos/build/contracts/eosio.msig

The eosio.msig used should be at the build directory which contains file eosio.msig.wasm, otherwise wrong message will be invoked.

6. Create and allocate token

We create and issue 1,000,000,000.0000 tokens and name it ‘SYS’. Then issue to eosio account.

cleos push action eosio.token create '{"issuer":"eosio", "maximum_supply": "1000000000.0000 SYS", "can_freeze": 0, "can_recall": 0, "can_whitelist": 0}' -p eosio.token
cleos push action eosio.token issue '{"to":"eosio","quantity":"1000000000.0000 SYS","memo":"issue"}' -p eosio

To check the balance and stats

cleos get currency balance eosio.token eosio
cleos get currency stats eosio.token 'SYS'

7. Set eosio.system contract to eosio

After setting eosio.system contract we will be able to stake our accounts.

cleos set contract eosio eos/build/contracts/eosio.system

Then stake tokens and expand the network. cleos stakes 8 KB of RAM on account creation, paid by the account creator.

8. Make eosio.msig a privileged account and Initialize system account

cleos push action eosio setpriv '["eosio.msig", 1]' -p eosio@active

cleos push action eosio init '["0", "4,SYS"]' -p eosio@active

The command initializes the system account with code zero (needed at initialization time)
and SYS token with precision 4; precision can range from [0 … 18].

9. Create accounts for BPs: bp1111111111 and bp2222222222

cleos system newaccount eosio bp1111111111 EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo --buy-ram-EOS '1000000.0000 SYS' --stake-net '1000000.0000 SYS' --stake-cpu '1000000.0000 SYS'

10. Register BPs

cleos system regproducer bp1111111111 EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo

Similarly register bp2222222222. To check producers:

cleos system listproducers

To start the two producer nodes:

nodeos -e --genesis-json path/to/genesis.json --producer-name bp1111111111 --private-key '[ "EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo","5K7EY...Hi8Uy61wU1o" ]' --http-server-address 127.0.0.1:8889 --p2plisten-endpoint 127.0.0.1:9877 --p2p-peer-address bios_node_ip:port --plugin eosio::producer_plugin --plugin eosio::chain_api_plugin --plugin eosio::net_api_plugin

11. Create voters accounts: voter1111111, voter2222222, voter3333333

cleos system newaccount eosio voter1111111 EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo EOS6sAgFvNDfiZ62UMosK7WULNVCuAotpR2raKzaBpopLevvYYWBo --buy-ram-EOS '50.0000 SYS' --stake-net '50.0000 SYS' --stake-cpu '50.0000 SYS'

12. Issue tokens to voters

Transfer 100,000,000 SYS to each voter.

cleos push action eosio.token transfer '["eosio", "voter1111111","100000000.0000 SYS","vote"]' -p eosio

Check the balance

cleos get currency balance eosio.token voter1111111

13. Delegate bandwidth

cleos system delegatebw voter1111111 voter1111111 "50000000.0000 SYS" "50000000.0000 SYS"

Sometime, the program will tell you failed. But the command successes actually. You could redo cleos get currency balance eosio.token voter1111111 to check if this command take effect or not.

14. Vote

Voting for producers can begin as accounts are staked and producers are registered.

cleos system voteproducer prods voter1111111 bp1111111111 bp2222222222

After 15% of the available votes have been voted, block producers will begin producing. In this case we need 150,000,000 votes.

• Check votes:

cleos system listproducers 
cleos get table eosio eosio voters

Some error and solutions I found is at here.

Special notice

• EOS.CONTRACT is based on locally compiled EOS.
• To compile EOS locally, it has to use git clone to clone it from github. You can not download from source and compile.
• The compilation requires lots of space. It is good to remain 40GB of disk and 8GB of memory. It also takes around 3 hours to compile.
• They can only be compiled at the current user’s home directory.