Build Bitcoin local network

Some experience on building Bitcoin local network

Basic info

Bitcoin is an client based on proof of work(POW).

Bitcoin core is a client of official bitcoin network. After some tests, I find the local network for bitcoin is good for testing wallet application but hard for testing mining function. In this page, I am going to discuss some experience of mine on building the local test environment for bitcoin core.

Compile & Installation

For ubuntu, just do:

sudo apt-get install build-essential libtool autotools-dev automake pkg-config bsdmainutils python3 libssl-dev libevent-dev libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev libminiupnpc-dev libzmq3-dev libqrencode-dev 
git clone https://github.com/bitcoin/bitcoin.git
cd bitcoin
./autogen.sh
./configure
make
make install # optional

For other system, official has a page at here described how to build and install.

Bitcoin core contains three different networks, mainnet, regtest and testnet. The difference of those three networks are as follows:

Mainnet

The original and main network for Bitcoin transactions, where satoshis have real economic value.

As we could see above, mainnet is the main network of bitcoin core. It has the real economic value, which means it has lots of restriction. And it make me hard to implement it locally. On the main net, I tried lots of different ways, including connect it with a virtual network, do not connect with network and change some source code. But none of them successfully build a local network.

To start a node under main net, just do: (The default port is 8444/8443)

bitcoind

To use rpc to communicate, do the following: (datadir should exist at first)

bitcoind -server -listen -port=17591 -rpcuser=123 -rpcpassword=123 -rpcport=16591 -datadir=$HOME/A/ -pid=$HOME/A/.pid -debug -daemon #-daemon for running in background -debug to get detailed output

In some old version, password and username can not be the same.

To connect to a specific node, use -connect tag.

bitcoind -server -listen -port=17591 -rpcuser=123 -rpcpassword=123 -rpcport=16591 -connect=127.0.0.1:17592 -datadir=$HOME/A/ -pid=$HOME/A/.pid -debug -daemon

-reindex-chainstate(Rebuild chain state from the currently indexed blocks ) and -reindex(Rebuild chain state and block index from the blk*.dat files on disk ) might be helpful some time. If need them, just add them to the end of the startup code

Use the following code to start 5 nodes and connect them together under main net:

#!/bin/bash
bitcoind -server -listen -port=17591 -rpcuser=123 -rpcpassword=123 -rpcport=16591 -datadir=$HOME/test/A/ -connect=127.0.0.1:17592 -pid=$HOME/test/A/.pid -debug
bitcoind -server -listen -port=17592 -rpcuser=123 -rpcpassword=123 -rpcport=16592 -datadir=$HOME/test/B/ -connect=127.0.0.1:17591 -pid=$HOME/test/B/.pid -debug
bitcoind -server -listen -port=17593 -rpcuser=123 -rpcpassword=123 -rpcport=16593 -datadir=$HOME/test/C/ -connect=127.0.0.1:17591 -pid=$HOME/test/C/.pid -debug
bitcoind -server -listen -port=17594 -rpcuser=123 -rpcpassword=123 -rpcport=16594 -datadir=$HOME/test/D/ -connect=127.0.0.1:17591 -pid=$HOME/test/D/.pid -debug
bitcoind -server -listen -port=17595 -rpcuser=123 -rpcpassword=123 -rpcport=16595 -datadir=$HOME/test/E/ -connect=127.0.0.1:17591 -pid=$HOME/test/E/.pid -debug

Those data dir should exist. To check them:

#!/bin/bash
DATADIR="~/test/"

for i in A B C D E
do
if [ ! -d $DATADIR$i ]; then
  mkdir -p $DATADIR$i;
fi
done

If you want to restart a new test or changed the version of bitcoind running on the machine(from high version to low version). You have to delete the datadir:

#!/bin/bash
DATADIR="~/test/"

for i in A B C D E
do
if [ -d $DATADIR$i ]; then
  rm -rf $DATADIR$i;
fi
done

To get info from any node by using rpc command:

bitcoin-cli -rpcport=16591 -rpcuser=123 -rpcpassword=123 RPC_COMMAND

RPC_COMMAND are commands like getpeerinfo, getblocktemplpate, getmininginfo, or some commands about transaction. The full list is on Bitcoin Core 0.17.0 RPC

bitcoin-cli is just a communication tool. So, you could use the same command to communicate with different nodes in different model(mainnet, testnet, regtest).

Moreover, you could also use it to communicate with other project’s node forked from bitcoind, like litecoind.

Regtest

A local testing environment in which developers can almost instantly generate blocks on demand for testing events, and can create private satoshis with no real-world value.

generate blocks on demand means it is not real case. It is fake mining. An easy example is it produce block quick and producer doesn’t get award they should have. Another example is the normal case should have nonce 0-0xfffff. But it only have nonce 0-4 for every block. So, it is a good tool to test transaction and wallet application locally.

To start a node under regtest, just do: (The default port is 18444/18443)

bitcoind -regtest

The other commands like whatever at the first part of main net. Just add the -regtest tag to the program.

Use the following code to start 5 nodes and connect them together under regtest:

#!/bin/bash
bitcoind -server -listen -port=17591 -rpcuser=123 -rpcpassword=123 -rpcport=16591 -datadir=$HOME/test/A/ -regtest -connect=127.0.0.1:17592 -pid=$HOME/test/A/.pid -debug
bitcoind -server -listen -port=17592 -rpcuser=123 -rpcpassword=123 -rpcport=16592 -datadir=$HOME/test/B/ -regtest -connect=127.0.0.1:17591 -pid=$HOME/test/B/.pid -debug
bitcoind -server -listen -port=17593 -rpcuser=123 -rpcpassword=123 -rpcport=16593 -datadir=$HOME/test/C/ -regtest -connect=127.0.0.1:17591 -pid=$HOME/test/C/.pid -debug
bitcoind -server -listen -port=17594 -rpcuser=123 -rpcpassword=123 -rpcport=16594 -datadir=$HOME/test/D/ -regtest -connect=127.0.0.1:17591 -pid=$HOME/test/D/.pid -debug
bitcoind -server -listen -port=17595 -rpcuser=123 -rpcpassword=123 -rpcport=16595 -datadir=$HOME/test/E/ -regtest -connect=127.0.0.1:17591 -pid=$HOME/test/E/.pid -debug

Those data dir should exist. See the former part for the way to do those procedure

Though it is possible to use command represented before, the official recommend to use this one:

bitcoin-cli -regtest -rpcport=16591 -rpcuser=123 -rpcpassword=123 RPC_COMMAND

RPC_COMMAND are commands like getpeerinfo, getblocktemplpate, getmininginfo, or some commands about transaction. The full list is on Bitcoin Core 0.17.0 RPC

Testnet

A global testing environment in which developers can obtain and spend satoshis that have no real-world value on a network that is very similar to the Bitcoin mainnet.

very similar means it is likely to be the same as the main net. no real-world value means you could get some points free online.

By comparing with mainnet, it seems there is restriction on the height on the testnet, which means difficulty has restriction. Right now, it is at the version 3. Its version could be changed by developer, but it is hardcoded. Difficulty does change when the height being higher and higher. And most things are similar to mainnet. It also has the same restrictions as the main net. So, testnet is best for testing transaction application and wallet application.

To start a node under testnet, just do: (The default port is 18444/18443)

bitcoind -testnet

The other commands like whatever at the first part of main net. Just add the -testnet tag to the program.

Use the following code to start 5 nodes and connect them together under testnet:

#!/bin/bash
bitcoind -server -listen -port=17591 -rpcuser=123 -rpcpassword=123 -rpcport=16591 -datadir=$HOME/test/A/ -testnet -connect=127.0.0.1:17592 -pid=$HOME/test/A/.pid -debug
bitcoind -server -listen -port=17592 -rpcuser=123 -rpcpassword=123 -rpcport=16592 -datadir=$HOME/test/B/ -testnet -connect=127.0.0.1:17591 -pid=$HOME/test/B/.pid -debug
bitcoind -server -listen -port=17593 -rpcuser=123 -rpcpassword=123 -rpcport=16593 -datadir=$HOME/test/C/ -testnet -connect=127.0.0.1:17591 -pid=$HOME/test/C/.pid -debug
bitcoind -server -listen -port=17594 -rpcuser=123 -rpcpassword=123 -rpcport=16594 -datadir=$HOME/test/D/ -testnet -connect=127.0.0.1:17591 -pid=$HOME/test/D/.pid -debug
bitcoind -server -listen -port=17595 -rpcuser=123 -rpcpassword=123 -rpcport=16595 -datadir=$HOME/test/E/ -testnet -connect=127.0.0.1:17591 -pid=$HOME/test/E/.pid -debug

Those data dir should exist. See the former part for the way to do those procedure

Though it is possible to use command represented before, the official recommend to use this one:

bitcoin-cli -testnet -rpcport=16591 -rpcuser=123 -rpcpassword=123 RPC_COMMAND

RPC_COMMAND are commands like getpeerinfo, getblocktemplpate, getmininginfo, or some commands about transaction. The full list is on Bitcoin Core 0.17.0 RPC

Version difference

The bubble is the separator of different version. Litecoind is forked out from bitcoind since 0.16.3. The methods talked above, getblocktemplate, is only for mainnet or testnet. For the regtest net, there is only one method, generate or generatetoaddress. 0.17.9 remove wallet functions like listaccounts. It also remove generate function. To confirm a block on regtest, use generatetoaddress instead.

To connect nodes

This graph is the basic way to connect nodes. The miner is always separate with the main net. Loop connection is permitted but not required. In order to start using mining feature(getblocktemplate). Be sure to have at least two nodes connected. Actually, there are more restriction than this.

To confirm a block

Bitcoin core full node is the client start by bitcoind by using command in the previous section.

Above is the basic structure for mining(confirm a block). If no other things happen, one block will be completely confirmed 100 blocks later. The confirm interval is 100 blocks.

Getblocktemplate is a method which will return the following structure as a return result as a json for the miner to use, which contains information miner need. The structure looks like this:

{
   "result":{
      "capabilities":["proposal"],
      ...
      "previousblockhash":"756c7b16ef069c782fe6faf9e393483821261206fe110d409c11fbc4a0b927b0",
      "transactions":[], 'Transactions information could be used for next block'
      "coinbaseaux":{"flags":""}, 'miner will use this and the nest value to build coinbase'
      "coinbasevalue":5000000000, 'bonus people have'
      "longpollid":"756c7b16ef069c782fe6faf9e393...06fe110d409c11fbc4a0b927b02", 'bitcoind technique'
      "target":"7fffff0000000000000000000000000000000000000000000000000000000000", 
      "mintime":1550614864,
      "noncerange":"00000000ffffffff",
	  ...
      "bits":"207fffff",
      "height":4,
      ...
   },
   ...
}

After version 0.17.9. You have to attach the argument segwit in order to get a return. Their official said this is safer. In order to attach argument, it is recommend to use curl:

curl --user 123 --data-binary '{"id": 1, "method": "getblocktemplate", "params": [{"rules": ["segwit"], "capabilities": ["coinbasetxn", "workid", "coinbase/append"]}]}' -H 'content-type: text/plain;' http://127.0.0.1:16592/

Miners are softwares used for mining

If you are using Miner software, then you don’t have to understand what getblocktemplate get. Miner software will handle them. But you have to make sure the Getblocktemplate method is working and coinbasevalue exist and it is not 0.

To make the coinbasevalue exist and it is not 0, it might be helpful to make sure libblkmakercompile successfully. Because there is some library dependence required. And Getblocktemplate never tell you what is the dependence and the main bitcoin core could compile without it. Maybe there are some better ways. But I fix every dependences by compile libblkmaker. I think it is because they use the same dependence at somewhere. But some people also say bitcoin’s getblocktemplate relay on libblkmaker.

There are two official recommend and commonly used miner. BFGMiner and Cgminer.

To start solo mining by using BFGMiner:

./bfgminer -o http://127.0.0.1:16592 -u 123 -p 123 --generate-to "PUBLIC_ADDRESS" --coinbase-sig "rig1: This is Joe's block!" --net-delay

–coinbase-sig is for comments, it is not required

–net-delay is to make it don’t communicate with server too frequently.

You could use -S to specify hardware you want to use:

bfgminer -S opencl:auto -o http://127.0.0.1:16592 -u 123 -p 123 --generate-to "PUBLIC_ADDRESS"

To start solo mining by using Cgminer:

cgminer -o http://localhost:8332 -u username -p password --btc-address PUBLIC_ADDRESS

This is from the official website, I did’t test it. It requires to use FPGA and ASIC miner.

If you want to build a miner from scratch, official do provide some instructions at their website

Actually, if you just want to build a miner application, most of people choose to make their application based on the libblkmaker official provided.

Special note

1. In order to simulate a real network case that lots of transaction happens every second, you could use the following code:

#!/bin/bash
for i in $(seq 1 1000)
do
  bitcoin-cli -regtest -rpcport=16591 -rpcuser=123 -rpcpassword=123 sendtoaddress "PUBLIC_ADDRESS" 0.0001
  sleep 2
done

2. Build Path should be correct. If you change the path of compiled file, you have to reconfigure and recompile before install or do other implementation.
   Bitcoind is not friendly for mining locally

3. For mainnet and testnet, they set up some restriction. So getblocktemplate never works. The coding for getblocktemplate is in mining.cpp Maybe I need to look through source in order to check where is the real restriction. But I guess they setup multiple restrictions. One I found is at chainparams.cpp. It is also a direction to try to delete as many information as it could to see if it works.

4. In bitcoind, Genesis block and its information, including time interval, is hard coded, even for genesis block.

5. Bitcoid is easy to compile. But it will occupy 3GB space on disk after compilation. And it takes around an hour to compile. To compile it, their official website described it clearly.