Using Podman Containers for Regtest Bitcoin Development

,

This article describes how to build and use a typical local bitcoin development setup using Podman containers. The final environment includes a lot of useful tools, namely:

  1. A bitcoin core node and daemon (serving compact block filters)
  2. bitcoin-cli enabled
  3. An Electrum server
  4. An Esplora server
  5. A block explorer
  6. Useful aliases for working with the daemon

You can follow along this article and build your image and containers from scratch if you like. I also maintain a repository with everything you need to build what’s in this article here: thunderbiscuit/regtest-in-a-pod. It’s a good idea to start from there and modify for your needs.

Environments like these are often a pain to set up correctly to work together. Usernames, passwords, cookies, port numbers, configuration files, the list goes on. Building up an easy-to-boot complete setup for bitcoin development is worth our time. Moreover, podman containers allow us to front-load the work of these complex setups. Once they’re all configured and set up, they don’t fail, and provide easy-to-work-with environments. Even better, they shine over time because it’s easy to iterate and fiddle with them ever so slightly to configure them perfectly for our use cases and preferences.

This article goes into details of how to set up such an environment on both macOS and Linux machines so as to deploy the setup locally for both mobile development (most often done on mac computers) and remotely on cloud droplets (most often done on Linux machines).

Basics of Podman

1. Machines

You need to create virtual machines to run your container in. A good machine for this type of environment is a 4 CPU, 4GB of memory, 20 GB disk machine. Note that podman will only use the disk space required by the containers, not the entirety of what is allocated to the machine.

# List currently available virtual machines
podman machine list
# NAME                     VM TYPE     CREATED         LAST UP     CPUS        MEMORY      DISK SIZE
# podman-machine-default*  qemu        1 minutes ago  Never       1           2GiB        10GiB

# Create a machine called regtest
podman machine init --cpus 4 --memory 4096 --disk-size 20 regtest

At this point, you’ll have two machines installed on your computer.

podman machine list
# NAME                     VM TYPE     CREATED            LAST UP     CPUS        MEMORY      DISK SIZE
# regtest                  qemu        10 seconds ago     Never       4           4GiB        20GiB
# podman-machine-default*  qemu        About an hour ago  Never       1           2GiB        10GiB

# Print info on specific machine
podman machine inspect regtest

2. Images

On virtual machine you need to create and run containers. To build containers, you need to blueprints for what the container should have, download, and build. This blueprint is called an image.

At this point, running podman system info or podman images will throw an error:

# Part 1
podman images
# Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
podman system info
# Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM

# Maybe the machine is not started yet?
podman machine start regtest

# Part 2
podman images
# Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM

We’re getting the errors above because (in Part 1) you do not have a machine running in the first place, and in Part 2, (even after starting the machine) it’s not your default machine, and so this new machine is not the connection the podman command connects to. To solve this, you can either specify the connection in each podman command you call or set the default connection to your new machine like so:

# Declare which machine to speak to
podman --connection regtest system info
podman --connection regtest images

# Set specific machine as default connection
podman system connection default regtest
podman system info
podman images

Great! We have our new machine. Now let’s turn it on (and off).

# boot up the machine
podman machine start regtest

# when you're done, turn it off
podman machine stop regtest

3. Containers

Containers are built from images. You can think of an image as the cookie cutter, and the container as the cookie built from that cutter. We start and stop containers like so:

podman start ContainerName
podman stop ContainerName

Setting up and configuring an image

To start, we’ll create a directory ~/podman/ on our local machine that will help store useful files for the task at hand. I store the files under a subdirectory named after the image I plan on creating.

mkdir ~/podman/regtest/

This directory will host 3 files:

  1. Containerfile
  2. start-services.sh
  3. aliases.sh

The Containerfile is the file that describes to podman how to build the image; start-services.sh is a script that gets run everytime a container built from this image is started; and aliases.sh is a script we’ll run in our shell to provide useful aliases for common commands.

Here is our Containerfile:

FROM debian:bookworm

# Install wget dependency
RUN apt-get update && apt-get install -y wget

# Setup bitcoin core binaries download
ENV BITCOIN_VERSION=26.0
ENV BITCOIN_TARBALL=bitcoin-${BITCOIN_VERSION}-aarch64-linux-gnu.tar.gz
ENV BITCOIN_URL=https://bitcoincore.org/bin/bitcoin-core-${BITCOIN_VERSION}/${BITCOIN_TARBALL}

# Install bitcoin core
RUN wget ${BITCOIN_URL} \
    && tar -xzvf ${BITCOIN_TARBALL} -C /opt \
    && rm ${BITCOIN_TARBALL} \
    && ln -s /opt/bitcoin-${BITCOIN_VERSION}/bin/* /usr/local/bin/

# Install dependencies for Esplora
RUN apt-get update && apt-get install -y \
    curl \
    git \
    build-essential \
    pkg-config \
    libssl-dev \
    libclang-dev \
    netcat-openbsd \
    nano \
    && rm -rf /var/lib/apt/lists/*

# Install dependencies for building Rust projects
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain 1.75.0
# Set the environment variable so that Cargo is in PATH
ENV PATH="/root/.cargo/bin:${PATH}"
RUN rustup default 1.75.0

# Build Esplora and Electrum services
WORKDIR /root/electrs/
RUN rustup update
RUN git clone https://github.com/Blockstream/electrs.git .
RUN git checkout new-index
RUN cargo build --release

# Build Fast Bitcoin Block Explorer
WORKDIR /root/fbbe/
RUN git clone https://github.com/RCasatta/fbbe .
RUN rustup override set 1.75.0
RUN cargo build --release

COPY start-services.sh /usr/local/bin/
ENTRYPOINT ["start-services.sh"]

Note that the last two commands look for a script called start-services.sh. Here is the content of this script:

#!/bin/bash

# Create the log directory
mkdir ~/log/

# Start the bitcoin daemon
bitcoind --chain=regtest --txindex --blockfilterindex --peerblockfilters --rpcbind=0.0.0.0 --rpcallowip=0.0.0.0/0 --rpcport=18443 --rest --printtoconsole > ~/log/bitcoin.log 2>&1 &
sleep 10

# Start the blockchain explorer
/root/fbbe/target/release/fbbe --network regtest --local-addr 0.0.0.0:3003 > ~/log/fbbe.log 2>&1 &
sleep 10

# Start the Esplora and Electrum services
/root/electrs/target/release/electrs -vvvv --daemon-dir /root/.bitcoin/ --http-addr 0.0.0.0:3002 --electrum-rpc-addr 0.0.0.0:60401 --network=regtest --lightmode > ~/log/esplora.log 2>&1 &
sleep 10

# Mine 3 blocks
COOKIE=$(cat /root/.bitcoin/regtest/.cookie | cut -d ':' -f2)
bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword="$COOKIE" generatetoaddress 3 bcrt1q6gau5mg4ceupfhtyywyaj5ge45vgptvawgg3aq

wait

Now we have everything we need to build the image, which we do using the following command:

cd ~/podman/regtest/
podman --connection regtest build --tag localhost/regtest:v1.0.0 --file ./Containerfile

# Verify that the image was created
podman --connection regtest images

Now that we have an image, we can build containers using it:

# Create the container
podman create --name RegtestBitcoinEnv --publish 18443:18443 --publish 18444:18444 --publish 3002:3002 --publish 3003:3003 --publish 60401:60401 localhost/regtest:v1.0.0

You can ask podman to show you what containers are available using

# Will print an empty list because your container is not running
podman --connection regtest ps

# Prints even containers that are not running
podman --connection regtest ps --all

We’re ready to start our container! Start and stop the container using

podman start RegtestBitcoinEnv
podman stop RegtestBitcoinEnv

You should now be able to access the bitcoin daemon running inside the container using a local version of bitcoin-cli:

bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=<randompassword> getblockchaininfo
bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=<randompassword> generatetoaddress 10 bcrt1q6gau5mg4ceupfhtyywyaj5ge45vgptvawgg3aq

Note that the rpcpassword argument needs the password provided in the cookie file created by the bitcoin daemon inside the container. You can log into a shell inside the container and go retrieve it (/root/.bitcoin/regtest/.cookie), but this is slow and will require you to fetch the cookie every time you restart the container (the cookie file changes at every boot of the bitcoin daemon).

Instead, we create a file containing aliases that will fetch this cookie for us and feed it to the bitcoin-cli tool when we query the daemon.

Just commands

I recommend using the just tool to interact with your container by defining a justfile and adding a few commands. Once that’s done, you’ll be able to use your command line like so:

just mine 21
just sendto <address>
just esploralogs
just stop

Here are a few examples of useful just commands:

@default:
  just --list

@cookie:
  podman exec RegtestBitcoinEnv cat /root/.bitcoin/regtest/.cookie | cut -d ':' -f2

@mine BLOCKS="1":
  COOKIE=$(just cookie) && bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=$COOKIE generatetoaddress {{BLOCKS}} bcrt1q6gau5mg4ceupfhtyywyaj5ge45vgptvawgg3aq

@sendto ADDRESS:
  COOKIE=$(just cookie) && bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=$COOKIE generatetoaddress 1 {{ADDRESS}}

podshell:
  podman exec -it RegtestBitcoinEnv /bin/bash

explorer:
  open http://127.0.0.1:3003

logs:
  podman logs RegtestBitcoinEnv

bitcoindlogs:
  podman exec -it RegtestBitcoinEnv tail -f /root/log/bitcoin.log

esploralogs:
  podman exec -it RegtestBitcoinEnv tail -f /root/log/esplora.log

explorerlogs:
  podman exec -it RegtestBitcoinEnv tail -f /root/log/fbbe.log

stop:
  podman stop RegtestBitcoinEnv && podman machine stop regtest

Aliases

If you do not have or do not wish to use the just tool, you can instead create direct aliases for the common commands you want available quickly. Add a file called aliases.sh to your directory and add the following contents:

COOKIE=$(podman exec RegtestBitcoinEnv cat /root/.bitcoin/regtest/.cookie | cut -d ':' -f2)

# Print the cookie password for this session to the console
alias podcookie="podman exec -i RegtestBitcoinEnv cat /root/.bitcoin/regtest/.cookie | cut -d ':' -f2"

# Common bitcoin-cli commands
alias podcli="bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=$COOKIE"
alias podmine="bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=$COOKIE generatetoaddress 1 bcrt1q6gau5mg4ceupfhtyywyaj5ge45vgptvawgg3aq"

# Podman related commands
alias podshell="podman exec -it RegtestBitcoinEnv /bin/bash"
alias podlogs="podman logs RegtestBitcoinEnv"
alias podstop="podman stop RegtestBitcoinEnv"

# Open the block explorer in a browser
alias explorer="open http://127.0.0.1:3003"

Take some time to get familiar with the commands above, and customize this script to your liking! Now all we have to do it to run this script into our shell to load up those aliases, and start playing with the container.

source aliases.sh

podlogs
podcli getblockchaininfo
podmine

Using your container

Once the work from the sections above is done, all that’s left to do is to start/stop the container whenever you need a bitcoin regtest enviroment! The workflow would usually be as follows:

# Start the machine
podman machine start regtest

# Start the container
podman start RegtestBitcoinEnv

# Set up the aliases for the session
source ~/podman/regtest/aliases.sh

Test your services

# Mine 1 block
bitcoin-cli --chain=regtest --rpcuser=__cookie__ --rpcpassword=<passwordfromcookie> generatetoaddress 1 bcrt1q6gau5mg4ceupfhtyywyaj5ge45vgptvawgg3aq

# Open your block explorer
open http://127.0.0.1:3003

# Query Esplora for the latest block
curl 127.0.0.1:3002/blocks/tip/height ; echo

# Query Electrum for it's current version
echo '{"jsonrpc": "2.0", "method": "server.version", "id": 0, "params": []}' | netcat 127.0.0.1 60401 | jq

Working with the tools in the container

1. Esplora

curl 127.0.0.1:3002/blocks/tip/height

Connect to your Esplora service inside an Android emulator using

val esploraUrl = "http://10.0.2.2:3002"

2. Electrum server

You can now connect your electrum clients on Android emulators to your electrum server using the following address:

val electrumUrl = "tcp://10.0.2.2:60401"

3. Block explorer

An fbbe block explorer is open at http://127.0.0.1:3003.

Tips and tricks

Here are some extra tips and tricks for working with podman and your new container. Get into a shell inside your container using

podman exec -it RegtestBitcoinEnv /bin/bash

If you rebuild images too much, you’ll max out the space on your VM. To clean up images, use

# Remove unused images
podman image prune

# Remove an image
podman image rm <imageid>

To remove a container, use

podman rm RegtestBitcoinEnv