Epoch 208

Since the ledger data for about 495 slots (89856108 and 89856602) is missing, we do not have PoH data for these slots. This means that the PoH validation that was conducted on the other slots could not be completed.

Triton One worked with engineers from Anza Labs to solve this issue, which involves sourcing transaction and block data from Google BigTable and validating it through replay. Replaying such old blocks is difficult, but Anza Labs has created a special version of solana-ledger-tool to support this replay.

Basic Theory

The basic theory of this validation is:

1. We have an account state snapshot, including a slot immediately before the gap, and another account state snapshot, which includes a slot immediately after the gap.

2. These account snapshots are based on two slots for which we have valid PoH information.

3. We have validated the PoH hashes for the two slots (before and after the gap) on which the account state snapshots are based.

4. We then replay the transactions sourced from Bigtable and validate that the bank hashes produced at each slot are valid.

5. After replay, we can produce an account state snapshot. This snapshot must match the one available after the Epoch 208 ledger gap.

Going through this process, we can produce a new valid ledger for these 495 slots to generate the bank hashes.

Availability of Bank Hashes

While the data to recompute POH is lost, we still have the bank hashes available, as these are included in the votes submitted by the validators. Bank hashes (see https://github.com/solana-labs/solana/blob/005c825b5c27bd57df9b0aadf0edb2e02215add0/runtime/src/bank.rs#L7047-L7052) contain:

• Previous block hash

• A hash of the delta of all accounts during the slot

• The number of signatures in the block

• The last blockhash of the block (i.e., the one referred to as the block’s ”blockhash”).

As part of the replay of epoch 208, the bank hashes are recomputed and can be compared with those inside the votes submitted by the validators. We can also compute the stake weight of the votes based on the data available in BigTable. Since votes are signed by the validators, they cannot be forged.

This means that during replay, you can reproduce the bank hashes.

Performing your own validation of Epoch 208

Currently, Triton One and Anza Labs have validated Epoch 208. We encourage other parties to run their own validation. The tooling is basic but should enable you to re-run the validation completely. You will need access to the required slots of a Google Bigtable archive (or other archive).

Pre-requisites

Hardware specs

To verify, you need at least the following hardware specs:

• 40 GiB available RAM (total system memory recommended at least 64 GiB)

• 2 TiB available disk space (ideally NVME)

• OS: Ubuntu 22.04.2 LTS

# print used memory, should have at lest 40 GiB available
free -h

# print available disk space,m should have at least 2TiB available
df -h

# print ubuntu version; must be 22.04 LTS
lsb_release -a

# print current version; must be at least GLIBC_2.32; GLIBC 2.35 is good
ldd --version

Bigtable access (or other ledger store)

You need read-only credentials for a Google Bigtable copy of the Solana history or an equivalent source. If you want to reproduce it but do not have access to your own Bigtable, you can contact Anza Labs or Triton One to request access.

Ledger and account snapshots

You need to download the account snapshots that you can use to compare against. The pre-account snapshot is hosted by Anza and is available in a Google Cloud storage bucket called mainnet-beta-ledger-us-ny5. This was produced by warehouse nodes running at the time. This snapshot is from a ledger that has been validated in the normal Old Faithful process.

The RocksDB snapshot at https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89423444/rocksdb.tar.bz2 runs up to slot 89856107. Ledger tool reports:

Ledger has data for 424323 slots 89423444 to 89856107

The latest account state snapshot before the missing slots is at slot 8985546. https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89855469/snapshot-89855469-9i59TwLmzXvbomizFZNuXgPygtx8zJ3P4nWDQuNxrTxK.tar.zst).

After the gap, the ledger snapshot at https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89855469/rocksdb.tar.bz2 starts at 89856602. Ledger tool reports:

Ledger has data for 424529 slots 89856602 to 90288016

The account state snapshot after the gap that we have available is at 89865552 https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89855469/hourly/snapshot-89865552-CWFaiDterTZ1EFQrJkFghnnzERFJNNFT2USBnYTssxsF.tar.zst

This means that if you want to use the existing snapshots for validation, you need to replay between at least slot 89855469 and 89865552 (10083 slots).

Building solana-ledger-tool

A special version of the solana-ledger-tool has been prepared, which has a patch to allow running the verification and skipping PoH verification (since PoH data is missing).

git clone [e-mail]:solana-labs/solana-ledger-gap.git
cd solana-ledger-gap
git checkout origin/steviez_v1.7.9

# install dependencies
sudo apt-get install \
    build-essential \
    pkg-config \
    libudev-dev llvm libclang-dev \
    protobuf-compiler \
    libssl-dev \
    apt-file

# install rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source $HOME/.cargo/env
. "$HOME/.cargo/env"

cd ledger-tool

nano Cargo.toml
# add the following lines to [dependencies]
```
openssl-sys = "0.9.58"
openssl = "0.10.30"
```

# install rust nightly 1.57.0
rustup install nightly-2021-12-06
rustup toolchain install nightly-2021-12-06

# build solana-ledger-tool
cargo +nightly-2021-12-06 build --release

# the final binary is located at ./target/release/solana-ledger-tool
./solana-ledger-gap/target/release/solana-ledger-tool

# copy the binary to a location that is in the PATH (e.g. /usr/local/bin)
cp ./target/release/solana-ledger-tool /usr/local/bin/solana-ledger-tool

Running verification

Running the verification will take some time. We recommend using a tmux session with an unlimited scrollback buffer to run the verification.

# create a working directory
export WORKING_DIR_E208=/solana/e208
mkdir $WORKING_DIR_E208
mkdir $WORKING_DIR_E208/ledger
cd $WORKING_DIR_E208

# download the pre-gap snapshots
apt install -y aria2 pv bzip2 lbzip2 zstd
aria2c -x 16 -s 16 -d $WORKING_DIR_E208/ledger https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89423444/rocksdb.tar.bz2
aria2c -x 16 -s 16 -d $WORKING_DIR_E208/ledger https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89855469/snapshot-89855469-9i59TwLmzXvbomizFZNuXgPygtx8zJ3P4nWDQuNxrTxK.tar.zst

# get genesis file
cd $WORKING_DIR_E208/ledger
wget https://api.mainnet-beta.solana.com/genesis.tar.bz2
tar -xvf genesis.tar.bz2 -C $WORKING_DIR_E208/ledger/

# extract rocksdb (will take ~10 minutes; 2 hours if using networked storage)
cd $WORKING_DIR_E208/ledger
pv rocksdb.tar.bz2 | tar -x -I lbzip2 -C ./

# Step 0: Create a snapshot immediately before the gap
# This will take ~10 minutes
alias solana-ledger-tool="$WORKING_DIR_E208/solana-ledger-tool"
sudo sysctl -w vm.max_map_count=1400000
sudo sysctl --system
cd $WORKING_DIR_E208/ledger
solana-ledger-tool create-snapshot --ledger ./ 89856107 ./

# Step I: Recreate shreds for the missing blocks (you will need bigtable credentials)
# You need a service account key for bigtable in JSON format (to the Solana Foundation's bigtable)
# It will be used to fetch the missing blocks.
export GOOGLE_APPLICATION_CREDENTIALS=/solana/e208/bigtable.json
solana-ledger-tool bigtable reconstruct-replayable-blocks --ledger ./ --store-transaction-status --starting-slot 89856107 --ending-slot 89856602 --snapshot-archive-path=$WORKING_DIR_E208/ledger/snapshot-89856107-2Uvbz6becuTbDVLdoUXfYN81wcwR5AxVGLJFbkG5Vy2w.tar.zst

# verify bounds
# should be: slots 89423444 to 89856602
solana-ledger-tool bounds --ledger $WORKING_DIR_E208/ledger

# create a snapshot after the gap
RUST_LOG=solana=info,solana_ledger::blockstore_processor=trace solana-ledger-tool create-snapshot --skip-poh-verify --ledger $WORKING_DIR_E208/ledger 89856602 $WORKING_DIR_E208/ledger/89856602/
# this will generate /solana/e208/ledger/89856602/snapshot-89856602-HZTS2FDwEFJFq3XiX6wHrS2297nsfbfRU69bqFkJDXZk.tar.zst

# Now, we need to take that and use it
---

# get Full State After the Gap
mkdir $WORKING_DIR_E208/ledger-after-gap
cd $WORKING_DIR_E208/ledger-after-gap
aria2c -x 16 -s 16 -d $WORKING_DIR_E208/ledger-after-gap https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89855469/rocksdb.tar.bz2
aria2c -x 16 -s 16 -d $WORKING_DIR_E208/ledger-after-gap https://storage.googleapis.com/mainnet-beta-ledger-us-ny5/89855469/hourly/snapshot-89865552-CWFaiDterTZ1EFQrJkFghnnzERFJNNFT2USBnYTssxsF.tar.zst

pv rocksdb.tar.bz2 | tar -x -I lbzip2 -C ./

# copy genesis file
cp $WORKING_DIR_E208/ledger/genesis.bin $WORKING_DIR_E208/ledger-after-gap/genesis.bin

# verify bounds: should be `Ledger has data for 424529 slots 89856602 to 90288016`
solana-ledger-tool bounds --ledger $WORKING_DIR_E208/ledger-after-gap

# Now we can use the snapshot at 89856602 and the ledger-after-gap to create a new snapshot at 89865552,
# which should have the same hash as the downloaded snapshot-89865552-CWFaiDterTZ1EFQrJkFghnnzERFJNNFT2USBnYTssxsF.tar.zst
solana-ledger-tool create-snapshot \
    --skip-poh-verify \
    --ledger $WORKING_DIR_E208/ledger-after-gap \
    --snapshot-archive-path $WORKING_DIR_E208/ledger/89856602/ \
    89865552 $WORKING_DIR_E208/ledger-after-gap/89865552/

After running this process, the $WORKING_DIR_E208/ledger-after-gap/89865552/ should have a file with the same filename as snapshot-89865552-CWFaiDterTZ1EFQrJkFghnnzERFJNNFT2USBnYTssxsF.tar.zstWe can verify the filename hash to confirm that the snapshots are identical. We can also extract and validate that the two account states match identically (i.e., have the exact same accounts and account data).

The two archive files, however, are slightly different in size since the process isn’t precisely reproducible regarding the specific compressed file. If used on the two snapshot-*.tar.zst files, SHA256SUM will return two different hashes. However, if compared, the contents (account states) are the same.