Starting a network

This guide sets up a real, public Keepword network: independent witnesses on the internet that audit each other’s logs, agree on who captures which page, and give verdicts that hold up because the witnesses are genuinely independent.

For installing a single node, see INSTALL.md. For why things work the way they do, see DESIGN.md §6.

1. What makes a network trustworthy

A verdict says “witnesses in N independent networks saw this”. The software counts autonomous systems (ASNs), the networks of internet providers, not machines or keys, because keys and machines are free and networks are not. It learns a witness’s ASN by watching where the witness’s connections come from; what a node says about itself is ignored.

So a real network needs:

Different providers usually means different ASNs: for example Hetzner (AS24940), OVH (AS16276), Scaleway (AS12876), DigitalOcean (AS14061), Vultr (AS20473), or a home connection such as Deutsche Telekom (AS3320). Two servers at the same provider are the same ASN, even in different data centers. A witness’s location is where its traffic leaves: behind a VPN or proxy, it is the VPN’s network.

A good first network is 5 witnesses, run by 3 or more people, at 5 different providers, in 3 countries. Three witnesses is the minimum that works at all.

2. Before you start (every witness)

• A Linux server (1 vCPU, 1 GB RAM and 20 GB disk are plenty to start).
• A domain name pointing at it (an A, and AAAA if it has IPv6), e.g. w1.example.org. Use a name you can keep: it’s how peers find the witness.
• Ports 80 and 443 open to everyone (Caddy gets a TLS certificate through port 80). Nothing else needs to be public.
• Time sync on: timedatectl should say System clock synchronized: yes.
• Outbound HTTPS allowed to drand (api.drand.sh), the OpenTimestamps calendars and iptoasn.com.

3. Launch

Step 1: the first witness

git clone https://git.keepword.net/keepword/keepword.git
cd keepword
sudo sh scripts/install.sh --domain w1.example.org --caddy

Check it answers from outside:

curl https://w1.example.org/v1/descriptor

This first witness is the bootstrap peer: the address the others dial first. There is nothing special about it afterwards. Every witness learns every other one through gossip, and any witness’s address works as a bootstrap peer.

Step 2: the other founding witnesses

On each, with its own domain, pointing at the first:

sudo sh scripts/install.sh --domain w2.example.net --caddy \
    --peer https://w1.example.org

Giving two bootstrap peers (repeat --peer) means a new witness still joins when one of them is down.

Step 3: check that the network formed

After two or three minutes (a few sync rounds), on any witness:

keepword net peers        # as root or with sudo
keepword net status

keepword net peers should list every other witness with a log size, a recent last sync, no error, and a location like AS16276 FR (seen directly). keepword net status should show:

• location with an ASN and observers, not not corroborated;
• candidates equal to the number of witnesses;
• epoch ... (drand);
• no warnings at the end. The warnings name anything that keeps the node out of the network, such as a missing endpoint, no ASN table, test settings or failing peers.

Step 4: a first real verdict

On any witness, ask the network to watch a page that doesn’t change often:

keepword request https://www.example.org/ --every 10m --for 1day

Within a sync round, the assigned witnesses show it in keepword watch list. After about ten minutes they have all captured it, and on any witness:

keepword verdict https://www.example.org/
# AGREED by 3 witnesses in 3 independent networks: ...

Then check that the evidence stands on its own. Export a bundle and verify it on a different machine:

keepword export https://www.example.org/ > bundle.json   # on a witness that captured it
keepword verify --bundle bundle.json                           # anywhere

Withdraw the test request when you’re done: keepword request https://www.example.org/ --cancel.

Step 5: make joining one command

Put the endpoints of two or three founding witnesses, ideally run by different people, into DEFAULT_SEEDS in crates/keepword/src/config.rs. Every node built from the code then dials them when it knows few peers, so a newcomer joins with just:

sudo sh scripts/install.sh --domain their.domain --caddy

--peer still adds peers of your own choosing. Seeds are only used until a node knows enough peers, and never on a network with network.allow_private_peers, so a LAN test cluster can’t wander into the public network. To run a separate network, set keepword config set network.seeds false on every witness (the installer’s --no-seeds, the container’s KEEPWORD_SEEDS=0) and bootstrap it with --peer. Do the same for test installs and CI, or they join the public network and linger in its peer tables.

There is no registration and no central server. A new witness counts in verdicts once two others have observed it, and is assigned requests from then on.

4. Settings the whole network must share

Every witness computes the assignment itself, so these must be the same on every witness, or witnesses disagree about who captures what:

These only change how your witness judges verdicts, but keep the defaults so all witnesses report the same thing: quorum.min_asns = 3, quorum.min_dissent_asns = 2, quorum.min_observers = 2, quorum.recheck_quorum = 3, quorum.recheck_secs = 1800, quorum.split_confirmations = 3 of quorum.split_rounds = 4, quorum.max_failed_claims = 5.

Rechecks need witnesses to spare. When the witnesses assigned to a URL disagree, witnesses drawn at random from the same countries, not assigned to it, capture it again, and only versions they reproduce count (DESIGN.md §6.5). A country needs at least quorum.recheck_quorum (3) witnesses on different networks beyond those assigned before a difference seen from there can be confirmed. Until then disagreements stay DISPUTED, and no split alert is raised.

Never enable these on a public witness. They exist for test networks, and keepword net status warns about them: quorum.trust_self_reported, beacon.allow_insecure_seed, network.allow_private_peers.

5. Each operator’s own choices

Change settings with keepword config set KEY VALUE, then sudo systemctl restart keepword.

6. Resources

Each witness talks to a random sample of 16 peers a minute, audits about 16 logs an hour with a consistency proof, and fetches other witnesses’ attestations only for the URLs it follows. So its costs barely depend on how big the network is:

network.max_peers (2000) bounds the peer table; raise it for a larger network.

7. Running it

• Upgrades: git pull and re-run the installer with the same options. Keys and data are kept. Nodes skip message types they don’t know, so witnesses can upgrade one at a time, but upgrade within days.
• Backups: /var/lib/keepword/witness.key is the witness’s identity. A lost key means a new witness, and the old log can never be extended.
• Monitoring: keepword net status (warnings), keepword alerts, journalctl -u keepword. A peer that keeps failing shows its error in keepword net peers.
• Automatic: syncing every minute, Bitcoin anchoring every hour, the IP-to-ASN table refresh every week.
• Retiring a witness: export the bundles that matter first (keepword export); bundles verify on their own, but nobody else keeps a full copy of a witness’s log. Keep the key. Peers retry it every ten minutes, stop assigning it requests after a week, and drop it first when their peer table fills up. keepword net remove-peer KEY drops it at once.

8. Moving on from a test cluster

Test witnesses have signed attestations with test settings (fake ASNs, a predictable seed) into their permanent logs. For the public network, start the witnesses over with new keys:

sudo sh scripts/install.sh --uninstall --purge     # deletes key and log
sudo sh scripts/install.sh --domain wN.example.org --caddy --peer https://w1.example.org

To keep a test witness’s key instead, undo every test setting (keepword config unset quorum.trust_self_reported, and likewise network.max_per_country, quorum.min_asns, quorum.min_dissent_asns, network.allow_private_peers, vantage.asn), set its public endpoint and peers, remove the old LAN peers with keepword net remove-peer, and restart it.

9. Troubleshooting