Aster-maintained iroh-docs fork
Find a file
Ben Hagen fc894617c4
Some checks failed
CI / codespell (push) Has been cancelled
CI / CI Test Suite (push) Has been cancelled
CI / Cross Build Only (push) Has been cancelled
CI / Android Build Only (push) Has been cancelled
CI / Android Build Only-1 (push) Has been cancelled
CI / Cross Test (push) Has been cancelled
CI / check_semver (push) Has been cancelled
CI / Checking fmt (push) Has been cancelled
CI / Checking docs (push) Has been cancelled
CI / clippy_check (push) Has been cancelled
CI / Minimal Supported Rust Version (push) Has been cancelled
CI / cargo deny (push) Has been cancelled
CI / Build & test wasm32 (push) Has been cancelled
CI / Tests (push) Has been cancelled
CI / Tests-1 (push) Has been cancelled
CI / Tests-2 (push) Has been cancelled
CI / Tests-3 (push) Has been cancelled
CI / Tests-4 (push) Has been cancelled
CI / Tests-5 (push) Has been cancelled
CI / Tests-6 (push) Has been cancelled
CI / Tests-7 (push) Has been cancelled
CI / Tests-8 (push) Has been cancelled
fix(store): migrate redb 2.x tuple tables on open (#105)
## Description

Fixes #104.

Stores created or opened by iroh-docs 0.94 through 0.98 (redb 2.6)
cannot be opened under 0.99+ (redb 4): `Tables::new` fails with
`TableTypeMismatch` on `records-1`, `records-by-key-1`, and
`latest-by-author-1`. The issue write-up has the full root cause
analysis. In short, redb 3.0 changed the on-disk type tag for
variable-width tuples, and #100 jumped from redb 2.6 straight to redb
4.1, skipping the redb 3.x release line where the documented
`Legacy<...>` migration would have run. By the time a user is on 0.99+,
the migration window is closed inside any release pinned to redb 4
(which dropped the `Legacy` type entirely).

This PR keeps redb 4.1 as the primary dependency and adds redb 3.1
alongside it for the one job that requires it. When `Store::persistent`
sees a `TableTypeMismatch` from the initial `Tables::new`, it:

1. Opens the file with redb 3.
2. Reads the three affected tables through `Legacy<...>` wrappers (which
match the redb 2.6 stamping).
3. Copies every table into a fresh redb 3 file at a temp path in the
same directory, using plain tuple types on the write side. That commit
re-stamps the metadata as `Internal2`.
4. Renames the original to `<path>.backup-redb-v2-tuples` and persists
the temp file in its place.

redb 4 then opens the result normally and the existing data migrations
(`migrations::run_migrations`) run as before.

Changes:

- `Cargo.toml`: add `redb_v3 = { package = "redb", version = "3.1" }`.
- `src/store/fs/migrate_redb_v2_tuples.rs` (new): migration logic. Uses
the same `migrate_table!` / `migrate_multimap_table!` macro style as the
existing `migrate_v1_v2.rs`.
- `src/store/fs.rs`: `Store::persistent` catches
`TableError::TableTypeMismatch` from `new_impl`, runs the migration, and
retries once.
- Tests: one synthesises a redb 2.x-shaped file by writing through
`Legacy<...>` with redb 3, then asserts (a) redb 4 rejects it before
migration, (b) `Store::persistent` succeeds, (c) the backup file is
created, and (d) all rows survive when read back with redb 4 plain
types. A second test asserts a fresh open never creates a backup file.

## Breaking Changes

None for users already on 0.99+. Users coming from 0.94..=0.98 will see
their original file renamed to `<path>.backup-redb-v2-tuples` on first
open. The new file at the original path holds the same data.

## Notes & open questions

- redb 3 needs to ship as long as we want to migrate stores written by
0.94..=0.98 directly. Dropping the dependency later means those users
would have to install an intermediate iroh-docs release first.
- The fixed-width tuple tables (`namespaces-2`, `sync-peers-1`) are not
affected, because variable-width tuples are the only thing redb 3
changed the tag for. The migration still copies them so the resulting
file is a clean redb 3 write top to bottom.

## Change checklist

- [x] Self-review.
- [ ] Documentation updates following the [style
guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text),
if relevant.
- [x] Tests if relevant.
- [x] All breaking changes documented.
2026-06-01 13:37:51 +02:00
.cargo feat: build on wasm (#75) 2026-02-24 14:59:35 +01:00
.config feat: ci (#1) 2024-10-23 14:43:04 +02:00
.github feat!: update to latest 1.0.0-rc.0 deps and redb@4 (#100) 2026-05-08 19:15:48 +02:00
examples feat: update to iroh@main (#85) 2026-03-17 14:43:56 +01:00
proptest-regressions refactor: renames iroh-sync & iroh-bytes (#2271) 2024-05-06 16:29:45 +00:00
src fix(store): migrate redb 2.x tuple tables on open (#105) 2026-06-01 13:37:51 +02:00
tests feat!: update to latest 1.0.0-rc.0 deps and redb@4 (#100) 2026-05-08 19:15:48 +02:00
.gitignore chore: copy missing files 2024-10-23 12:16:33 +02:00
build.rs feat: build on wasm (#75) 2026-02-24 14:59:35 +01:00
Cargo.lock fix(store): migrate redb 2.x tuple tables on open (#105) 2026-06-01 13:37:51 +02:00
Cargo.toml fix(store): migrate redb 2.x tuple tables on open (#105) 2026-06-01 13:37:51 +02:00
CHANGELOG.md chore: Release iroh-docs version 0.100.0 2026-05-27 11:44:30 +02:00
cliff.toml chore: init changelog 2024-12-04 10:32:58 +01:00
code_of_conduct.md chore: copy missing files 2024-10-23 12:16:33 +02:00
deny.toml chore: upgrade to iroh 0.96 2026-01-28 21:19:46 -05:00
LICENSE-APACHE feat!: update to latest 1.0.0-rc.0 deps and redb@4 (#100) 2026-05-08 19:15:48 +02:00
LICENSE-MIT feat!: update to latest 1.0.0-rc.0 deps and redb@4 (#100) 2026-05-08 19:15:48 +02:00
Makefile.toml chore: copy missing files 2024-10-23 12:16:33 +02:00
README.md feat!: update to latest 1.0.0-rc.0 deps and redb@4 (#100) 2026-05-08 19:15:48 +02:00
release.toml chore: fix release config 2024-12-04 11:22:32 +01:00

iroh-docs

Documentation Crates.io downloads Chat License: MIT License: Apache 2.0 CI

Multi-dimensional key-value documents with an efficient synchronization protocol.

The crate operates on Replicas. A replica contains an unlimited number of Entries. Each entry is identified by a key, its author, and the replica's namespace. Its value is the 32-byte BLAKE3 hash of the entry's content data, the size of this content data, and a timestamp. The content data itself is not stored or transferred through a replica.

All entries in a replica are signed with two keypairs:

  • The Namespace key, as a token of write capability. The public key is the NamespaceId, which also serves as the unique identifier for a replica.
  • The Author key, as a proof of authorship. Any number of authors may be created, and their semantic meaning is application-specific. The public key of an author is the [AuthorId].

Replicas can be synchronized between peers by exchanging messages. The synchronization algorithm is based on a technique called range-based set reconciliation, based on this paper by Aljoscha Meyer:

Range-based set reconciliation is a simple approach to efficiently compute the union of two sets over a network, based on recursively partitioning the sets and comparing fingerprints of the partitions to probabilistically detect whether a partition requires further work.

The crate exposes a generic storage interface with in-memory and persistent, file-based implementations. The latter makes use of [redb], an embedded key-value store, and persists the whole store with all replicas to a single file.

Getting Started

The entry into the iroh-docs protocol is the Docs struct, which uses an Engine to power the protocol.

Docs was designed to be used in conjunction with iroh. Iroh is a networking library for making direct connections, these connections are peers send sync messages and transfer data.

Iroh provides a Router that takes an Endpoint and any protocols needed for the application. Similar to a router in webserver library, it runs a loop accepting incoming connections and routes them to the specific protocol handler, based on ALPN.

Docs is a "meta protocol" that relies on the iroh-blobs and iroh-gossip protocols. Setting up Docs will require setting up Blobs and Gossip as well.

Here is a basic example of how to set up iroh-docs with iroh:

use iroh::{endpoint::presets, protocol::Router, Endpoint};
use iroh_blobs::{BlobsProtocol, store::mem::MemStore, ALPN as BLOBS_ALPN};
use iroh_docs::{protocol::Docs, ALPN as DOCS_ALPN};
use iroh_gossip::{net::Gossip, ALPN as GOSSIP_ALPN};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // create an iroh endpoint that includes the standard discovery mechanisms
    // we've built at number0
    let endpoint = Endpoint::bind(presets::N0).await?;

    // build the blobs protocol
    let blobs = MemStore::default();

    // build the gossip protocol
    let gossip = Gossip::builder().spawn(endpoint.clone());

    // build the docs protocol
    let docs = Docs::memory()
        .spawn(endpoint.clone(), (*blobs).clone(), gossip.clone())
        .await?;

    // create a router builder, we will add the
    // protocols to this builder and then spawn
    // the router
    let builder = Router::builder(endpoint.clone());

    // setup router
    let _router = builder
        .accept(BLOBS_ALPN, BlobsProtocol::new(&blobs, None))
        .accept(GOSSIP_ALPN, gossip)
        .accept(DOCS_ALPN, docs)
        .spawn();

    // do fun stuff with docs!
    Ok(())
}

License

Copyright 2026 N0, INC.

This project is licensed under either of

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.