- Rust 100%
|
Some checks failed
CI / codespell (push) Has been cancelled
CI / CI Test Suite (push) Has been cancelled
CI / Cross Build Only (push) Has been cancelled
CI / Android Build Only (push) Has been cancelled
CI / Android Build Only-1 (push) Has been cancelled
CI / Cross Test (push) Has been cancelled
CI / check_semver (push) Has been cancelled
CI / Checking fmt (push) Has been cancelled
CI / Checking docs (push) Has been cancelled
CI / clippy_check (push) Has been cancelled
CI / Minimal Supported Rust Version (push) Has been cancelled
CI / cargo deny (push) Has been cancelled
CI / Build & test wasm32 (push) Has been cancelled
CI / Tests (push) Has been cancelled
CI / Tests-1 (push) Has been cancelled
CI / Tests-2 (push) Has been cancelled
CI / Tests-3 (push) Has been cancelled
CI / Tests-4 (push) Has been cancelled
CI / Tests-5 (push) Has been cancelled
CI / Tests-6 (push) Has been cancelled
CI / Tests-7 (push) Has been cancelled
CI / Tests-8 (push) Has been cancelled
## Description Fixes #104. Stores created or opened by iroh-docs 0.94 through 0.98 (redb 2.6) cannot be opened under 0.99+ (redb 4): `Tables::new` fails with `TableTypeMismatch` on `records-1`, `records-by-key-1`, and `latest-by-author-1`. The issue write-up has the full root cause analysis. In short, redb 3.0 changed the on-disk type tag for variable-width tuples, and #100 jumped from redb 2.6 straight to redb 4.1, skipping the redb 3.x release line where the documented `Legacy<...>` migration would have run. By the time a user is on 0.99+, the migration window is closed inside any release pinned to redb 4 (which dropped the `Legacy` type entirely). This PR keeps redb 4.1 as the primary dependency and adds redb 3.1 alongside it for the one job that requires it. When `Store::persistent` sees a `TableTypeMismatch` from the initial `Tables::new`, it: 1. Opens the file with redb 3. 2. Reads the three affected tables through `Legacy<...>` wrappers (which match the redb 2.6 stamping). 3. Copies every table into a fresh redb 3 file at a temp path in the same directory, using plain tuple types on the write side. That commit re-stamps the metadata as `Internal2`. 4. Renames the original to `<path>.backup-redb-v2-tuples` and persists the temp file in its place. redb 4 then opens the result normally and the existing data migrations (`migrations::run_migrations`) run as before. Changes: - `Cargo.toml`: add `redb_v3 = { package = "redb", version = "3.1" }`. - `src/store/fs/migrate_redb_v2_tuples.rs` (new): migration logic. Uses the same `migrate_table!` / `migrate_multimap_table!` macro style as the existing `migrate_v1_v2.rs`. - `src/store/fs.rs`: `Store::persistent` catches `TableError::TableTypeMismatch` from `new_impl`, runs the migration, and retries once. - Tests: one synthesises a redb 2.x-shaped file by writing through `Legacy<...>` with redb 3, then asserts (a) redb 4 rejects it before migration, (b) `Store::persistent` succeeds, (c) the backup file is created, and (d) all rows survive when read back with redb 4 plain types. A second test asserts a fresh open never creates a backup file. ## Breaking Changes None for users already on 0.99+. Users coming from 0.94..=0.98 will see their original file renamed to `<path>.backup-redb-v2-tuples` on first open. The new file at the original path holds the same data. ## Notes & open questions - redb 3 needs to ship as long as we want to migrate stores written by 0.94..=0.98 directly. Dropping the dependency later means those users would have to install an intermediate iroh-docs release first. - The fixed-width tuple tables (`namespaces-2`, `sync-peers-1`) are not affected, because variable-width tuples are the only thing redb 3 changed the tag for. The migration still copies them so the resulting file is a clean redb 3 write top to bottom. ## Change checklist - [x] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [x] Tests if relevant. - [x] All breaking changes documented. |
||
|---|---|---|
| .cargo | ||
| .config | ||
| .github | ||
| examples | ||
| proptest-regressions | ||
| src | ||
| tests | ||
| .gitignore | ||
| build.rs | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| cliff.toml | ||
| code_of_conduct.md | ||
| deny.toml | ||
| LICENSE-APACHE | ||
| LICENSE-MIT | ||
| Makefile.toml | ||
| README.md | ||
| release.toml | ||
iroh-docs
Multi-dimensional key-value documents with an efficient synchronization protocol.
The crate operates on Replicas. A replica contains an unlimited number of Entries. Each entry is identified by a key, its author, and the replica's namespace. Its value is the 32-byte BLAKE3 hash of the entry's content data, the size of this content data, and a timestamp. The content data itself is not stored or transferred through a replica.
All entries in a replica are signed with two keypairs:
- The Namespace key, as a token of write capability. The public key is the NamespaceId, which also serves as the unique identifier for a replica.
- The Author key, as a proof of authorship. Any number of authors may be created, and their semantic meaning is application-specific. The public key of an author is the [AuthorId].
Replicas can be synchronized between peers by exchanging messages. The synchronization algorithm is based on a technique called range-based set reconciliation, based on this paper by Aljoscha Meyer:
Range-based set reconciliation is a simple approach to efficiently compute the union of two sets over a network, based on recursively partitioning the sets and comparing fingerprints of the partitions to probabilistically detect whether a partition requires further work.
The crate exposes a generic storage interface with in-memory and persistent, file-based
implementations. The latter makes use of [redb], an embedded key-value store, and persists
the whole store with all replicas to a single file.
Getting Started
The entry into the iroh-docs protocol is the Docs struct, which uses an Engine to power the protocol.
Docs was designed to be used in conjunction with iroh. Iroh is a networking library for making direct connections, these connections are peers send sync messages and transfer data.
Iroh provides a Router that takes an Endpoint and any protocols needed for the application. Similar to a router in webserver library, it runs a loop accepting incoming connections and routes them to the specific protocol handler, based on ALPN.
Docs is a "meta protocol" that relies on the iroh-blobs and iroh-gossip protocols. Setting up Docs will require setting up Blobs and Gossip as well.
Here is a basic example of how to set up iroh-docs with iroh:
use iroh::{endpoint::presets, protocol::Router, Endpoint};
use iroh_blobs::{BlobsProtocol, store::mem::MemStore, ALPN as BLOBS_ALPN};
use iroh_docs::{protocol::Docs, ALPN as DOCS_ALPN};
use iroh_gossip::{net::Gossip, ALPN as GOSSIP_ALPN};
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// create an iroh endpoint that includes the standard discovery mechanisms
// we've built at number0
let endpoint = Endpoint::bind(presets::N0).await?;
// build the blobs protocol
let blobs = MemStore::default();
// build the gossip protocol
let gossip = Gossip::builder().spawn(endpoint.clone());
// build the docs protocol
let docs = Docs::memory()
.spawn(endpoint.clone(), (*blobs).clone(), gossip.clone())
.await?;
// create a router builder, we will add the
// protocols to this builder and then spawn
// the router
let builder = Router::builder(endpoint.clone());
// setup router
let _router = builder
.accept(BLOBS_ALPN, BlobsProtocol::new(&blobs, None))
.accept(GOSSIP_ALPN, gossip)
.accept(DOCS_ALPN, docs)
.spawn();
// do fun stuff with docs!
Ok(())
}
License
Copyright 2026 N0, INC.
This project is licensed under either of
- Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)
at your option.
Contribution
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.