Slovary Pack Format v1

An open, self-contained format for dictionary content that the Slovary app can install. Anyone can build a compatible pack: this document is the complete specification, and the reference converter is published alongside it.

The app also imports the widely used StarDict format (.ifo/.idx/.dict or .dict.dz); StarDict files are converted to this format on import.

We do not host, distribute or curate third-party dictionary content. This specification exists so that users can bring their own.

1. Overview

A dictionary is delivered as one or two files:

The vector pack is optional. Without it the dictionary is fully searchable lexically (exact, prefix, substring, fuzzy, full-text); with it the dictionary also participates in meaning-based search.

Several dictionaries can travel together in a bundle — one file, one link, installed in a single step. See §6a.

2. Dictionary pack (.slovarypack)

A plain SQLite 3 database. No extensions, no triggers, no attached files.

2.1 pack_meta

CREATE TABLE pack_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);

Readers ignore pack_meta keys they do not know; a new optional key is therefore an additive change and keeps format_version 1.

2.2 entries

CREATE TABLE entries (
    local_id   INTEGER PRIMARY KEY,   -- 1..entries_count, dense, pack-local
    term_upper TEXT NOT NULL,         -- headword, upper-cased, no markup
    definition TEXT NOT NULL,         -- article body, restricted HTML (§4)
    lang       TEXT NOT NULL          -- same value as pack_meta.lang
);
CREATE INDEX idx_entries_term ON entries (term_upper);

local_id is local to the pack. It is never a global identifier. The app assigns its own table index when installing and addresses entries as (pack_id, local_id).

Why this matters: the app's bundled database historically used a composite global id (table_index << 32 | source_rowid). With user-supplied packs that scheme collides as soon as two independently built packs pick the same table index, and the failure is silent — wrong articles for right queries. Packs therefore never carry global ids.

term_upper must be upper-cased with toUpperCase() semantics for its language and must contain no HTML. Searching and sorting both use this column.

2.3 entries_fts

CREATE VIRTUAL TABLE entries_fts USING fts5(
    definition_plain,
    content='',
    contentless_delete=1,
    tokenize='unicode61 remove_diacritics 2',
    detail=none
);

Contentless FTS5 index over the markup-stripped definition text. Row i of the index must correspond to entries.local_id = i.

If the index is missing the app rebuilds it on import — that is slower but not an error.

FTS5 caveat inherited from the main database: the trigram tokenizer cannot match strings shorter than three characters. This format uses unicode61 instead, so short words work, but substring search is handled by the app's own cascade rather than by FTS.

3. Vector pack (.vectors.zip)

A ZIP archive with exactly three members. Vectors are INT8-quantised per-dimension.

3.1 vectors_meta.json

{
  "format_version": 1,
  "pack_id": "org.gnu.gcide",
  "entries_count": 189421,
  "count": 189421,
  "d": 384,
  "embedding_model": "intfloat/multilingual-e5-small",
  "model_revision": "2026-05-01",
  "quantization": "int8-per-dim-minmax",
  "query_prefix": "query: ",
  "passage_prefix": "passage: ",
  "vmin": [ /* d floats */ ],
  "vdiff": [ /* d floats */ ]
}

Dequantisation of code c (0…255) in dimension j:

value[j] = vmin[j] + (c + 0.5) * vdiff[j] / 255.0

3.2 vectors.sq8

Dense little-endian uint8 array of count × d bytes, row-major, ordered by ascending local_id (row i ⇒ local_id = i + 1). No header, no padding.

3.3 ids.bin

Little-endian int64 array of count entries: the local_id for each row, in the same order. Present so that a pack may ship vectors for a subset of entries in a later format version; in v1 it is the identity sequence and is verified as such.

3.4 What the app does with it

The query is encoded on-device with the declared model and scored against every vector by inner product (exhaustive scan, no ANN index). Top matches are resolved back to entries rows. Expect roughly 400 bytes per entry on disk.

4. Allowed markup in definition

The article body is rendered inside a WebView. Only this tag set survives import; everything else is stripped, including all attributes not listed.

Structural blocks: <b-q>, <tb-q>, <g-m>, <t-p>, <v-t>, <o-u>, <s-o>, <f-r>, <d-t>, <e-x>, <x-p>, <n-m>, <o-t>, <r-f>, <i-n>

Inline: <b>, <i>, <u>, <sup>, <sub>, <br>, <b-r>, <n-r>, <b-b>, <ab-r>, <c-r>, <s-m>, <k-f>, <t-r>, <ya-z>, <r-l>, <s-n>, <m-k>, <c-b>, <c-g>, <e-t>

Semantic (vector) data: <d-block>, <d-ex>, <d-abbr>, <d-gram>, <d-ref>, <d-num>, <d-src>, <d-form>, <d-small>, <d-mark>, <d-sep>, <d-phr>, <d-term-var>, <d-lang>, <d-type>, <d-shade>, <d-etym>, <d-note>, <d-gl>, <d-pos>, <d-use>, <d-caption>, <d-syn>, <d-tr>, <d-quote>

Attributes (on any allowed tag, value must match the shape exactly or the attribute is dropped): data-bl (0|1), data-ileft, data-itop, data-fsize (1–3 digits), data-case (1–2 lowercase letters), data-s (one digit), data-ref-collapsed (0|1), data-t (cross-link lemma, ≤ 120 chars, no <>&" or control characters). Without them the article layout is lost: packs built when this list held no d-* tags imported as one run-on paragraph per article.

Cross-reference: <r> — looked up as a term when tapped: data-t when present, else its text content.

Image: <img src="..."> where src is a relative path inside the pack's optional image directory. Remote URLs are removed.

Removed unconditionally: <script>, <style>, <iframe>, <object>, <embed>, <link>, <meta>, event-handler attributes (onclick, onerror, …), javascript: and data: URLs, and anything else not on the list above.

This is a hard security boundary, not a style preference. The article WebView exposes a JavaScript bridge to the app; a script tag surviving import would reach it. Sanitisation happens on import, before the row is written, so the stored data is already safe and no renderer can be tricked later.

5. Validation on import

The app refuses a pack and reports the reason when:

1. format_version has an unknown major version.
2. pack_meta is missing a required key.
3. entries_count disagrees with COUNT(*) from entries.
4. local_id values are not a dense 1..entries_count sequence.
5. A vector pack's pack_id, entries_count, d, or embedding_model does not match its dictionary pack / the app's on-device encoder.
6. vectors.sq8 is not exactly count × d bytes.

Warnings that do not block installation: missing FTS index (rebuilt), missing optional metadata, unknown extra pack_meta keys.

The imported file is never opened as the app's working database. Rows are read out of it read-only, sanitised, and written into a fresh app-owned database; the source file is then discarded.

6. Building a pack

The reference converter builds a pack from StarDict or from a simple JSONL file, and a companion script produces the vector pack. Both are published with this specification.

Minimal JSONL input — one object per line:

{"term": "abacus", "definition": "A counting frame...", "lang": "en"}

# dictionary pack
python make_pack.py --in dict.jsonl --out org.example.mydict.slovarypack \
    --pack-id org.example.mydict --name "My Dictionary" --lang en \
    --license public-domain

# from StarDict instead
python make_pack.py --in mydict.ifo --out org.example.mydict.slovarypack ...

# optional vector pack (needs the E5 model; GPU recommended)
python make_vectors.py --pack org.example.mydict.slovarypack \
    --out org.example.mydict.vectors.zip

Host the resulting files anywhere. In the app: Settings → Dictionaries → Import, paste the URL or pick the file.

6a. Bundle: many dictionaries in one file

A bundle is a plain ZIP that carries whole dictionaries instead of one. It exists so a collection can be handed over as a single link: importing twenty dictionaries and their vectors by pasting forty URLs is not a workable instruction, for a user or for a reviewer.

Rules:

• The name carries the link between the two. A vector member is applied to

the dictionary whose pack_id matches the part of the name before .vectors.zip; a vector member with no matching pack is ignored. The contents are still validated as in §3 — a mismatched pack_id or entry count inside vectors_meta.json is refused even when the file name fits.

• Subdirectories are ignored; only the base name of each member is used.

A member whose name would escape the extraction directory is skipped.

• Order does not matter. Packs are installed in name order, and each one's

vectors immediately after it.

• Nothing else is read. Other members (README, licences) are left alone.
• A member that fails does not abort the rest. With dozens of dictionaries

in one file, losing all of them to one bad member would be the wrong trade; the app installs what it can and reports what it could not.

• Extension: .zip works. .slovarybundle is the suggested name when you want

the file to be self-describing — the app identifies the format by looking inside, never by the extension.

An importer distinguishes a bundle from a StarDict archive by looking for a member ending in .slovarypack; only the ZIP central directory is read, so this costs nothing on a large file.

The listing, and why it is the first member

A large bundle runs to hundreds of megabytes and a reader rarely wants all of it. bundle.json describes the contents so an importer can ask before downloading anything:

{
  "format": "slovary-bundle",
  "format_version": 2,
  "entries_count": 357644,
  "dictionaries": [
    {"pack_id": "org.gnu.gcide", "name": "GCIDE", "lang": "en",
     "entries_count": 124187, "size": 58720256, "vectors_size": 41943040,
     "pack_version": "2026-08-13", "license": "GPL-3.0-or-later"}
  ]
}

size and vectors_size are the member sizes in bytes as stored in the archive — what fetching the member costs; for a DEFLATE member (v3) that is the compressed size, and unpacked_size gives the file's size after inflating (vectors_size absent or 0 means the bundle carries no vectors for that dictionary). Everything here is advertising: the app still validates each member on install, so a listing that lies costs the bundle its credibility, not the app its integrity.

Two fields decide what the importer offers before anything is fetched, and leaving them out makes a technically valid bundle behave worse than it should:

• pack_version is compared against the version already installed. Equal

means "you have this" — the row arrives marked and unticked. Different means an update, and arrives ticked. Omit it and every dictionary looks new, so reopening the same link offers to download the whole collection again. Use the same value as the pack's own pack_meta.pack_version.

• voice_terms (bool, make_bundle.py writes it for every row) repeats the

member's own pack_meta.voice_terms (§2.1). A reader holding that dictionary at exactly the listed pack_version takes the mark from here — the pack it installed may predate the key — so a mark change needs no new version and no download. Absent = the listing says nothing (older builders); readers that do not know it ignore it.

• lang groups the rows and is what the app un-hides after installing:

a dictionary in a language the user had switched off would otherwise install correctly and then be invisible in search. Use the pack's pack_meta.lang.

ZIP64 is not read. A bundle over 4 GB, or with more than 65535 members, parks 0xFFFFFFFF in the fields that hold member offsets, and the importer refuses to guess: it falls back to downloading the whole file rather than fetching a range that would point at another dictionary's bytes. Keep a bundle under 4 GB if selective installation matters.

It is written first so that it lands in the opening kilobytes of the archive, and STORED. That combination is what lets an importer read it with a single ranged request (Range: bytes=0-32767) instead of downloading the file. A second request for the tail yields the ZIP central directory, which gives the offset, compressed length and method of every member — so the dictionaries the user actually ticked can then be fetched one ranged request each. A ZIP member, STORED or DEFLATE, is one contiguous byte range: STORED, the range IS the file; DEFLATE (v3 dictionaries), the range is a raw DEFLATE stream (RFC 1951, no zlib/gzip wrapper) that inflates to the file. The method is the one in the member's local header (offset 8: 0 STORED, 8 DEFLATE); the range starts after the local header, its name and its extra field, and is compressed size bytes long. Any other method is refused.

A bundle without a listing (or one built by format_version 1, where dictionaries was a count) is still valid — an importer simply falls back to taking the whole thing.

Version 3: compressed dictionaries and a signed listing

format_version 3 adds two independent things. A v3 reader handles both; a bundle may use either.

Compressed dictionaries. <pack_id>.slovarypack members may be DEFLATE (method 8; the reference builder uses level 9 — a SQLite pack shrinks 5–7×). bundle.json, bundle.json.sig and .vectors.zip members stay STORED (the head must be readable as is; a vector pack is already compressed). Each dictionary row then carries unpacked_size. A reader fetches the member's range as above and raw-inflates it; it checks the free space against unpacked_size before writing and the inflated length against it after. A reader older than v3 (every Slovary app build before the auto-update release) writes the compressed range as if it were the file and refuses it as an invalid pack — those builds need an app update to install from a v3 bundle selectively. One version number covers both features because no reader of a "signed but STORED-only" v3 was ever shipped (v3 was defined on 2026-10-22 together with compression); a second number would buy nothing. make_bundle.py --stored still writes a STORED-only v2 bundle for old readers.

Signed listing. A listing can be signed, so an importer that updates dictionaries without asking can prove every byte it installs came from the publisher:

1. sha256 per member, inside bundle.json. Each dictionary row carries sha256 — lowercase hex SHA-256 of the bytes of its <pack_id>.slovarypack member as stored in the archive, i.e. exactly the bytes a ranged request for the member returns (local header, name and extra field excluded; for a DEFLATE member the compressed stream, not the inflated file) — and, when it has vectors, vectors_sha256 for its <pack_id>.vectors.zip member (STORED, so also the file's hash). Nothing in the listing hashes inflated bytes: the reader verifies the hash BEFORE inflating, inflating an authenticated stream is deterministic, and the inflated file is then validated as in §5.
2. bundle.json.sig as the SECOND member, STORED, immediately after bundle.json: the Ed25519 signature over the exact bytes of the bundle.json member (no re-serialisation, no normalisation of whitespace or key order), written as standard base64 in ASCII (88 characters, no line break required; a reader trims whitespace). It lands in the same 32 KB head read as the listing.

{
  "format": "slovary-bundle",
  "format_version": 3,
  "entries_count": 357644,
  "signature": {"alg": "ed25519", "member": "bundle.json.sig",
                "key": "51288eb60ee77372f1d390b5c3ec78e8acfeb39b7aa021c83b9fd0530da46150"},
  "dictionaries": [
    {"pack_id": "org.gnu.gcide", "name": "GCIDE", "lang": "en",
     "entries_count": 124187, "size": 11534336, "unpacked_size": 58720256,
     "vectors_size": 41943040,
     "pack_version": "2026-08-13", "license": "GPL-3.0-or-later",
     "sha256": "9f2c…", "vectors_sha256": "41d0…"}
  ]
}

• Which key. The signature verifies against ONE public key the reader

already holds; the signature.key field (hex of the 32-byte raw public key) only names it and is never trusted on its own. The Slovary app holds the key of its update manifests (manifestPublicKeyHex, 51288eb6…0da46150, also config/keys/mobile_manifest_ed25519.pub in the publisher's repository): key different from the held key ⇒ treat the bundle as unsigned. Third-party bundles are unsigned by default — their publishers have no key the app trusts, and that is fine.

• Reader, signed bundle (v3, signature present): read member #1

(bundle.json) and member #2 (bundle.json.sig) from the head; verify the signature over the member-#1 bytes before parsing them. Valid ⇒ every member fetched later must match its listed sha256/vectors_sha256 (over the fetched bytes, before inflating) before it is installed; a mismatch rejects that member exactly like a failed validation (§5). Invalid signature, missing or unreadable .sig member, or a signature object that names a member the archive does not have ⇒ the listing is tampered: offer nothing from it and log why. The verdict is about the listing, not one member.

• Reader, unsigned bundle (v1/v2, or v3 without signature): as

before, inflating DEFLATE members — the listing is advertising, every member is still validated (§5). An automatic updater must not install from an unsigned listing without the user's confirmation. A reader that has already seen a valid signed listing from a source should refuse to downgrade that source to an unsigned one silently.

• Hash scope. sha256 covers dictionary and vector members, and each

article delta row carries its own sha256 (next section). A future member type is added the same way: its own row kind under a new top-level key with its own sha256 field — never folded into an existing row — and readers ignore keys they do not know. Because the signature covers the whole bundle.json, any member a future listing names is authenticated the same way without changing the signing scheme.

• Builder. make_bundle.py writes v3 with DEFLATE dictionaries by

default; --sign-key <PEM Ed25519 private key> adds the signature (needs cryptography, imported only then); --stored writes the STORED-only v2 listing. Without --sign-key it needs only the standard library.

Article deltas (deltas, format_version 3)

A weekly refresh edits a handful of articles, yet a changed dictionary costs its whole member (a refresh of 3 edited articles: 16.7 MB of DEFLATE members). A bundle may therefore also carry, per dictionary, delta members: the articles that differ between an older published version of the dictionary (the base) and the version in this bundle. A reader that holds the base applies the delta to its installed copy; any other reader takes the whole member exactly as before.

The article key is (pack_id, local_id). It is a stable key only when the publisher keeps it stable: a newer build of a dictionary must keep every old local_id on the same article and append new articles after the old ones (the reference release builder does). A delta is a plain diff by local_id, so it is correct for any two packs; a publisher that renumbers gets a delta as large as the dictionary, and simply does not ship one (rule of thumb of the reference builder: no delta when it would exceed half of the dictionary's own member + vectors).

Listing. A top-level key deltas (absent = none), one row per delta:

"deltas": [
  {"pack_id": "org.example.terms_en", "from_version": "2026-10-20",
   "to_version": "2026-10-22", "member": "org.example.terms_en.from-2026-10-20.slovarydelta",
   "size": 2311, "unpacked_size": 24576, "sha256": "5be1…",
   "base_entries_count": 2896, "entries_count": 2896,
   "upserts": 1, "deletes": 0, "vectors": true,
   "vectors_space": "2b34d61f…"}
]

vectors_space fingerprint. Lowercase hex SHA-256 over: the UTF-8 text slovary-vectors-space/1\n<d>\n<embedding_model>\n<model_revision>\n<quantization>\n<n_vmin>\n<n_vdiff>\n (d as a decimal integer, absent strings empty, n_* = list lengths), then every vmin value and then every vdiff value as an IEEE-754 binary64, little-endian. It is computed over the parsed numbers of vectors_meta.json (§3.1), never its text, so two copies of the same set hash the same however a writer formats floats; count is not part of it. A reader computes it from its installed vectors_meta.json and compares. Reference: make_delta.vectors_space(); golden: {"d": 4, "embedding_model": "m", "model_revision": "r", "quantization": "int8-per-dim-minmax", "vmin": [0.0, -0.5, 1e-05, 0.1], "vdiff": [1.0, 2.5, 3, 0.30000000000000004]} → 2b34d61f95fa66f90c5788d70fab20c0af9db1621c0d30ecbf66cb3ece39c9b8.

A bundle may carry several deltas for one dictionary, one per base, each squashed straight to to_version — a reader never chains deltas. A phone that missed several releases downloads exactly ONE delta per changed dictionary, covering everything it missed; a phone whose installed version is older than every offered base takes the whole dictionary member.

Reference release policy: bases = the last N = 8 published versions of the dictionary (8 weekly releases ≈ 2 months), newest first. A base is dropped when its delta alone exceeds half of the dictionary's whole cost (deflated pack + vectors), and the oldest bases are dropped once the deltas kept for that dictionary together exceed its whole cost — a long tail of bases must never cost the bundle more than the dictionary itself. Measured 2026-10-22: one edited article ≈ 10 KB per delta (≈ 8 KB of it the vector header), so 8 bases ≈ 80 KB against 0.6–15 MB per dictionary.

Member. DEFLATE (method 8), inflating to a SQLite 3 database:

CREATE TABLE delta_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE pack_meta  (key TEXT PRIMARY KEY, value TEXT NOT NULL); -- the target pack's pack_meta, verbatim
CREATE TABLE upserts (local_id INTEGER PRIMARY KEY, term_upper TEXT NOT NULL,
                      definition TEXT NOT NULL, lang TEXT NOT NULL,
                      base_term_upper TEXT);                         -- NULL = appended
CREATE TABLE deletes (local_id INTEGER PRIMARY KEY, base_term_upper TEXT NOT NULL);
CREATE TABLE vector_rows (local_id INTEGER PRIMARY KEY, codes BLOB NOT NULL); -- may be empty

delta_meta keys: format = slovary-delta, format_version = 1, pack_id, from_version, to_version, base_entries_count, entries_count, and, when vector_rows is filled, vectors_meta = the exact text of the new vectors_meta.json (§3.1). The listing row and delta_meta must agree on every shared field; a disagreement rejects the delta.

• upserts holds the new row for every local_id whose term_upper,

definition or lang differs from the base, and for every appended local_id (> base_entries_count). Rows are sanitised exactly like a pack's (§4). base_term_upper is the base row's headword: a reader compares it with what it holds under that local_id and refuses the delta on a mismatch — a cheap proof that it is patching the dictionary the delta was made from.

• deletes lists the base local_ids that are gone. local_id stays dense

(§2.2), so deletes are always the tail entries_count+1 … base_entries_count; a delta that would leave a hole is invalid.

• vector_rows: for every upserted local_id, its d INT8 codes in the

layout of vectors.sq8 (§3.2). A builder writes them only when the base vectors and the new vectors share d, embedding_model, model_revision, quantization, vmin and vdiff exactly (the incremental encoder keeps the base's vmin/vdiff for this reason) — then the new vectors.sq8 is the old one with those rows written over or appended, and nothing else changes. Otherwise vectors is false. The row and the member agree: vectors: false ⇒ vector_rows is empty and delta_meta has no vectors_meta (also when the bundle carries no vectors for the dictionary — the builder then strips them); a reader rejects a member that contradicts its row.

Reader (one that knows deltas; others ignore the key and the members):

1. For an installed dictionary the listing offers in a newer pack_version, pick the delta with from_version = installed version, to_version = the listed version and base_entries_count = installed entry count. None ⇒ whole member.
2. If the reader keeps vectors for that dictionary and the bundle carries vectors for it: when the delta has vectors: true, its vectors_meta matches the installed set on every field named above and the installed vector count = base_entries_count, the vector rows are applied too; otherwise the reader applies the article delta and takes the dictionary's whole <pack_id>.vectors.zip member (validated against the new entries_count, §3) — the dictionary member itself is still not fetched. A reader without vectors for it ignores vector_rows. With vectors_space listed, the reader makes this choice (and sizes the download) before fetching: its installed set's fingerprint equal to vectors_space and its vector count = base_entries_count ⇒ patch; otherwise delta + whole vectors member. Without it, the reader may estimate either way and decides on apply.
3. Fetch the member by range, check sha256 over the stored bytes before inflating (signed listing: mandatory), inflate, open read-only, check delta_meta against the listing row (a listed vectors_space must equal the fingerprint of delta_meta.vectors_meta).
4. Apply in one transaction: every base_term_upper matches, delete the tail, write the upserts (the full-text index follows the rows), update the dictionary's version, entry count and metadata from pack_meta; then check COUNT(*) = entries_count and that local_id runs 1 … entries_count. Vectors (patched rows, or the whole new vector member of step 2) are written beside the live set and swapped in with the commit. Any failure ⇒ roll back (the installed dictionary is untouched) and take the whole member, as if no delta existed.

Old readers. A v3 reader that predates this section ignores the deltas key (readers ignore unknown keys) and every member not named <pack_id>.slovarypack / <pack_id>.vectors.zip, so it keeps installing whole dictionaries from the same bundle. The signature still covers the listing: the rows are authenticated for readers that do read them.

Reference builders: scripts/db/pack_tools/make_delta.py (one delta from two packs and their vectors) and make_bundle.py --deltas <dir> (adds the members and the deltas rows).

Reference builder: scripts/db/pack_tools/make_bundle.py.

7. Version history