MDC Platform

Features at a glance

Installation

git clone https://github.com/saji1970/ModelDB.git
cd ModelDB/mdc

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Two ways to run it:

python -m mdc serve takes --host (default 127.0.0.1), --port (default 8000) and --database (default data/mdc.duckdb). Keep it on the default host - see Security. The full command list is under Command line.

python -m mdc db init seeds the full spec-minimum synthetic payments dataset (10,000 merchants / 50,000 customers / 500,000 transactions, deterministic, seed 42) if you want the larger analytics dataset rather than the small dev seed.

Core concepts

MDC is three things sharing one storage layer, kept deliberately separate because they solve different problems:

• The polymorphic object store - classifies any uploaded bytes (a model checkpoint, an image, a document, a log file) by real content signature, not filename, and routes it through a tiering decision. This is what § AI models and § Other objects cover.
• The database layer - unlimited named databases, each with its own schema-validated tables, created and queried entirely through chat or the CLI. This is what Multiple databases, Tables & queries, Import, export & design and Deleting things cover.
• State and events - for data that changes over time, such as what an AI agent knows, decided or is doing. Instead of overwriting the current value, MDC keeps a starting state plus every change, each with its time, so the state at any past moment can be rebuilt. This is what State & events and Agent state streams cover.

All three are driven by the same principle: an LLM proposes an interpretation, but a deterministic validator - never the LLM itself - decides what operation actually reaches storage. No natural-language command generates raw SQL.

State & events: storing an AI model's data with its history

A normal table keeps only the latest value: update a field and the old one is gone. For an AI agent that is the wrong shape. What it believed yesterday, why it changed its goal, and what its state was when it made a particular decision are all questions you may need to answer later. MDC answers them with event sourcing: the state is never edited, it is derived from a base plus an ordered log of changes.

base (event 0)     mood=calm  goal=idle  level=1          10:00
event 1            mood=busy  goal="write report"           11:00
event 2            level=2                                  11:40

state as of 10:30  ->  mood=calm  goal=idle  level=1
state as of 12:00  ->  mood=busy  goal="write report"  level=2

Nothing here relies on a model to reconstruct anything. A language model, or any other part of an agent, decides what to record; MDC stores it and rebuilds it with deterministic code. How to try it: Agent state streams.

Nobody on your team has to learn SQL

Every example on this page - filtering by a threshold, scoping to a specific database, searching across tables - is plain sentences, not query syntax. That's not a simplified subset for beginners standing next to "the real interface"; it is the interface. Support staff, analysts, and anyone else who needs an answer from the data can just ask for it, with no SELECT/WHERE/JOIN training required, and nothing to get subtly wrong the way a hand-written query can.

Versus a traditional database: size, speed, ease, cost

MDC is not a faster or smaller database than PostgreSQL or SQLite, and it does not replace one for heavy workloads. It is much closer than it was: lookups by id take well under a millisecond and bulk loads are fast. What it offers is a different trade: a database you can run and use without SQL, drivers or a managed database service, at the price of some space and speed. This page gives the measurements, the access difference, and a cost example with its assumptions, so you can decide whether that trade fits. Everything here is for MDC Platform, with mdc-lite measured separately below.

Storage size

The same merchant-style records, each store loaded in one transaction and checkpointed:

MDC is larger than both: about 1.7 times PostgreSQL and 3 times SQLite at 10,000 records, and about 1.1 and 1.9 times at 100,000, where the fixed overhead of the file matters less. It keeps each record's fields as JSON text next to typed columns for the id, version, times and a SHA-256 checksum, and holds a primary-key index so a record can be found by id. A plain DuckDB table with real typed columns held the 100,000 rows in 3.0 MB, a quarter of MDC's size, which shows where the remaining space goes; storing each table's fields as typed columns is the way to close that gap and has not been done.

This is smaller than it was. An earlier version stored every record as its own JSON block and took 3.3 MB for the 10,000 records; the records now live in a table, and the ARCHIVE tier no longer stores letters:

Performance

Reads are hundreds to thousands of times faster than before. The first version read every record in a table to answer any question: a lookup took 954 ms at 10,000 records and grew in proportion. Records now sit in a table: a lookup by id goes through the primary key (0.33 ms at 10,000 records and 1.0 ms at 100,000), a table is read with one query, and a filter on =, <, >, between or IN is first narrowed inside the database. MDC still re-checks every filter itself, so the meaning of each operator did not change.

Still slower than the ordinary databases, and honestly so: inserting one record at a time takes about 14 times PostgreSQL's time (0.44 ms, which is also what a one-row insert costs in DuckDB, the engine underneath); loading in one transaction takes 3 times; finding by a field value or filtering takes 4 to 7 times, and still grows with the table because there is no index on field values. A bulk load (create_many, used by imports) of 10,000 records takes 0.47 s, about 9 times quicker than inserting them one at a time, and quicker than raw DuckDB through its Python client (3.4 s). The rest: event streams append in about 0.9 ms; rebuilding a stream of 1,000 events with no snapshots took about 100 ms (streams take a snapshot every 50 events by default, which this run turned off); an ARCHIVE put of 256 KB took 3 ms and a get under 1 ms (they were 80 ms and 48 ms when the letters were stored).

mdc-lite, 10,000 entries: a put averaged 110 µs and a get 17 µs, against about 200 µs and 4 µs for a SQLite key-value table. mdc-lite's put being quicker than SQLite's one-at-a-time commit is not a like-for-like win: I did not check whether it forces each file to disk before returning, which SQLite does.

Ease of access

What you give up for that ease: no joins, no multi-statement transactions, no indexes, no SQL tooling or BI connectors, no managed backups, patching or failover, and one writing process per database file. If you need any of those, use a database.

Cost: not running a managed database

A managed database bills for an instance by the hour whether or not anyone is using it, plus storage, backups beyond a free allowance, and data transfer. MDC Platform is a small program you can run on a usage-billed container host. An illustration for a small always-on app with about 5 GB of data, prices as of 2026-10-20:

On those assumptions the saving against the smallest RDS instance is about $9 a month (roughly 60%) and against a Multi-AZ or larger instance more. Google Cloud SQL and the other providers bill the same way (instance hours, storage, backups, transfer); use each provider's calculator for your region and size. AWS and Azure figures here come from third-party price summaries, because AWS's own page did not show them to me, so confirm them in the AWS Pricing Calculator and the Azure page; Railway's are from its own documentation.

Storing & retrieving AI models

Upload a .safetensors file and MDC parses its real header, splits each tensor into row-aligned blocks, and stores every tensor as its own independently addressable, independently tiered object - not one opaque blob.

curl -X POST "http://127.0.0.1:8000/objects?filename=model.safetensors" \
  --data-binary @model.safetensors
# {"object_id":"AIM-58F0BB747C","type":"AI_MODEL","tensor_count":2,"total_parameters":16}

curl http://127.0.0.1:8000/models/AIM-58F0BB747C
# manifest: tensor_count, total_parameters, dtype, checksum, ...

curl http://127.0.0.1:8000/models/AIM-58F0BB747C/tensors/linear.weight
# raw bytes of just that one tensor - the rest of the model is never touched

The same thing through chat:

move on a model cascades to every one of its tensor blocks - each tensor gets its own independent tier decision at upload time, so moving just the model's manifest would silently leave the actual weights wherever they were.

Storing other object types

The same upload endpoint handles images (real dimension parsing from file headers), documents (text extraction + chunking), logs, tensors, and tabular files - classified from content, never from a file extension.

Every object - model, image, document, or table row - carries a SHA-256 checksum verified on every read. A tampered or corrupted read never silently returns bad bytes; it fails with a clear integrity error instead.

Storage tiers

Tier is a real consequence, not a label - only HOT maps to genuinely different physical storage (an in-memory backend) today; ARCHIVE is the one exception that goes further still, actually encoding bytes to simulated DNA base sequences (2 bits per base, A/C/G/T - a real, working prototype encoder, not physical synthesis). The encrypted sequences are saved in the database file (packed four bases to the byte; the letters A, C, G and T are produced on demand), so archived objects survive a restart, cost no more space than the data, and are removed with their database.

The core concept: 2 bits per base

Every byte in the ARCHIVE tier is mapped directly to 4 DNA bases - the same information-density idea real DNA-storage research is built on:

00 → A      01 → C      10 → G      11 → T

A real, working error-correction scheme (N-way repetition with majority voting across independently-corrupted copies) and a seeded corruption simulator sit on top of the encoding - modeling substitution, insertion, deletion, and dropout errors the way real DNA storage research does. This is a simulation: a string of ACGT characters in a database, not a molecule - no physical synthesis or sequencing anywhere in this codebase.

Every payload is AES-256-GCM encrypted before it's DNA-encoded - the ACGT sequence is ciphertext-as-base-letters, not plaintext-as-base-letters, so it's unreadable without the key this backend holds. The mapping above is public (you're reading it), so encoding plaintext directly would only be obfuscation; encryption is what actually makes the archive tier unreadable outside MDC's own API/CLI. mdc-lite stores every entry the same way.

Read the full whitepaper → for the error-correction design, the corruption model, and an honest, specific answer to "does this use quantum encryption?" (short version: no - and here's what quantum computing actually does and doesn't threaten about long-retention archival data).

Phase 2 (planned, not yet built): extending DNA-tier features further on the platform side - stronger ECC, a Windows CLI bridge that can open an mdc-lite store directly when a phone/wearable is connected, and research into applying the same tier beyond archival storage.

Multiple databases

No hardcoded limit on how many databases exist - each is a fully isolated DuckDB file with its own schema registry, created lazily the moment you ask for it, entirely from chat:

Tables & queries

Read another database's table without switching to it, using an explicit qualifier:

show data in products in database inventory
describe table products in database inventory

Table creation only ever produces a validated, typed SchemaRegistry collection - never raw SQL DDL generated from chat text, the same injection-avoidance principle as everything else in MDC.

Universal search

find searches every database's tables and documents at once, with deterministic clause extraction for amount constraints and explicit scoping - no need to know which database or table something lives in:

Import, export & schema design

Everything below works from the chat window - no SQL, and nothing is written unless the whole request is valid.

Import: the paperclip (or drop a file on the chat)

A file that defines a database (design document, ER diagram, SQL script, export file) creates it under the name the file gives it; add ?database= to the REST call to choose another. An import never overwrites: an existing table, or a record_id that already exists, stops the whole import with a clear message. The role of the caller decides what is allowed - creating a database needs create_database, new tables create_table, rows write. Files over 25 MB are refused.

The design-document format - this is also what a design document export produces:

# Database: shop

## Table: customers

| Field   | Type     | Required |
|---------|----------|----------|
| name    | string   | yes      |
| credit  | decimal  | no       |
| joined  | datetime | no       |

Types are string, decimal, integer, boolean, datetime; SQL-style names (varchar(50), int, date, ...) are read as the nearest of those and flagged.

Export: ask in the chat, get a download button

CSV and Excel cells that start with =, +, - or @ are written so a spreadsheet shows them as text rather than running them as formulas; importing the CSV back restores the original value.

Build a schema from a prompt or an ER diagram

Describe it, or paste an erDiagram into the chat. MDC shows what it understood and creates nothing until you say yes:

This is a deterministic grammar, not a language model: a field without a type is a string unless its name suggests otherwise (total, price -> decimal; age, count -> integer; is_..., active -> boolean; ..._at, date, joined -> datetime), and every guess is listed. Free-form requests such as "I need a database for a school" are not interpreted - that needs a language model, which this build does not include.

Agent state streams (event sourcing)

A stream keeps an agent's state as a base snapshot plus an append-only log of timestamped changes. Nothing is edited in place, so MDC can rebuild the state exactly as it was at any moment - what the agent knew at 10:30 yesterday, or right after its third change.

A time is an ISO timestamp (2026-10-19T10:30:00Z, no zone means UTC), a date (2026-10-19 means the end of that day, UTC), or 15 minutes ago. Asking for a time before the stream was created is an error, not an empty state.

What is sealed, and what that does and doesn't protect

Every record - the base state, each event, and the timestamps inside them - is encrypted with AES-256-GCM under a key derived per stream from the server's master key (MDC_DNA_KEY, or the key file the DNA tier already uses - how to set it up and rotate it). Someone who gets at the database file sees ciphertext: not the fields, not the values, not what changed, not the event times. Each record is also bound to its stream and position, so a copied, swapped, altered or deleted event is detected on read and reported, never silently replayed.

Where the AI model fits

The agent decides what its state is and when it changed, and records that by posting a new state or a change. How a change is encoded is the job of a codec: the built-in one writes set/unset operations on paths. A library user can supply their own codec (for example one that records decisions or compacted summaries) with EventStore(..., codecs={...}). Replay is always the codec's deterministic apply - a state is rebuilt by running the log, never by asking a model to remember - and since a codec is Python code, the REST API and chat only use the built-in one.

Also: one writing process per database (a second writer is caught and refused, not merged); recorded times never run backwards (back-filling with at is allowed, rewriting the past is not); one record may be at most 1 MB; master-key rotation is not implemented.

Deleting things

Every deletion here is permanent - there is no recycle bin - so nothing is removed on the first request. In chat, MDC says exactly what would be lost and waits. Over REST, the dangerous ones must repeat the name as ?confirm=.

The REST reply to an unconfirmed request is a 422 that states the counts, so a script or a person sees what the call would destroy before repeating it.

• Anything other than the confirmation cancels - "no", a different request, even an empty message - so a stray "yes" later deletes nothing.
• Protected: the default database, and its built-in merchants table, cannot be dropped.
• Roles: dropping needs the new drop permission, held by db_admin and admin only; an editor can still delete rows and objects. Chat checks it when you ask and again when you confirm.
• If someone else is working in a database that gets dropped, their next message says so and moves them back to default.

Storage Explorer

python -m mdc serve, then open http://127.0.0.1:8000: a Windows-Explorer-style app over everything MDC stores.

There is no login screen. Run on your own machine, the page carries a per-process admin token so it works with no setup; the deployed container asks for the API token instead (Deployment).

Security & access

Encryption keys: setup & rotation

One 32-byte master key seals the data MDC stores. Event streams use a key derived from it for each stream (HKDF-SHA256), so one stream's key never opens another; every record gets a fresh random nonce and is bound to its stream and position, so a copied, swapped, edited or deleted record is detected. The ARCHIVE tier uses the master key directly, before the A/C/G/T encoding. Both are AES-256-GCM.

Set it up, step by step

1. Generate a key.

   python -m mdc key generate
   # 64 hex characters, printed once and not saved anywhere

2. Choose where it lives (first match wins):

   Option	How	Use it when
   MDC_DNA_KEY	Environment variable holding the 64 hex characters	A real deployment. The key then comes from your secrets store, never from the data disk
   MDC_DNA_KEY_FILE	Path to a file holding the key (mode 0600). Created with a new key if it does not exist	The key is a mounted secret or sits on another disk
   Default file	~/.mdc/dna.key, made on first use. In the container HOME is /data, so /data/.mdc/dna.key	Local development. The key is on the same volume as the data

3. Hand it to the server.

   # local
   export MDC_DNA_KEY=<the 64 hex characters>
   python -m mdc serve
   
   # Docker
   docker run -p 8000:8000 -v modeldb:/data \
     -e MDC_API_TOKENS=<token> -e MDC_DNA_KEY=<key> modeldb
   
   # key in a file outside the data volume
   docker run -p 8000:8000 -v modeldb:/data -v /secrets/mdc.key:/run/secrets/mdc.key:ro \
     -e MDC_DNA_KEY_FILE=/run/secrets/mdc.key modeldb

   On Railway add MDC_DNA_KEY under the service's Variables (a sealed variable), not in the repository.
4. Check which key is in use. The server prints this at startup; the command prints the same line and never the key:

   python -m mdc key show
   # Sealing key: from MDC_DNA_KEY (fingerprint 3f9a1c07).

   The fingerprint is 8 characters derived one-way from the key. Compare it with the one you recorded to be sure the right key is configured.
5. Use MDC as normal. Streams are sealed from the first record, and the first stream written also stores a sealed key check, so later a wrong key is reported as a wrong key, not as damage.
6. Back the key up, separately from the data. Losing it makes every sealed record unreadable, and there is no recovery. Record the fingerprint with the backup.

Rotate the key

Rotation re-seals every stream and every archived object in every database under a new key. Plaintext is never written: each record is opened and re-sealed in memory, and a record that does not authenticate stops the rotation before anything is changed.

1. Stop the server and back up the data directory (data/, or /data in the container).
2. Rotate, giving the new key one of three ways, with the current key still configured:

   # make a new key
   python -m mdc key rotate --database data/mdc.duckdb --generate
   
   # or use a key you made
   python -m mdc key rotate --database data/mdc.duckdb --new-key-env MDC_DNA_KEY_NEW
   python -m mdc key rotate --database data/mdc.duckdb --new-key-file new.key

   With --generate and a key file, the old key is kept as <file>.<fingerprint>.old, the new key is staged as <file>.new before anything is re-sealed, and then replaces the file. With MDC_DNA_KEY the new key is printed once, before anything changes, so a crash cannot leave data sealed under a key you never saw.
3. Switch the server to the new key (MDC_DNA_KEY or MDC_DNA_KEY_FILE) and start it. The old key now fails with a clear message.
4. If it was interrupted (power cut, killed process), run the same command again: records already under the new key are recognized and skipped. Delete the old key and the backup only when you are sure.

What this does and does not cover

Monitoring: live statistics & Kafka

MDC can publish what it is doing on a separate port, and stream the same statistics to Kafka, so a dashboard or monitoring system can watch it. Both are off until you ask. The port is read-only, and the numbers are counts and sizes only: never data, keys or tokens.

Turn it on

# a metrics port, kept to this machine
python -m mdc serve --metrics-port 9464

# reachable by a monitoring server, and protected
MDC_METRICS_TOKEN=choose-a-long-random-value \
  python -m mdc serve --metrics-port 9464 --metrics-host 0.0.0.0

# also stream to Kafka every 10 seconds
pip install confluent-kafka        # or kafka-python
python -m mdc serve --kafka-bootstrap broker1:9092 --kafka-topic mdc.stats

The container reads the same settings from environment variables, so on Docker or Railway you set variables instead of flags:

On startup the server prints the address it is publishing on. A bad setting (a taken port, a missing Kafka library) stops mdc serve with a message; in the container it is logged and the database still starts.

What the port serves

What is in a snapshot

mdc_process_resident_memory_bytes 97271808
mdc_process_cpu_seconds_total{mode="user"} 0.263
mdc_database_file_bytes{database="default"} 12288
mdc_table_rows{database="default",table="merchants"} 3
mdc_database_objects_by_tier{database="default",tier="ARCHIVE"} 1
mdc_http_requests_total{status_class="2xx"} 1

For CPU in Prometheus, graph rate(mdc_process_cpu_seconds_total[1m]): it is exact however often you scrape. mdc_process_cpu_percent is measured between snapshots, and snapshots are shared for 2 seconds, so very frequent readers see the same value.

Kafka

MDC publishes one JSON message per snapshot (the same JSON as /stats), keyed by the host name so a topic shared by several servers keeps each one's messages in order. It is a producer only: you run the broker, create the topic (or let your broker auto-create it), and install a client library, confluent-kafka (preferred) or kafka-python. MDC does not bundle one. A failed publish is counted (mdc_kafka_errors_total) and retried at the next interval; a broker outage never stops the database. Authentication and TLS to the broker are not configurable yet: it connects with the library's defaults.

Deployment (Docker & Railway)

docker build -t modeldb .
docker run -p 8000:8000 -v modeldb:/data \
  -e MDC_API_TOKENS=choose-a-long-random-token modeldb

The image runs the same app as mdc serve, without its interactive first-run prompt, on PORT and on 8000 (every address, IPv6 where available), with GET /healthz for health checks. railway.json builds the Dockerfile, checks /healthz, and requires a volume mounted at /data.

The deployed Storage Explorer asks you to paste the token (kept for the browser tab); a wrong one is refused and asked for again. /admin redirects to /.

REST API reference

Every route below requires Authorization: Bearer <token> - see Connect your own NLU or UI for how to issue one.

Command line

Where things live: data/mdc.duckdb is the default database; every other one is data/databases/<name>.duckdb with its table structure in <name>.schema.json. Credentials are in ~/.mdc/.

Chat command reference

Connect your own NLU or UI

MDC's own chat layer doesn't have to be the only front door. Two integration patterns, depending on whether your system has already done its own language understanding by the time it talks to MDC:

Authenticate first — every route but the built-in UI needs a token

CORS is deliberately wide open (allow_origins=["*"]) so a third-party integration can call this API from a different origin without a browser rejecting the response first - that's the point of everything on this page. Open CORS was never itself a security boundary against a non-browser client though, so every route except the built-in browser UI's own HTML page requires a bearer token:

$ mdc token issue rasa-integration --role editor
mdc_kP3x...   (shown once - store it now)

Send it as Authorization: Bearer <token> on every request. Revoke it later with mdc token revoke rasa-integration, or set MDC_API_TOKENS (comma-separated) to accept fixed tokens without the on-disk store, for CI/deployment.

--role scopes what that specific token can do - omit it and a token defaults to admin (full access, matching the original all-tokens-are-equal behavior), or pick something narrower for an integration that shouldn't have it:

A token with an insufficient role still authenticates - a valid-but-underprivileged token gets 403, not the 401 reserved for a missing or invalid token.

User accounts and roles (CLI)

Separately from API tokens, MDC keeps user accounts. The first time you run mdc or mdc serve with no user configured yet, it interactively asks for an admin username and password and creates that first account as admin - the same four roles as tokens, since both share one vocabulary. Manage accounts afterward with:

$ mdc user create alice --role editor
Password: ********
Repeat for confirmation: ********
Created user alice with role editor.

$ mdc user list
  admin  (admin)
  alice  (editor)

$ mdc user set-role alice db_admin
$ mdc user delete alice

Passwords are never stored in a recoverable form - hashlib.scrypt derives a key from the password and a random per-user salt, and only that salt and derived key are persisted (mode 0600), the same reasoning already used for API tokens.

Pattern 1 — pass-through (simplest, works today, zero new code)

Forward raw user text to POST /chat and let MDC's own deterministic parser do the work - the same pipeline the browser chat panel and CLI already use. This is the right choice for most integrations: your system doesn't need to understand databases, models, or storage tiers at all, just detect "this is a data question" and relay the text.

# A RASA custom action (actions.py) - the entire integration
import requests
from rasa_sdk import Action

class ActionQueryMDC(Action):
    def name(self):
        return "action_query_mdc"

    def run(self, dispatcher, tracker, domain):
        reply = requests.post(
            "http://localhost:8000/chat",
            json={"message": tracker.latest_message.get("text"), "session_id": tracker.sender_id},
            headers={"Authorization": "Bearer mdc_kP3x..."},
        ).json()
        dispatcher.utter_message(text=reply["message"])
        return []

Note session_id: tracker.sender_id - reusing RASA's own conversation id keeps MDC's per-session state (which database you switched to, pronoun resolution) aligned with RASA's own multi-turn state, so "switch to database inventory" followed later by "show data in products" behaves correctly across turns.

Pattern 2 — structured (for a system that already extracted intent)

If your NLU has already parsed "add Widget at $9.99 to inventory" into its own slots, reconstructing MDC's sentence syntax from those slots is unnecessary indirection. Call the structured REST endpoints directly with the extracted values instead:

# RASA custom action calling the structured endpoint directly,
# using slots RASA's own NLU already extracted
requests.post(
    f"http://localhost:8000/databases/inventory/tables/products/rows",
    json={"values": {"name": tracker.get_slot("item_name"), "price": tracker.get_slot("price")}},
    headers={"Authorization": "Bearer mdc_kP3x..."},
)

Every structured endpoint runs through the exact same validation as the chat/CLI path it mirrors (schema-registry-only table creation, typed field coercion on insert) - there's no separate, less-checked code path for programmatic callers.

A custom "ChatGPT-style" UI

A minimal chat interface is one fetch() call - the same POST /chat endpoint the Storage Explorer's own chat panel uses:

async function ask(message) {
  const res = await fetch("http://localhost:8000/chat", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer " + MDC_TOKEN,
    },
    body: JSON.stringify({ message, session_id: crypto.randomUUID() }),
  });
  const { message: reply, data } = await res.json();
  return { reply, data };  // data is a table's rows / a search result list, when there is one
}