Reference Implementation

NATS/JetStream Reference Implementation

How BSFG is implemented using NATS and JetStream as the substrate

Audience: Implementation engineers, operators. Use: Understand how BSFG's abstract architecture maps to NATS and JetStream. This describes one possible implementation; the BSFG architecture itself is substrate-neutral.

Important Note

This document describes the NATS/JetStream reference implementation of BSFG. The architecture and guarantees defined in Architecture Specification are independent of this implementation. Alternative implementations using PostgreSQL, Kafka, S3, or other substrates can fully satisfy the BSFG contract.

Overview

The NATS/JetStream reference implementation uses:

How BSFG Roles Map to JetStream

Ingress Store Buffer (ISB) → JetStream Stream

The ISB is implemented as a JetStream stream in the external zone:

Subject: facts.<category>[.<prefix>]
Storage: File or Memory
Replicas: >= 1 (configurable)
MaxAge: 7 days (default; configurable per category)
Retention Policy: limits (size) or interest (consumer acknowledgment)

Producers call AppendFact, which appends the fact to the ISB stream via stream.publish(). JetStream confirms durability once the message is replicated according to the configured replication factor (default RF=1 for local persistence, RF≥2 for HA clusters).

Ingress Forward Buffer (IFB) → JetStream Durable Consumer

The IFB is implemented as a durable consumer in the internal zone:

Consumer Name: ingress_consumer
Stream: (points to internal ISB copy)
Mode: pull (explicit acknowledgment)
AckPolicy: explicit
FilterSubject: facts.>
MaxAckPending: 0 (enforce strict ordering)

The durable consumer provides idempotent semantics: each fact is indexed by message ID in a deduplication window. Redelivery of the same message ID is automatically deduplicated; the consumer returns the same sequence number.

Cursor Tracker → Durable Consumer Checkpoint

The cursor position (highest_contiguous_committed_offset) is the durable consumer's acknowledged offset:

last_ack_offset = consumer.getNextMsgInfo().offset

When a consumer calls ConfirmReceipt(consumer_name, offset), the BSFG node invokes:

consumer.ackSync(lastMsgMetadata)

This advances the consumer's cursor (also called delivered_consumer_seq) to the confirmed offset. The cursor is persisted and survives node restarts.

Egress Store Buffer (ESB) → JetStream Stream (Mirror)

Mirror of the ISB for outbound flow. Implemented identically in the internal zone.

Egress Forward Buffer (EFB) → JetStream Durable Consumer (Mirror)

Mirror of the IFB for external pickup. Implemented identically in the external zone.

JetStream Domain Setup

Each BSFG zone runs a zone-local JetStream domain (isolated cluster or single node):

# JetStream configuration per zone

# External Zone (enterprise-bsfg)
---
zone: external
nats_server:
  jetstream:
    store_dir: /var/lib/nats/js
    max_file_slice: 30GB

streams:
  - name: facts_operational
    subjects:
      - facts.operational.>
    max_age: 10y
    storage: file
    replicas: 1  # or 3 for HA

  - name: facts_audit
    subjects:
      - facts.audit.>
    max_age: 7y
    storage: file

  - name: facts_telemetry
    subjects:
      - facts.telemetry.>
    max_age: 3y
    storage: file

# Object Store (for artifacts)
object_store:
  name: bsfg_artifacts
  bucket: bsfg
  replicas: 1

Each stream is scoped to its zone. No stream mirroring across zone boundaries. Cross-zone facts are transferred via the four RPC operations (AppendFact, FetchFacts, etc.), not by subscribing to remote streams.

NATS Accounts and Authorization

Zone-level authorization is enforced via NATS accounts:

enterprise-bsfg
  └─ enterprise_account (application access to facts.operational, facts.audit, etc.)
  └─ system_account (BSFG node system calls to $JS.>)

plant-a-bsfg
  └─ plant_a_account (application access to facts.operational, facts.audit, etc.)
  └─ system_account (BSFG node system calls to $JS.>)

Each zone's accounts are zone-scoped and do not share credentials with other zones.

Authorization Rules (Subject-Level ACLs)

# Enterprise NATS Account
accounts:
  enterprise_account:
    limits:
      max_connections: 100
    permissions:
      publish:
        allow: ['facts.operational.>', 'facts.audit.>', 'facts.documents.>']
      subscribe:
        allow: ['facts.operational.>', 'facts.audit.>', 'facts.documents.>']

  system_account:
    permissions:
      publish:
        allow: ['$JS.>']
      subscribe:
        allow: ['$JS.>']

Each zone allows:

Backend Adapters (Hexagonal Ports)

The StoreBuffer, ForwardBuffer, and CursorTracker ports are implemented by various backends. NATS/JetStream is one example:

ISB/ESB (Store Buffer) Adapters

IFB/EFB (Forward Buffer) Adapters

Cursor Tracker Adapters

The NATS/JetStream implementation uses JetStream consumer offsets as the cursor tracker; no separate cursor storage is required.

Transport: Connect RPC + mTLS

All cross-zone communication uses Connect RPC over HTTP/2 with TLS 1.3+:

Client (plant-a-bsfg)
  │ (mTLS handshake, verify CN = enterprise-bsfg)
  ├─ ClientHello
  │   ├─ Certificate: CN=plant-a-bsfg
  │   └─ Supported Protocols: h2
  │
  ├─ ServerHello (CN=enterprise-bsfg)
  │
  └─ RPC Call over HTTP/2
      POST /bsfg.AppendFact/AppendFact
      Authorization: Bearer <JWT or mTLS>
      Body: {fact, envelope, ...}

Each BSFG node has an mTLS certificate:

Subject: CN=<zone>-bsfg
Issuer: Enterprise PKI CA
Key Usage: TLS Server Auth, TLS Client Auth
SANs: [<node-hostname>, <node-ip>]
Validity: 1 year (recommend renewal 30 days before expiry)

JetStream Immutability and Atomicity

Append-Only Semantics

JetStream streams are append-only:

BSFG enforces immutability at the data model level: facts are never updated, only new facts are appended (e.g., corrections are new facts with a correction_predicate).

Atomic Idempotent Insert (Durable Consumer Deduplication)

JetStream durable consumers enforce idempotent append via message deduplication:

// Producer publishes with message ID header
await stream.publish(subject, payload, {
  msgID: stable_message_id,  // e.g., hash or UUID
  timeout: 10000
});

// If the same msgID arrives again, JetStream deduplicates within the deduplication window
// The producer receives the same publish response (same sequence number)

The deduplication window is configured per stream (default is the stream's MaxAge). Within the window:

Data Format: Envelope and Fact

Envelope (Transport Metadata)

BSFG-level metadata required for routing and idempotency:

{
  "message_id": "msg_abc123",
  "from_zone": "enterprise-bsfg",
  "to_zone": "plant-a-bsfg",
  "object_schema": "order_created_v1",
  "timestamp": "2026-03-06T14:30:00Z"
}

Fact (Business Payload)

Domain-specific data:

{
  "subject": "order:ENT-12345",
  "predicate": "order_created",
  "object_json": {
    "order_id": "ENT-12345",
    "customer": "CUST-999",
    "ship_to": "PlantA",
    "total_amount": 15000.00,
    "created_at": "2026-03-06T14:30:00Z"
  }
}

Full Message Published to JetStream

{
  "envelope": { ... },
  "fact": { ... }
}

Canonical JSON encoding ensures deterministic hashing for deduplication.

Naming Conventions

For detailed NATS/JetStream naming rules, see NATS Naming Profile.

Quick summary:

Operational Notes

High Availability (HA)

For production:

Patching and Updates

Coordinate patches with the operational runbook: Operations Runbook.

Monitoring and Metrics

Key metrics to monitor:

See Observability & Operations for the full metrics guide.

Security Considerations

mTLS Certificate Management

NATS Account Credentials

JetStream Data Encryption

See Security Model for the full security model.

Troubleshooting

Consumer Lag Growing

Check:

nats consumer info <stream> <consumer>

Look for Ack Floor vs. Last Sequence — if gap is growing, consumer is not keeping up.

Fix:

Stream Retention Exceeded

Check:

nats stream info <stream>

Look at Bytes and MaxBytes — if Bytes > MaxBytes, eviction is happening.

Fix:

mTLS Handshake Failure

Check:

nats server check

Look for certificate expiry warnings or CN mismatches.

Fix:

Cross-Links

← Back: Observability & Operations Guide Next: NATS Naming Profile →