GenosDB: Introducing Oplog-Driven Intelligent Delta Sync and Full-State Fallback

Full-state replication is the simplest approach to P2P sync — and the most wasteful. Every reconnection sends the entire database, regardless of what changed. GenosDB's oplog-driven delta sync replaces this with a hybrid protocol: an operation log tracks every mutation, peers exchange only what they missed, and a full-state fallback handles extended disconnections automatically.

This release is the culmination of intensive engineering efforts to build a professional-grade synchronization layer suitable for demanding, real-time distributed applications.

Core Architecture

The centerpiece of this release is a new dual-mode synchronization engine designed to balance performance with absolute reliability.

1. Delta Synchronization via Sliding-Window Oplog

The primary synchronization path is now driven by a lightweight, in-memory Operation Log.

• Operation Log (Oplog): A fixed-size, sliding-window Oplog is now maintained by each peer. This log stores a history of the most recent mutation operations (upsert, remove, link), represented as minimal, lightweight entries ({type, id, timestamp}).
• Delta Exchange Protocol: Upon connection, peers now initiate a syncRequest handshake, exchanging their latest known HybridClock timestamp. The responding peer queries its Oplog to compute the minimal set of operations (the "delta") required to bring the requesting peer up to date.
• Payload Compression: To further optimize network throughput, the computed delta payload (an array of "hydrated" operation objects) is serialized using MessagePack and subsequently compressed via pako (deflate). This results in a significant reduction in payload size, often exceeding 80%, minimizing latency on all connections.

2. Robust Full-State Sync Fallback Mechanism

To handle cases of severe desynchronization, such as a peer rejoining after an extended offline period, we have implemented a robust fallback mechanism.

• Consistency Boundary Detection: The syncRequest handler intelligently detects when a requesting peer's last known timestamp predates the oldest entry in the responder's sliding-window Oplog.
• Automatic State Transition: When this boundary condition is met, the system transparently aborts the delta-sync process and transitions to a full-state synchronization. It sends the entire current graph object, ensuring guaranteed eventual consistency.
• State Reconciliation: The receiving peer's syncReceive handler is designed to atomically apply the full state, which includes replacing its local graph and clearing its now-irrelevant Oplog to prevent historical conflicts.

Delta-Aware Security

The new synchronization protocol required a significant enhancement to our SoftwareSecurityManager (SSM) to maintain our zero-trust security model.

• Deep Packet Inspection for Deltas: The verifyIncomingOperations method in the SSM has been refactored to be "container-aware". It now identifies deltaSync payloads, decompresses them, and performs granular permission verification (using the rbac.can() function) on each individual operation within the delta.
• Atomic Validation: The SSM reconstructs and re-compresses a new delta payload containing only the subset of operations for which the original sender had explicit permission. This prevents privilege escalation or unauthorized data mutation via sync packets.
• Dependency Injection: To maintain modularity and keep the RBAC module lightweight, the SSM now receives compression/serialization tools (pako, msgpack) via dependency injection from the main GDB instance, cleanly separating concerns between the security and transport layers.

Bug Fixes

This architectural overhaul also allowed us to resolve several critical bugs:

• Resolved Repetitive Sync Loop: Corrected a race condition where a peer could re-request deltas it had just processed by implementing more rigorous management of the globalTimestamp after applying a change batch.
• Fixed Cross-Session State Corruption: The root cause of a bug where a rejoining peer could overwrite a more recent state has been addressed. The system is now fully resilient to this scenario through consistent conflict resolution on all write paths.

Impact

GenosDB v0.4.0 is a high-performance distributed database for browser environments. This new engine lays the foundation for future optimizations and scaling capabilities. Our focus now shifts to intensive end-to-end testing, API stabilization, and comprehensive documentation as we move towards a v1.0.0 release.

We invite you to explore the new architecture and provide feedback.

This article is part of the official documentation of GenosDB (GDB).
GenosDB is a distributed, modular, peer-to-peer graph database built with a Zero-Trust Security Model, created by Esteban Fuster Pozzi (estebanrfp).

📄 Whitepaper | overview of GenosDB design and architecture

🛠 Roadmap | planned features and future updates

💡 Examples | code snippets and usage demos

📖 Documentation | full reference guide

🔍 API Reference | detailed API methods

📚 Wiki | additional notes and guides

💬 GitHub Discussions | community questions and feedback

🗂 Repository | Minified production-ready files

📦 Install via npm | quick setup instructions

🌐 Website | GitHub | LinkedIn