Modern synchronization is deceptively difficult.
At small scale, syncing data across devices seems straightforward:
upload local changes, download remote updates, and keep everything "in sync."
But the moment an application becomes:
* offline-first,
* collaborative,
* privacy-focused,
* encrypted,
* multi-device,
* and backend-light,
synchronization becomes one of the hardest engineering problems in the entire system.
At Sortify, we faced this challenge directly.
We wanted users to:
* fully own their data,
* store encrypted workspaces in their own cloud providers,
* collaborate across devices and users,
* work offline,
* and still experience reliable synchronization without destructive data loss.
We intentionally chose not to build a traditional centralized sync backend.
Instead, we designed a synchronization engine centered around:
* encrypted snapshot storage,
* client-side merge orchestration,
* optimistic concurrency control,
* deterministic conflict handling,
* and replayable local intent journaling.
This article explores the architecture, the failures we encountered, the distributed systems lessons we learned, and the synchronization model we ultimately implemented.
---
# The Problem With Traditional Sync Models
Most synchronization systems follow one of two architectures.
## Centralized Server Synchronization
The most common approach is:
* all clients communicate with a central backend,
* the backend stores authoritative state,
* the backend resolves conflicts,
* clients become relatively thin.
This model works well for:
* real-time collaborative editors,
* enterprise SaaS platforms,
* heavily coordinated multi-user systems.
But it comes with tradeoffs:
* infrastructure cost,
* operational complexity,
* trust requirements,
* and major privacy implications.
The server often becomes:
* the owner of user data,
* the merge authority,
* and the visibility layer for all collaborative content.
That did not align with Sortify's philosophy.
---
## Pure File Synchronization
At the opposite end is simple file synchronization:
* upload a database file,
* overwrite remote copy,
* download latest version later.
This is simple, but dangerously naive under concurrency.
If two users upload snapshots concurrently:
* later uploads overwrite earlier uploads,
* changes disappear,
* and synchronization becomes non-deterministic.
Initially, our architecture resembled this model more than we wanted to admit.
That became obvious once real-world collaborative edge cases appeared.
---
# Our Constraints
Before designing the synchronization system, we defined several non-negotiable principles.
## 1. User-Owned Data
Sortify users should remain in control of their own storage.
Instead of storing workspace databases on our servers, we integrate with providers such as:
* Google Drive
* Dropbox
* OneDrive
This means:
* users control retention,
* users control deletion,
* users control account access,
* and Sortify itself does not become the permanent owner of workspace content.
---
## 2. End-to-End Encryption
Workspace snapshots are encrypted before upload using **AES-256-GCM** with a 256-bit workspace-specific key and a 96-bit random IV generated per upload.
The encryption key is stored in platform-native secure storage: **iOS Keychain** on Apple devices, **Android Keystore** on Android devices. It never leaves the device and is never transmitted to any server, including MokingBird's own infrastructure.
Cloud providers store opaque encrypted blobs rather than readable workspace content.
This ensures:
* cloud providers cannot inspect workspace data,
* MokingBird cannot inspect workspace contents in the ordinary course of operations,
* synchronization remains privacy-preserving by architecture rather than by policy,
* and user trust boundaries remain clear.
---
## 3. Offline-First Behavior
Users must be able to:
* create rooms,
* move items,
* rename entities,
* update metadata,
* and continue working offline.
Synchronization therefore cannot assume permanent connectivity.
---
## 4. Multi-User Collaboration
Workspaces are collaborative.
Multiple users may:
* edit the same workspace,
* move items simultaneously,
* rename rooms concurrently,
* or sync from multiple devices.
This introduces distributed synchronization challenges even without a traditional backend.
---
# The First Major Failure
Our early synchronization engine used a snapshot-authoritative merge strategy.
The logic was conceptually simple:
* pull remote snapshot,
* compare against local database,
* apply remote state,
* delete rows missing remotely.
This appeared reasonable initially.
But it created a catastrophic edge case.
---
## The "Missing Means Delete" Problem
Suppose a user:
* creates several items offline,
* those items exist only locally,
* synchronization runs before upload succeeds,
* remote snapshot does not contain those items yet.
The pull engine interpreted this as:
> "these rows do not exist remotely, therefore they should be deleted locally."
As a result:
* items vanished,
* rooms disappeared,
* counters dropped to zero,
* and local state was erased despite never being intentionally deleted.
This revealed a critical distributed systems truth:
> Absence is not deletion.
Especially in eventually consistent systems.
---
# Understanding the Real Problem
The issue was not simply:
* timestamps,
* ordering,
* or retries.
The deeper issue was architectural.
We had combined:
* snapshot synchronization,
* partial journaling,
* and destructive reconciliation semantics.
That combination was unsafe under concurrency.
---
# Reframing the Architecture
We eventually shifted toward a fundamentally different model:
> client-side merge with optimistic concurrency guarded snapshot publishing.
This became the foundation of the new synchronization engine.
---
# The New Synchronization Model
The final architecture follows this sequence:
1. Read remote metadata/version token.
2. Pull latest snapshot if remote changed.
3. Merge locally using deterministic policies.
4. Replay unsynced local intent journal.
5. Export and encrypt merged snapshot.
6. Upload conditionally using optimistic concurrency tokens.
7. Retry on precondition failure.
This transformed synchronization from:
> blind overwrite synchronization
into:
> distributed optimistic synchronization.
---
# Client-Side Merge With Snapshot Storage
One of the most unusual characteristics of Sortify's synchronization engine is where merge logic happens.
Traditional systems:
* merge on the server.
Sortify:
* merges entirely on the client.
Cloud providers are treated as:
* encrypted blob stores,
* versioned snapshot containers,
* not synchronization authorities.
This architecture preserves privacy while still allowing convergence across clients.
---
# Optimistic Concurrency Control (OCC)
A major architectural upgrade was introducing optimistic concurrency control.
Without OCC:
* two users can upload snapshots simultaneously,
* later uploads overwrite earlier snapshots,
* remote history becomes unstable.
We solved this by using provider-issued version tokens.
Examples include:
* Google Drive revision metadata,
* Dropbox rev identifiers,
* OneDrive etags.
Before uploading a snapshot, Sortify verifies:
> "Is the remote file still the same version I merged against?"
If not:
* upload is rejected,
* the client pulls latest state again,
* merges again,
* and retries safely.
This small change fundamentally transformed synchronization reliability.
---
# Replayable Local Intent Journal
Another critical architectural addition was the local change journal.
Every mutation now:
* updates local tables,
* and appends a journal entry in the same transaction.
Examples include:
* item updates,
* room renames,
* photo changes,
* membership changes,
* tombstone deletes.
The journal represents:
> local user intent not yet safely published remotely.
During synchronization:
* remote snapshot changes are applied first,
* then unsynced journal entries are replayed.
This guarantees that local intent is preserved during merge operations.
---
# Tombstone Convergence
Traditional delete behavior creates major synchronization hazards.
Originally, some user-facing deletes physically removed rows from the database.
That approach becomes incompatible with eventual consistency.
Instead, Sortify moved toward tombstone-based convergence.
Rather than deleting rows:
```text
is_active = 0
deleted_at = timestamp
```
Deletes become:
* explicit state transitions,
* synchronizable events,
* and conflict-resolvable operations.
Most importantly:
> remote absence alone is never interpreted as delete intent.
This eliminated the entire class of "items vanished" failures.
---
# Deterministic Merge Policy
Distributed synchronization becomes dangerous when merge outcomes are non-deterministic.
If two clients resolve conflicts differently:
* synchronization never converges.
We therefore implemented deterministic merge policies:
* **Last-Write-Wins by `updated_at`:** the entity with the higher timestamp wins.
* **Lexical tie-breaker on equal timestamps:** when `updated_at` values are identical, the winning version is determined by lexical comparison of `updated_by` (user identifier) or device ID. This comparison is deterministic: every device running the merge on the same two records will produce the same winner.
* **Deletion semantics:** a tombstone (`is_active = 0`) with an `updated_at` timestamp participates in LWW the same way any other update does. An explicit delete can be overridden by a later update, and vice versa, by timestamp.
Even equal timestamps now produce stable, identical results regardless of which device runs the merge.
This is subtle, but essential: non-deterministic conflict resolution causes perpetual divergence because two clients can both believe they published the authoritative state while holding different data.
---
# Bounded Retry Orchestration
Optimistic concurrency naturally introduces retries.
Consider:
* User A syncs,
* User B syncs milliseconds later,
* both began from the same remote snapshot.
One upload succeeds.
The other receives a precondition failure.
Rather than overwriting:
* Sortify retries automatically,
* re-pulls latest state,
* merges again,
* republishes safely.
Retries are bounded to avoid:
* infinite loops,
* battery drain,
* synchronization storms.
---
# Implementation Phases
The rebuild of the synchronization engine was structured as eight discrete implementation phases, each targeting a specific failure class.
| Phase | Change | Failure Class Addressed |
|---|---|---|
| 1 | Non-destructive pull merge | Delete-on-missing data loss |
| 2 | `workspace_sync_state` persistence | Implicit sync position causing redundant work |
| 3 | OCC tokens on Google Drive, Dropbox, OneDrive | Blind overwrite race conditions |
| 4 | Bounded pull-merge-push retry loop | Unrecovered precondition failures |
| 5 | Complete `change_journal` coverage across all mutation paths | Local intent invisible to merge engine |
| 6 | Deterministic LWW tie-breaker | Non-deterministic conflict divergence |
| 7 | iCloud gated as unsupported for collaborative sync | False concurrency guarantees on iCloud Drive |
| 8 | Tombstone delete routing through `is_active = 0` | Hard-delete rows defeating eventual consistency |
All eight phases are shipped in the current release.
---
# Provider-Aware Conditional Uploads
Different providers expose concurrency semantics differently.
For example:
## Google Drive
### OAuth Scope Decision
Sortify originally requested the `drive.file` OAuth scope, which limits access to files the app itself created. This appears to be the minimum necessary scope and is appropriate for single-user workspaces.
Under `drive.file`, however, collaborative multi-user workspaces fail. When a workspace owner creates the shared workspace folder and invites a collaborator, the collaborator's app cannot access the folder — because the folder was created by the owner's Sortify instance, not the collaborator's. The `drive.file` scope does not permit a second user's app to read or write files it did not create, even if those files are in a folder explicitly shared with the collaborator at the Google Drive level.
Collaboration requires each workspace member's app to read and write the encrypted snapshot in the shared workspace folder regardless of which member created it. This requires the `https://www.googleapis.com/auth/drive` scope.
Sortify uses this scope exclusively for Sortify workspace folders and Sortify sync artifacts. It does not enumerate unrelated Drive content, access files outside the workspace folder hierarchy, or store any data in the user's Drive other than the encrypted workspace blob and associated metadata file.
### Concurrency Semantics
Google Drive synchronization leverages:
* revision-aware file metadata,
* `If-Match` conditional upload semantics (etag-based precondition),
* and persistent remote token tracking in `workspace_sync_state`.
An upload that specifies an etag that no longer matches the current remote file is rejected by Google Drive with a precondition failure. Sortify treats this as a signal to re-pull, re-merge, and retry.
---
## Dropbox
Dropbox exposes `rev` identifiers on every file. Conditional uploads are supported natively: an upload specifying `mode: update` and an expected `rev` value is accepted only if the current remote `rev` matches. If the file has changed since the `rev` was captured, the request is rejected.
This is functionally equivalent to compare-and-swap: upload succeeds only if the file is still at the version the client merged from.
---
## OneDrive
OneDrive exposes `eTag` and `cTag` fields via the Microsoft Graph API. Conditional uploads are supported using `If-Match: <etag>` request headers. An upload that specifies an etag that no longer matches the remote file's current etag is rejected with a precondition failure, triggering the retry loop.
---
## iCloud — Gated as Unsupported for Collaborative Sync
iCloud Drive does not expose the optimistic concurrency primitives that Sortify's sync architecture depends on. Google Drive, Dropbox, and OneDrive all provide explicit version tokens and conditional upload semantics (etag, rev, If-Match). iCloud Drive does not expose equivalent public API primitives for file-based conditional uploads, making it impossible to implement the compare-and-swap guarantee that prevents concurrent overwrites.
CloudKit — Apple's structured database sync layer — does support record versioning and conflict detection, but adopting it would require a fundamentally different sync path: structured record sync rather than encrypted blob sync, with Apple-specific implementation logic that breaks the provider-neutral architecture.
The current `ICloudService` implementation contains placeholder methods (`_saveRecord`, `_fetchRecord`, `_queryRecords`) that simulate storage behavior without enforcing real server-side version checks. These stubs exist to preserve the provider interface shape but do not constitute a production-safe collaborative sync implementation.
Collaborative workspace flows — workspace creation, invitation acceptance, and multi-device sync — are explicitly gated and blocked on iCloud in the current release.
The design principle behind this decision:
> providing no guarantee is safer than providing a false one.
---
## Provider Support Summary
| Provider | Version Token | Conditional Upload Mechanism | Collaborative Sync |
|---|---|---|---|
| **Google Drive** | revision / etag | `If-Match` header | Supported |
| **Dropbox** | `rev` identifier | `mode: update` + expected `rev` | Supported |
| **OneDrive** | `eTag` / `cTag` | `If-Match` via Microsoft Graph | Supported |
| **iCloud Drive** | Not exposed publicly | Not supported | **Gated — unsupported** |
---
# Why We Chose Snapshot Synchronization Instead of Delta Server Sync
A natural question is:
> why not upload only incremental changes?
The answer lies in our architectural priorities.
Delta synchronization typically requires:
* server-side coordination,
* change feeds,
* remote merge orchestration,
* and trusted backend infrastructure.
Sortify intentionally avoids centralized synchronization ownership.
By using:
* encrypted snapshots,
* local merge orchestration,
* and optimistic concurrency,
we preserve:
* user-controlled storage,
* end-to-end privacy,
* backend simplicity,
* and offline resilience.
This approach trades:
* bandwidth efficiency,
for:
* privacy,
* architectural independence,
* and deployment simplicity.
For Sortify's goals, that tradeoff was worth it.
---
# The Importance of Sync State Persistence
Another major improvement was introducing durable synchronization state.
We added `workspace_sync_state`, a persisted table with one row per workspace. It tracks:
```text
workspace_id — stable app-level workspace identifier
provider_file_id — provider-side file reference for the encrypted snapshot
provider_metadata_id — provider-side file reference for the metadata file
remote_version_token — etag / rev / revision captured at last successful sync
last_sync_attempt_at — timestamp of most recent sync attempt
last_sync_success_at — timestamp of last successful round-trip
last_error — structured error classification from last failure
retry_count — consecutive failure count for backoff logic
sync_status — current state (idle, syncing, failed, provider_blocked)
```
Before this table existed, sync state was implicit: the engine had to re-discover its position on every cycle by re-reading provider metadata. Errors were only visible in logs. There was no way to distinguish "remote is unchanged, skip the download" from "we have never synced this workspace before."
With `workspace_sync_state`, the engine always knows:
* whether the remote has changed since the last successful merge,
* which version it last merged from (for the OCC precondition),
* and whether the provider is in a blocked/error state requiring backoff.
Synchronization state is now explicit rather than implicit, which is a prerequisite for reliable observability and provider portability.
---
# What the Architecture Looks Like Today
Today, synchronization behaves approximately like this:
1. Resolve remote metadata and version token.
2. Determine whether remote changed.
3. Download and decrypt remote snapshot if needed.
4. Merge remote state into local state.
5. Replay local unsynced journal entries.
6. Export merged encrypted snapshot.
7. Attempt conditional upload.
8. Retry on precondition failure.
9. Persist new synchronization tokens.
10. Mark journal entries synchronized.
This produces eventual consistency without requiring centralized merge infrastructure.
---
# Remaining Challenges
No synchronization system is perfect.
Our architecture still has tradeoffs.
## Snapshot Size
Full snapshot uploads are heavier than true delta synchronization.
For small-to-medium collaborative workspaces, this is acceptable.
Larger workspaces may eventually require:
* compression,
* chunked snapshots,
* or incremental export optimization.
---
## Retry Storms
Heavy concurrent synchronization can create repeated retries under rapid edits.
Future improvements may include:
* randomized retry jitter,
* batching,
* smarter scheduling,
* and sync coalescing.
---
## Clock Dependency
Last-Write-Wins still depends on timestamps.
Future improvements could explore:
* logical clocks,
* hybrid logical clocks,
* or operation vectors.
At current scale, timestamp-based convergence remains a practical compromise.
---
## Sync Frequency and Provider Backoff
Sync runs on a configurable interval driven by the user's sync frequency setting (`30s`, `1m`, `5m`, `15m`). The default for new installs is `5m`. The scheduler reads the setting on startup and recreates the timer when the setting changes, without requiring an app restart.
When a provider returns a hard configuration error — such as a 403 from a disabled API endpoint — the sync engine applies a provider-level cooldown (10–15 minutes) rather than retrying on every scheduler tick. This prevents log noise and unnecessary battery/network usage during provider misconfiguration states. The cooldown is tracked in `workspace_sync_state.sync_status` as `provider_blocked` with a backoff expiry.
---
# The Most Important Lesson
The biggest architectural lesson from this journey was simple:
> synchronization correctness is less about "syncing data" and more about preserving user intent safely under concurrency.
That realization changed everything.
---
# Final Thoughts
Building synchronization without a central merge backend is difficult.
Building it while preserving:
* privacy,
* offline support,
* encrypted storage,
* collaboration,
* and user-controlled infrastructure,
is even harder.
But we believe this architecture represents the right balance for Sortify.
Instead of centralizing ownership of user data, we built:
* client-side intelligence,
* deterministic convergence,
* optimistic synchronization,
* and privacy-preserving collaboration.
The result is not merely "cloud sync."
It is a distributed synchronization architecture designed around a simple principle:
> users should not have to surrender ownership of their data to collaborate safely.
