Sortify Intelligent Synchronization: Optimizing Client-Side Merge While Preserving User Privacy

Modern synchronization is deceptively difficult.

At small scale, syncing data across devices seems straightforward: upload local changes, download remote updates, and keep everything “in sync.”

But the moment an application becomes:

• offline-first,
• collaborative,
• privacy-focused,
• encrypted,
• multi-device,
• and backend-light,

synchronization becomes one of the hardest engineering problems in the entire system.

At Sortify, we faced this challenge directly.

We wanted users to:

• fully own their data,
• store encrypted workspaces in their own cloud providers,
• collaborate across devices and users,
• work offline,
• and still experience reliable synchronization without destructive data loss.

We intentionally chose not to build a traditional centralized sync backend.

Instead, we designed a synchronization engine centered around:

• encrypted snapshot storage,
• client-side merge orchestration,
• optimistic concurrency control,
• deterministic conflict handling,
• and replayable local intent journaling.

This article explores the architecture, the failures we encountered, the distributed systems lessons we learned, and the synchronization model we ultimately implemented.

The Problem With Traditional Sync Models

Most synchronization systems follow one of two architectures.

Centralized Server Synchronization

The most common approach is:

• all clients communicate with a central backend,
• the backend stores authoritative state,
• the backend resolves conflicts,
• clients become relatively thin.

This model works well for:

• real-time collaborative editors,
• enterprise SaaS platforms,
• heavily coordinated multi-user systems.

But it comes with tradeoffs:

• infrastructure cost,
• operational complexity,
• trust requirements,
• and major privacy implications.

The server often becomes:

• the owner of user data,
• the merge authority,
• and the visibility layer for all collaborative content.

That did not align with Sortify’s philosophy.

Pure File Synchronization

At the opposite end is simple file synchronization:

• upload a database file,
• overwrite remote copy,
• download latest version later.

This is simple, but dangerously naive under concurrency.

If two users upload snapshots concurrently:

• later uploads overwrite earlier uploads,
• changes disappear,
• and synchronization becomes non-deterministic.

Initially, our architecture resembled this model more than we wanted to admit.

That became obvious once real-world collaborative edge cases appeared.

Our Constraints

Before designing the synchronization system, we defined several non-negotiable principles.

1. User-Owned Data

Sortify users should remain in control of their own storage.

Instead of storing workspace databases on our servers, we integrate with providers such as:

• Google Drive
• Dropbox
• OneDrive

This means:

• users control retention,
• users control deletion,
• users control account access,
• and Sortify itself does not become the permanent owner of workspace content.

2. End-to-End Encryption

Workspace snapshots are encrypted before upload using AES-256-GCM with a 256-bit workspace-specific key and a 96-bit random IV generated per upload.

The encryption key is stored in platform-native secure storage: iOS Keychain on Apple devices, Android Keystore on Android devices. It never leaves the device and is never transmitted to any server, including MokingBird's own infrastructure.

Cloud providers store opaque encrypted blobs rather than readable workspace content.

This ensures:

• cloud providers cannot inspect workspace data,
• MokingBird cannot inspect workspace contents in the ordinary course of operations,
• synchronization remains privacy-preserving by architecture rather than by policy,
• and user trust boundaries remain clear.

3. Offline-First Behavior

Users must be able to:

• create rooms,
• move items,
• rename entities,
• update metadata,
• and continue working offline.

Synchronization therefore cannot assume permanent connectivity.

4. Multi-User Collaboration

Workspaces are collaborative.

Multiple users may:

• edit the same workspace,
• move items simultaneously,
• rename rooms concurrently,
• or sync from multiple devices.

This introduces distributed synchronization challenges even without a traditional backend.

The First Major Failure

Our early synchronization engine used a snapshot-authoritative merge strategy.

The logic was conceptually simple:

• pull remote snapshot,
• compare against local database,
• apply remote state,
• delete rows missing remotely.

This appeared reasonable initially.

But it created a catastrophic edge case.

The “Missing Means Delete” Problem

Suppose a user:

• creates several items offline,
• those items exist only locally,
• synchronization runs before upload succeeds,
• remote snapshot does not contain those items yet.

The pull engine interpreted this as:

“these rows do not exist remotely, therefore they should be deleted locally.”

As a result:

• items vanished,
• rooms disappeared,
• counters dropped to zero,
• and local state was erased despite never being intentionally deleted.

This revealed a critical distributed systems truth:

Absence is not deletion.

Especially in eventually consistent systems.

Understanding the Real Problem

The issue was not simply:

• timestamps,
• ordering,
• or retries.

The deeper issue was architectural.

We had combined:

• snapshot synchronization,
• partial journaling,
• and destructive reconciliation semantics.

That combination was unsafe under concurrency.

Reframing the Architecture

We eventually shifted toward a fundamentally different model:

client-side merge with optimistic concurrency guarded snapshot publishing.

This became the foundation of the new synchronization engine.

The New Synchronization Model

The final architecture follows this sequence:

1. Read remote metadata/version token.
2. Pull latest snapshot if remote changed.
3. Merge locally using deterministic policies.
4. Replay unsynced local intent journal.
5. Export and encrypt merged snapshot.
6. Upload conditionally using optimistic concurrency tokens.
7. Retry on precondition failure.

This transformed synchronization from:

blind overwrite synchronization

into:

distributed optimistic synchronization.

Client-Side Merge With Snapshot Storage

One of the most unusual characteristics of Sortify’s synchronization engine is where merge logic happens.

Traditional systems:

• merge on the server.

Sortify:

• merges entirely on the client.

Cloud providers are treated as:

• encrypted blob stores,
• versioned snapshot containers,
• not synchronization authorities.

This architecture preserves privacy while still allowing convergence across clients.

Optimistic Concurrency Control (OCC)

A major architectural upgrade was introducing optimistic concurrency control.

Without OCC:

• two users can upload snapshots simultaneously,
• later uploads overwrite earlier snapshots,
• remote history becomes unstable.

We solved this by using provider-issued version tokens.

Examples include:

• Google Drive revision metadata,
• Dropbox rev identifiers,
• OneDrive etags.

Before uploading a snapshot, Sortify verifies:

“Is the remote file still the same version I merged against?”

If not:

• upload is rejected,
• the client pulls latest state again,
• merges again,
• and retries safely.

This small change fundamentally transformed synchronization reliability.

Replayable Local Intent Journal

Another critical architectural addition was the local change journal.

Every mutation now:

• updates local tables,
• and appends a journal entry in the same transaction.

Examples include:

• item updates,
• room renames,
• photo changes,
• membership changes,
• tombstone deletes.

The journal represents:

local user intent not yet safely published remotely.

During synchronization:

• remote snapshot changes are applied first,
• then unsynced journal entries are replayed.

This guarantees that local intent is preserved during merge operations.

Tombstone Convergence

Traditional delete behavior creates major synchronization hazards.

Originally, some user-facing deletes physically removed rows from the database.

That approach becomes incompatible with eventual consistency.

Instead, Sortify moved toward tombstone-based convergence.

Rather than deleting rows:

is_active = 0
deleted_at = timestamp

Deletes become:

• explicit state transitions,
• synchronizable events,
• and conflict-resolvable operations.

Most importantly:

remote absence alone is never interpreted as delete intent.

This eliminated the entire class of “items vanished” failures.

Deterministic Merge Policy

Distributed synchronization becomes dangerous when merge outcomes are non-deterministic.

If two clients resolve conflicts differently:

• synchronization never converges.

We therefore implemented deterministic merge policies:

• Last-Write-Wins by updated_at: the entity with the higher timestamp wins.
• Lexical tie-breaker on equal timestamps: when updated_at values are identical, the winning version is determined by lexical comparison of updated_by (user identifier) or device ID. This comparison is deterministic: every device running the merge on the same two records will produce the same winner.
• Deletion semantics: a tombstone (is_active = 0) with an updated_at timestamp participates in LWW the same way any other update does. An explicit delete can be overridden by a later update, and vice versa, by timestamp.

Even equal timestamps now produce stable, identical results regardless of which device runs the merge.

This is subtle, but essential: non-deterministic conflict resolution causes perpetual divergence because two clients can both believe they published the authoritative state while holding different data.

Bounded Retry Orchestration

Optimistic concurrency naturally introduces retries.

Consider:

• User A syncs,
• User B syncs milliseconds later,
• both began from the same remote snapshot.

One upload succeeds. The other receives a precondition failure.

Rather than overwriting:

• Sortify retries automatically,
• re-pulls latest state,
• merges again,
• republishes safely.

Retries are bounded to avoid:

• infinite loops,
• battery drain,
• synchronization storms.

Implementation Phases

The rebuild of the synchronization engine was structured as eight discrete implementation phases, each targeting a specific failure class.

All eight phases are shipped in the current release.

Provider-Aware Conditional Uploads

Different providers expose concurrency semantics differently.

For example:

Google Drive

OAuth Scope Decision

Sortify's current Google Drive implementation is based on least-privilege, targeted workspace authorization. The core workspace is stored in an encrypted workspace.sortifypkg file, and optional photos are stored in a separate encrypted photo_workspace.sortifypkg file.

Google Drive's drive.file behavior is strict: a collaborator's app may not be allowed to read a package created by another member's app instance until that exact file has been authorized. Sortify handles this through targeted Picker authorization instead of using Google Drive as a browsable account-wide store.

The practical rule is:

• solo workspaces can use files Sortify creates directly;
• collaborative Google Drive workspaces may require a Picker authorization step for the shared workspace package;
• optional Google Drive photo sync may require a separate Picker authorization step for the photo package.

Expected Google Drive 404/File not found responses before targeted authorization are treated as "authorization required", not as destructive sync failures. The app should route the member to the Picker or workspace settings action instead of showing raw provider errors.

Concurrency Semantics

Google Drive synchronization leverages:

• revision-aware file metadata,
• If-Match conditional upload semantics (etag-based precondition),
• and persistent remote token tracking in workspace_sync_state.

An upload that specifies an etag that no longer matches the current remote file is rejected by Google Drive with a precondition failure. Sortify treats this as a signal to re-pull, re-merge, and retry.

Dropbox

Dropbox exposes rev identifiers on every file. Conditional uploads are supported natively: an upload specifying mode: update and an expected rev value is accepted only if the current remote rev matches. If the file has changed since the rev was captured, the request is rejected.

This is functionally equivalent to compare-and-swap: upload succeeds only if the file is still at the version the client merged from.

OneDrive

OneDrive exposes eTag and cTag fields via the Microsoft Graph API. Conditional uploads are supported using If-Match: <etag> request headers. An upload that specifies an etag that no longer matches the remote file's current etag is rejected with a precondition failure, triggering the retry loop.

iCloud — Gated as Unsupported for Collaborative Sync

iCloud Drive does not expose the optimistic concurrency primitives that Sortify's sync architecture depends on. Google Drive, Dropbox, and OneDrive all provide explicit version tokens and conditional upload semantics (etag, rev, If-Match). iCloud Drive does not expose equivalent public API primitives for file-based conditional uploads, making it impossible to implement the compare-and-swap guarantee that prevents concurrent overwrites.

CloudKit — Apple's structured database sync layer — does support record versioning and conflict detection, but adopting it would require a fundamentally different sync path: structured record sync rather than encrypted blob sync, with Apple-specific implementation logic that breaks the provider-neutral architecture.

The current ICloudService implementation contains placeholder methods (_saveRecord, _fetchRecord, _queryRecords) that simulate storage behavior without enforcing real server-side version checks. These stubs exist to preserve the provider interface shape but do not constitute a production-safe collaborative sync implementation.

Collaborative workspace flows — workspace creation, invitation acceptance, and multi-device sync — are explicitly gated and blocked on iCloud in the current release.

The design principle behind this decision:

providing no guarantee is safer than providing a false one.

Provider Support Summary

Why We Chose Snapshot Synchronization Instead of Delta Server Sync

A natural question is:

why not upload only incremental changes?

The answer lies in our architectural priorities.

Delta synchronization typically requires:

• server-side coordination,
• change feeds,
• remote merge orchestration,
• and trusted backend infrastructure.

Sortify intentionally avoids centralized synchronization ownership.

By using:

• encrypted snapshots,
• local merge orchestration,
• and optimistic concurrency,

we preserve:

• user-controlled storage,
• end-to-end privacy,
• backend simplicity,
• and offline resilience.

This approach trades:

• bandwidth efficiency,

for:

• privacy,
• architectural independence,
• and deployment simplicity.

For Sortify’s goals, that tradeoff was worth it.

The Importance of Sync State Persistence

Another major improvement was introducing durable synchronization state.

We added workspace_sync_state, a persisted table with one row per workspace. It tracks:

workspace_id          — stable app-level workspace identifier
provider_file_id      — provider-side file reference for the encrypted snapshot
provider_metadata_id  — provider-side file reference for the metadata file
remote_version_token  — etag / rev / revision captured at last successful sync
last_sync_attempt_at  — timestamp of most recent sync attempt
last_sync_success_at  — timestamp of last successful round-trip
last_error            — structured error classification from last failure
retry_count           — consecutive failure count for backoff logic
sync_status           — current state (idle, syncing, failed, provider_blocked)

Before this table existed, sync state was implicit: the engine had to re-discover its position on every cycle by re-reading provider metadata. Errors were only visible in logs. There was no way to distinguish "remote is unchanged, skip the download" from "we have never synced this workspace before."

With workspace_sync_state, the engine always knows:

• whether the remote has changed since the last successful merge,
• which version it last merged from (for the OCC precondition),
• and whether the provider is in a blocked/error state requiring backoff.

Synchronization state is now explicit rather than implicit, which is a prerequisite for reliable observability and provider portability.

What the Architecture Looks Like Today

Today, synchronization behaves approximately like this:

1. Resolve remote metadata and version token.
2. Determine whether remote changed.
3. Download and decrypt remote snapshot if needed.
4. Merge remote state into local state.
5. Replay local unsynced journal entries.
6. Export merged encrypted snapshot.
7. Attempt conditional upload.
8. Retry on precondition failure.
9. Persist new synchronization tokens.
10. Mark journal entries synchronized.

This produces eventual consistency without requiring centralized merge infrastructure.

Remaining Challenges

No synchronization system is perfect.

Our architecture still has tradeoffs.

Snapshot Size

Full snapshot uploads are heavier than true delta synchronization.

For small-to-medium collaborative workspaces, this is acceptable.

Larger workspaces may eventually require:

• compression,
• chunked snapshots,
• or incremental export optimization.

Retry Storms

Heavy concurrent synchronization can create repeated retries under rapid edits.

Future improvements may include:

• randomized retry jitter,
• batching,
• smarter scheduling,
• and sync coalescing.

Clock Dependency

Last-Write-Wins still depends on timestamps.

Future improvements could explore:

• logical clocks,
• hybrid logical clocks,
• or operation vectors.

At current scale, timestamp-based convergence remains a practical compromise.

Sync Frequency and Provider Backoff

Sync runs on a configurable interval driven by the user's sync frequency setting (30s, 1m, 5m, 15m). The default for new installs is 5m. The scheduler reads the setting on startup and recreates the timer when the setting changes, without requiring an app restart.

When a provider returns a hard configuration error — such as a 403 from a disabled API endpoint — the sync engine applies a provider-level cooldown (10–15 minutes) rather than retrying on every scheduler tick. This prevents log noise and unnecessary battery/network usage during provider misconfiguration states. The cooldown is tracked in workspace_sync_state.sync_status as provider_blocked with a backoff expiry.

The Most Important Lesson

The biggest architectural lesson from this journey was simple:

synchronization correctness is less about “syncing data” and more about preserving user intent safely under concurrency.

That realization changed everything.

Final Thoughts

Building synchronization without a central merge backend is difficult.

Building it while preserving:

• privacy,
• offline support,
• encrypted storage,
• collaboration,
• and user-controlled infrastructure,

is even harder.

But we believe this architecture represents the right balance for Sortify.

Instead of centralizing ownership of user data, we built:

• client-side intelligence,
• deterministic convergence,
• optimistic synchronization,
• and privacy-preserving collaboration.

The result is not merely “cloud sync.”

It is a distributed synchronization architecture designed around a simple principle:

users should not have to surrender ownership of their data to collaborate safely.

2026 Implementation Update: Hybrid Provider Sync And Managed SyncNative

The architecture has evolved into a hybrid model with two sync families sharing the same local-first and zero-knowledge principles.

Provider Package Sync

Google Drive, OneDrive, and Dropbox remain user-owned provider sync engines. They use:

• local SQLite as the operational store;
• encrypted workspace packages in the user's provider account;
• local change journaling;
• non-destructive pull merge;
• tombstones for delete intent;
• provider version tokens and conditional upload for optimistic concurrency.

Google Drive now uses targeted Picker authorization for exact package files under drive.file behavior. The core workspace package and optional photo package are authorized separately when needed. A Drive 404 before authorization is treated as provider action required, not as proof of data loss.

Managed Sortify Cloud And Enterprise SyncNative

Sortify Cloud and Enterprise SyncNative are managed encrypted event transports. They do not make Firebase a plaintext inventory database.

The managed sync pipeline is:

1. Local SQLite mutation writes a normal local row and a journal entry.
2. The managed change bridge converts journal entries into encrypted event envelopes.
3. Firestore stores encrypted event metadata and payload fields.
4. Devices reserve/apply monotonic workspace sequences.
5. Other members pull/listen for encrypted events after their last applied sequence.
6. The remote applier decrypts with the local workspace key and applies the change to SQLite.
7. Encrypted checkpoints periodically provide a compact recovery/bootstrap base.

Sortify Cloud is the simpler workspace-level managed collaboration tier. Enterprise SyncNative uses the same encrypted transport foundation with organization-level administration, workspace grouping, member controls, recovery/security operations, and audit-safe metadata.

Membership And Recovery

Active membership is the authorization boundary for collaboration. Invited secondary users can participate in a workspace based on membership, even if they are not the paying workspace owner. Creation and enterprise administration remain entitlement-gated.

Recovery uses encrypted account and workspace key envelopes. Login recovery can discover active memberships, restore the workspace key envelope for the current user, and run a pull-only restore so an empty local database does not overwrite remote state. Managed workspace recovery can also create an approval request for an owner/admin to approve an encrypted replacement key for a trusted member device.

Removal and leaving revoke future access where the backend/provider can enforce it. They do not guarantee remote wipe of data already cached on a device that remains offline.

Optional Photo Sync

Photos are no longer treated as mandatory workspace state. Core item data can sync without photos. Each member decides per workspace and per device whether to upload/download photos.

• Google Drive stores optional photos in encrypted photo_workspace.sortifypkg and may require targeted Picker authorization.
• OneDrive and Dropbox use equivalent encrypted photo packages.
• Sortify Cloud and Enterprise use encrypted managed media in Firebase Storage when enabled.
• Photos are compressed as inventory reference images, not preserved as original-quality media backups.

This keeps normal room/item sync reliable even when provider media authorization, storage rules, or a member's photo preference blocks photo transfer.