A fault-tolerant sync service for SAP and CRM data

This sync service keeps the sales and operations platform aligned with SAP, external CRMs and tools added later. I designed it as a separate integration layer so failures, retries and new connectors could be handled independently of the core platform. I led its build in 2020.

Scheduled SAP files were the only integration path

Inventory, contracts and payments live in SAP, and every stage of the sales process runs against them.

The client's SAP setup did not expose an API we could use. It writes files into an FTP folder on its own schedule, and the platform has to keep up with whatever lands there. SAP holds the contracts finance runs on, so the platform reads from it and does not write back.

Lead data runs in the other direction. During the CRM migration, the service moved leads into the platform and kept both systems in step until the outgoing CRM was retired. The platform now holds the master lead record and publishes updates to the CRM used by marketing.

A record for every file, a queue for every stream

Before processing anything, the service records every file and lead it receives. That gives the operations team one place to see what completed, what failed and where it stopped.

Each sync runs in two passes. The first records what is waiting, either new files in the FTP folder or leads changed since their last sync, before anything is processed. The second works through those records one at a time on a queue. Every stream has its own queue, so a backlog in one does not slow the others, and any of them can be switched off in production without a release.

A record carries its status, its attempt count, the error it hit, and a link to the log from the run that failed. Failures retry automatically and stop after a defined number of attempts, so a file that cannot be processed leaves the queue but stays on the list. A file is marked done only after it has been applied and moved out of the FTP folder. That makes partial failures easier to find and reduces the risk of applying a file twice.

Records do not always arrive in the order they should. A payment can land before the contract it pays for exists on the platform, so it is held and matched on a later run rather than rejected.

SAP supplies inventory, contracts and payment data, which is what most of the platform's screens are built on. Receipts and invoices arrive as PDFs and are filed against their sales contract, so a buyer opens them in the app instead of asking their seller. Commission statements are matched to the seller they belong to.

The service also exposes an API for third-party tools, which is how landing page forms reach it through Zapier. Once the platform holds a lead, the sync publishes it to the CRM for qualification and scoring, then brings the relevant updates back. However a lead arrives, it carries the campaign that produced it, goes into that campaign's queue, and is distributed to the seller teams assigned to it by the rules that campaign runs under.

The trade-off is per-file specificity over a generic pipeline. Every new kind of file needs its own parsing and its own record type. In return, an operational question about a contract or lead can be traced to a row with a status and a log behind it.

When the systems around it change

The service has continued running as the tools around it changed. When marketing adopted a new CRM for qualification and scoring, I led the work to add another stream without changing the core platform.

I still own it in production, tracing specific records and correcting malformed or inaccurate upstream data before it reaches the platform.