A fault-tolerant sync service for SAP and CRM data

I built this service in 2020 to keep the sales and operations platform in sync with SAP and the client's CRM. I kept it separate from the main platform so retries, failures, and new integrations could be handled without adding all of that machinery to the product itself.

SAP gave us files, not an API

The inventory, contracts, and payments used throughout the sales platform come from SAP. The client's SAP setup did not have an API we could use; it exported files to an FTP folder on its own schedule.

That meant the integration had to cope with whatever appeared in that folder, including records arriving out of order or with bad upstream data. SAP remained the source of truth for contracts and finance, so the platform only reads from it.

Lead data moved differently. During the first CRM migration, the service copied leads into the platform and kept both systems aligned until the outgoing CRM was retired. Today the platform owns the lead, publishes it to the CRM marketing uses for qualification and scoring, then pulls the relevant changes back.

Making every sync visible

The service records each file or lead before processing it. Operations can see what is waiting, what completed, what failed, and where it stopped instead of having to infer all of that from a log file.

Each sync runs in two steps. The first discovers new files or changed leads and records them. The second processes those records one at a time on a queue. Each data stream has its own queue, so a backlog in one does not stop the rest, and we can switch an individual stream off in production without deploying code.

A sync record stores its status, attempt count, last error, and the log for the failed run. Failed jobs retry automatically up to a limit, then stop retrying but remain visible for investigation. A file is only marked complete after its data has been applied and the source file has moved out of the FTP folder, reducing the chance of it being applied twice.

Some records arrive before the data they depend on. A payment, for example, can appear before its contract exists in the platform. The service holds it and tries the match again on a later run instead of throwing the record away.

The data moving through it

SAP supplies inventory, contracts, and payments. Receipts and invoices arrive as PDFs and are attached to the correct sales contract, which lets buyers open them in the mobile app instead of asking their seller. Commission statements are matched to the seller they belong to.

The service also exposes an API for other tools. Landing-page forms reach it through Zapier, then the resulting leads go into the right campaign and distribution queue. Once a lead reaches the platform, the sync publishes it to the CRM for qualification and scoring and brings the useful updates back.

I chose specific processors for each type of file rather than trying to make one generic import pipeline. Adding a new file format takes more code, but when someone asks why a contract or payment did not appear, we can trace it to a record with a status, attempts, and a log.

Still running as the surrounding tools change

The service has stayed in production while the tools around it have changed. When marketing adopted another CRM, I added a new stream without changing the core sales platform.

I still look after it in production, including tracing individual records and dealing with malformed or inaccurate data before it reaches the rest of the system.