The Situation
A 7-hour weekly upload. A 15% failure rate. A platform used to send emergency alerts.
A Seattle-based Fortune 10 organization used a third-party emergency notification system to contact employees in situations such as gas leaks, fires, severe weather, or active shooter events. The platform accepted employee data through CSV uploads over FTP. As the organization scaled, the process used to manage and upload that data had become operationally untenable and created serious risk.
The company was uploading a full database drop-and-replace payload every week. The workflow took around 7 hours to run and had a 15% failure rate.
Even worse, failures were often hard to detect. Error notifications were vague, and in some cases the system owner had to log in manually and verify whether the upload had actually succeeded.
This was not just an inefficient workflow. It was a reliability problem tied to emergency communications. The business needed a process that was faster to manage, easier to trust, and less likely to fail silently.
Why It Mattered
This was a reliability problem tied to emergency communications.
The platform sent voice and text alerts during gas leaks, fires, severe weather, and active shooter events. An employee data workflow that failed 15% of the time and required manual verification was not acceptable in that context.
Silent failures in a workflow tied to emergency alerting are not a process inconvenience. They are an operational risk.
What We Did
We redesigned the sync process and moved the organization onto a more reliable integration path.
entropy forge reviewed the existing data management process and the integration options supported by the third-party vendor, then redesigned the workflow from the ground up.
Identify the weekly full-refresh model as the core bottleneck
We reviewed the data management process and integration options supported by the vendor. The weekly full drop-and-replace payload was the root cause — the model no longer matched the operational need at scale.
Redesign around a monthly full sync with monitoring, alerting, and retry behavior
We changed the full-refresh cycle to monthly, added automated monitoring to make failures visible before they went undetected, and built in retry behavior so the process could recover without manual intervention.
Move the organization onto the vendor's v2 REST API
We worked directly with the vendor to migrate from the CSV/FTP integration to the v2 REST API, continuing the monthly drop-and-replace path for full syncs while enabling a faster update path for ongoing changes.
Implement near-real-time employee data updates via vendor APIs and internal data service
The resulting system used the vendor APIs and an internal employee data service to detect and update employee data in near real time, replacing the need to wait for the next weekly batch.
Result
A more reliable integration with better visibility and less operational risk.
- Replaced a 7-hour, failure-prone batch process with a controlled sync pattern.
The workflow moved from a long-running weekly upload with weak visibility to a more observable and resilient integration model.
- Automated monitoring and retry behavior replaced manual verification.
Instead of logging in to check whether an upload had succeeded, the system surfaced failures automatically and handled recoverable errors without human involvement.
- Near-real-time employee data updates replaced the weekly batch dependency.
The business no longer had to rely on a brittle update cycle that took 7 hours and failed 15% of the time.
Get Started
Have a critical workflow running on brittle sync logic, manual checks, or silent failure risk?
Request a Workflow Assessment and bring us the integration your team does not fully trust.
- Focused on one high-friction workflow
- Diagnostic + implementation oriented
- Built around your current systems
- Not a long strategy engagement
Or call us directly at +1 (425) 954-3051