Data Center Data Availability RCA
Incident date: 25–29 September 2026
Summary
A database storage issue interrupted data ingestion and access through the Data Center user interface and API. A standby database had stopped keeping up with the primary. This caused recovery records to accumulate until storage was exhausted.
Incoming events were correctly buffered during the outage. Access returned in stages, and full service was restored on 29 September 2026 at 09:47 CEST, ahead of the estimated time of 12:00 CEST. Based on our recovery checks, no data has been lost.
Impact
- Data ingestion stopped on 25 September at approximately 16:00 CEST.
- Data access through the user interface and API was temporarily unavailable.
- A temporary switch to a remaining database initially exposed older data. The newer data was subsequently recovered and incoming events were reconciled.
- Parking occupancy dashboards were unavailable or out of date until the relevant data was restored.
No Impact
- Sensor operation and critical control functions were not affected.
- The earlier standby replication failure did not directly affect customer queries: standby databases are not used to serve application requests.
- Incoming events accumulated in the message buffer and were processed after recovery.
Timeline
6 September 2026, shortly after 02:00 CEST: A standby database began falling behind during infrastructure maintenance. The primary continued serving complete data.25 September 2026, approximately 16:00 CEST: Storage exhaustion interrupted the primary database. An attempted failover did not restore sustained service; incoming events were buffered as intended.28 September 2026, 08:15 CEST: The first incident report was received.28 September 2026, 08:50 CEST: An initial check confirmed that data was unavailable through the user interface and API.28 September 2026, 15:45 CEST: The status update recorded that the issue had been identified and recovery work had begun.28 September 2026, 16:08 CEST: Data from before 6 September was available again. Recovery of the remaining data continued.28–29 September 2026: The newer database history was recovered, additional incoming data was reconciled, and buffered events were processed.29 September 2026, 09:47 CEST: Full service was restored, 2 hours and 13 minutes ahead of the announced estimate.
Root Cause
A database used for event processing ran out of storage after recovery records accumulated. The root cause is that a standby database could no longer follow its primary's database history, preventing records needed for replication from being released.
A separate notification delivery failure delayed awareness of the issue.
Resolution
The correct database history was restored from a volume snapshot and archived recovery records. Data received during the temporary switch was reconciled with the restored database. Ingestion was then redirected to the recovered service, and buffered events were processed. Recovery checks found no data loss.
Follow-Up Actions
- Strengthen alert delivery with end-to-end checks and an independent fallback for critical alerts.