Postmortem Summary: Salesforce–Call Center Integration Outage
Incident Date: April 17, 2025
Duration: 9:30 AM – 4:30 PM MT
Resolution Time: 5:00 PM MT
Summary: On April 17, 2025, an integration issue between Salesforce and the Call Center system resulted in the failure of outbound messages containing new student data from 9:30 AM to approximately 4:30 PM MT. The root cause was traced to the expiration of an authentication credential used by limited integration services. As a result, new student records were not transmitted to the Call Center during this window.
Impact:
- No new student records were sent to the Call Center during the affected period.
- Associated outreach activity was paused for impacted students.
- All failed transmissions have since been automatically retried and are expected to be fully processed by 5:00 PM MT on April 17.
Root Cause:
Expired authentication credential used in the integration between Salesforce and Call Center systems.
Mitigation & Prevention Strategies Implemented:
- Credential Monitoring & Alerts: Expanded credential expiration monitoring has been enabled with automated alerts configured to notify system owners.
- Auto-Renewal Mechanism: Where supported, credentials have been updated to use auto-renewing authentication methods (e.g., OAuth refresh tokens or connected app certificates with scheduled rotation).
Next Steps:
- Automate credential rotation where feasible.
- Conduct quarterly reviews of integration and credential lifecycle management (expanded from semi-annual)