Root Cause Analysis (RCA) Report

Incident Summary: On November 11, 2024, a disruption impacted the Financial Cloud CRM platform, specifically affecting communications and new user creation. This downtime was traced to a routine database update in the billing module, which introduced a new library dependency. This dependency required implementation and deployment. Communication services were queued during this process, and user updates could not be processed.

Root Cause: The issue arose from a routine database upgrade. The database upgrade introduced unforeseen system conflicts that temporarily disrupted the functionality of platform services, including communications and new user creations.

Resolution:
To restore affected components, our team identified the issue and implemented a fix, followed by monitoring to ensure system stability. The resolution steps were executed as follows:

1. Issue Identification (08:00 UTC): Root cause analysis confirmed the library dependency conflict resulting from the update.
2. Fix Implementation (12:15 UTC): A fix was deployed to address the conflicts, restoring system functionalities.
3. System Monitoring (12:15–12:30 UTC): Post-deployment monitoring confirmed the stability and reliability of the restored services.
4. Incident Resolution (12:30 UTC): Services confirmed as fully operational, with all affected components verified for performance.

Timeline of Events:
11/11/2024 08:00 UTC: Issue identified. Communications and new user creations were disrupted due to the new dependency conflict.
11/11/2024 12:15 UTC: Fix implemented, and monitoring began.
11/11/2024 12:30 UTC: Full resolution confirmed, and services restored.

Preventive Measures:

Enhanced Dependency Testing: Implementing additional testing protocols for new library dependencies to identify potential conflicts during pre-deployment stages on secondary hybrid services.

Impact Assessment:
The issue affected:
Agent – Communication Queued
Admin – User Creation Limited
API - Comms API Queued
No data loss occurred during this incident. All services were restored without further customer impact.

Contact Information: For further questions or support, please reach out to our support team.