Vantage connectivity issues
Incident Report for Nozomi Networks
Vantage production systems have been affected by a broken upgrade for one of our dependency during a standard maintenance operation on our clusters (US1, EU1, EU2, AP1, and GovCloud).
Such upgrade didn't raise any problem in our pre-production environments, while we immediately noticed networking issues due to the upgrade just after the promotion into production starting on January 31st at 22:57 UTC.
After some investigation and triage, we performed a rollback and systems came back up and running on January 31st at 23:42 UTC.
We're now improving our processes with additional testing scenarios to prevent similar problems from happening again.
Posted Jan 31, 2024 - 23:00 UTC