Vantage production systems have been affected by a broken upgrade for one of our dependency during a standard maintenance operation on our clusters (US1, EU1, EU2, AP1, and GovCloud). Such upgrade didn't raise any problem in our pre-production environments, while we immediately noticed networking issues due to the upgrade just after the promotion into production starting on January 31st at 22:57 UTC. After some investigation and triage, we performed a rollback and systems came back up and running on January 31st at 23:42 UTC. We're now improving our processes with additional testing scenarios to prevent similar problems from happening again.