| 1 |
Hi, |
| 2 |
|
| 3 |
Many of you reported that various infra-owned services were not functioning |
| 4 |
properly these past two days. We had some package upgrades roll out that |
| 5 |
were bad, and a subset of services were non-functional while we worked |
| 6 |
through the incident. |
| 7 |
|
| 8 |
(a) The upgrades did not appear to cause any data loss, but service |
| 9 |
availability was reduced (500s / 502s for some HTTP services.) |
| 10 |
(b) Some hosts may have had problems running some commands. E.g. I know |
| 11 |
that 'grep' didn't work for a while. If you observed errors related to |
| 12 |
missing 'GLIBC_2.33' symbols then this was likely related. |
| 13 |
(c) Many replicated services (including infra-status) rely on gitweb to |
| 14 |
take updates from git; and gitweb was not up, so they were unable to |
| 15 |
receive updates. |
| 16 |
(d) More impacted notes will follow in a postmortem that will come later |
| 17 |
this week for this incident. |
| 18 |
|
| 19 |
Thanks to all of you who reported problems and apologies for the service |
| 20 |
disruptions. |
| 21 |
|
| 22 |
-A |
Archives