1 |
Hi, |
2 |
|
3 |
Many of you reported that various infra-owned services were not functioning |
4 |
properly these past two days. We had some package upgrades roll out that |
5 |
were bad, and a subset of services were non-functional while we worked |
6 |
through the incident. |
7 |
|
8 |
(a) The upgrades did not appear to cause any data loss, but service |
9 |
availability was reduced (500s / 502s for some HTTP services.) |
10 |
(b) Some hosts may have had problems running some commands. E.g. I know |
11 |
that 'grep' didn't work for a while. If you observed errors related to |
12 |
missing 'GLIBC_2.33' symbols then this was likely related. |
13 |
(c) Many replicated services (including infra-status) rely on gitweb to |
14 |
take updates from git; and gitweb was not up, so they were unable to |
15 |
receive updates. |
16 |
(d) More impacted notes will follow in a postmortem that will come later |
17 |
this week for this incident. |
18 |
|
19 |
Thanks to all of you who reported problems and apologies for the service |
20 |
disruptions. |
21 |
|
22 |
-A |