1 |
Yes guys, I know that Bugzilla is down. |
2 |
|
3 |
Last night, while I was sleeping, we got a slew of IPs hitting the |
4 |
dependency graph generation. This wouldn't have been a problem normally, |
5 |
but they seemed to hit graphs that took an inordinate amount of memory |
6 |
to generate with GraphViz (collectively 8GiB of RAM and 32GiB of swap). |
7 |
This morning, I got into the box, saw some OOMs of the GraphViz |
8 |
processes, killed off the remaining ones, banned the IPs, and then had |
9 |
to rush off to a work meeting. |
10 |
|
11 |
I came back this evening, to find the box not responding again, and my |
12 |
last SSH shell was painfully slow then just hung - not died, but hung, |
13 |
the TCP connection is still alive, but the shell isn't responding |
14 |
(shortly after I had seen a loadavg exceeding 1k). |
15 |
|
16 |
Whomever attacked it came back I think. And I can't get in to block them |
17 |
right now. I've contacted the sponsor so that they can hard reboot the |
18 |
box for me, but I don't expect any action from them for the next 5-6 |
19 |
hours at least. |
20 |
|
21 |
Meantime, I'm enacting a plan B, to at least get us some slow Bugzilla |
22 |
functionality, via the second bugzilla box that normally runs the |
23 |
background computations (duplicates etc). I do however fully expect |
24 |
whomever the attacker is to come right back at it - so I'm turning off |
25 |
the dependency graphs. |
26 |
|
27 |
-- |
28 |
Robin Hugh Johnson |
29 |
Gentoo Linux Developer & Infra Guy |
30 |
E-Mail : robbat2@g.o |
31 |
GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 |