Gentoo Archives: gentoo-dev

From: "Robin H. Johnson" <robbat2@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] bugs.gentoo.org status report, 2009/03/14 06h00 UTC
Date: Sat, 14 Mar 2009 06:03:10
Message-Id: 20090314015604.GB29345@curie-int
1 Yes guys, I know that Bugzilla is down.
2
3 Last night, while I was sleeping, we got a slew of IPs hitting the
4 dependency graph generation. This wouldn't have been a problem normally,
5 but they seemed to hit graphs that took an inordinate amount of memory
6 to generate with GraphViz (collectively 8GiB of RAM and 32GiB of swap).
7 This morning, I got into the box, saw some OOMs of the GraphViz
8 processes, killed off the remaining ones, banned the IPs, and then had
9 to rush off to a work meeting.
10
11 I came back this evening, to find the box not responding again, and my
12 last SSH shell was painfully slow then just hung - not died, but hung,
13 the TCP connection is still alive, but the shell isn't responding
14 (shortly after I had seen a loadavg exceeding 1k).
15
16 Whomever attacked it came back I think. And I can't get in to block them
17 right now. I've contacted the sponsor so that they can hard reboot the
18 box for me, but I don't expect any action from them for the next 5-6
19 hours at least.
20
21 Meantime, I'm enacting a plan B, to at least get us some slow Bugzilla
22 functionality, via the second bugzilla box that normally runs the
23 background computations (duplicates etc). I do however fully expect
24 whomever the attacker is to come right back at it - so I'm turning off
25 the dependency graphs.
26
27 --
28 Robin Hugh Johnson
29 Gentoo Linux Developer & Infra Guy
30 E-Mail : robbat2@g.o
31 GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85

Replies

Subject Author
[gentoo-dev] bugs.gentoo.org status report, 2009/03/19 10h00 UTC "Robin H. Johnson" <robbat2@g.o>