Gentoo Archives: gentoo-dev

From: "Robin H. Johnson" <robbat2@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] status report, 2009/03/14 06h00 UTC
Date: Sat, 14 Mar 2009 06:03:10
Message-Id: 20090314015604.GB29345@curie-int
Yes guys, I know that Bugzilla is down.

Last night, while I was sleeping, we got a slew of IPs hitting the
dependency graph generation. This wouldn't have been a problem normally,
but they seemed to hit graphs that took an inordinate amount of memory
to generate with GraphViz (collectively 8GiB of RAM and 32GiB of swap).
This morning, I got into the box, saw some OOMs of the GraphViz
processes, killed off the remaining ones, banned the IPs, and then had
to rush off to a work meeting.

I came back this evening, to find the box not responding again, and my
last SSH shell was painfully slow then just hung - not died, but hung,
the TCP connection is still alive, but the shell isn't responding
(shortly after I had seen a loadavg exceeding 1k).

Whomever attacked it came back I think. And I can't get in to block them
right now.  I've contacted the sponsor so that they can hard reboot the
box for me, but I don't expect any action from them for the next 5-6
hours at least.

Meantime, I'm enacting a plan B, to at least get us some slow Bugzilla
functionality, via the second bugzilla box that normally runs the
background computations (duplicates etc). I do however fully expect
whomever the attacker is to come right back at it - so I'm turning off
the dependency graphs.

Robin Hugh Johnson
Gentoo Linux Developer & Infra Guy
E-Mail     : robbat2@g.o
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85


Subject Author
[gentoo-dev] status report, 2009/03/19 10h00 UTC "Robin H. Johnson" <robbat2@g.o>