[gentoo-user] Strange network behaviour: NIC goes down, DHCP lease renewal fails

[Marc Joliet]

Hi all,

at work I'm (well, *we* are) facing an interesting problem.  Since we are sort
of stabbing in the dark here, I thought I'd ask here.  Also, since this is from
work, I will not be able to diverge very many details (not to mention that as a
student worker I simply don't *know* many details).  However, I do have
permission from my boss to ask about this in an anonymised fashion.

The symptom we're seeing is that the NIC goes down and DHCP packets stop getting
through after a certain amount of time.  What happens is:

1.) The NIC is brought up (some built-in Intel model).

2.) A DHCP client configures it.

3.) The network connection is lost at some point (the amount of time this takes
    varies, but it can be as little as 20 minutes).

4.) Eventually the lease runs out and the DHCP client tries to renew it, but
    gets no response.  Sometimes, after many hours (at least 6), it will get a
    DHCPACK, but that's it.  One of our sysadmins says that not only does
    the DHCP server never see the packets, but the managed switch that the PC
    is directly attached to *also* never does (again, except for when the
    occasional DHCPACK comes).

4.) Restart the network device.  A reboot is not required, but it is necessary
    to terminate the DHCP client.  After that everything works again.

5.) GOTO 3.

(Note that I have observed that steps 3 and 4 do not necessarily occur in
order.)

This has been rather baffling, since this problem is limited to 3 computers.

One of them (the longest running) runs Gentoo, courtesy of me.  This is the
first one we saw the problem with.  Since we couldn't figure it out (switching
from dhcpcd to dhclient, turning off the firewall, monitoring with tcpdump,
etc., all with help from one of our sysadmins; Google, too, of course), Gentoo
was "blamed", so we got a replacement PC with Fedora 20 on it, which *also*
showed this behaviour.  Both PCs run some special software (some of it mine).
Thus, at some point this software was "blamed".

So we started experimenting: we configured the Fedora PC to *not* start the
special software, and have not seen any problems all week.  Yesterday afternoon
I then started *one* of the programs, and had not seen any problems yet by the
time I went home.

So that would speak *for* that theory, right? Well, for comparison, my boss
recently started running a separate PC, also with a bog-standard Fedora 20.
Guess what: it *also* shows the *exact* same behaviour as the other two PCs
("journalctl -u NetworkManager" shows pages upon pages of unanswered
DHCPREQUESTs, with the occasional response thrown in). Note here that this PC
is on a different switch and in a different VLAN.

The choice of Fedora comes from the fact that we use a Fedora based distro
internally, so it is "known".  PCs running it have *not* shown the behaviour
above (AFAIK not even *once*).  Thus, one of the few things I can think of is
finding out what is different about them relative to the standard Fedora.

Right now my main ideas on what the culprit could be are:

- The computers' kernel/network device is improperly configured.  That is,
  maybe special configuration is needed for the computers to work properly as
  clients in the network.  I'm thinking of support for some (from my
  perspective) obscure protocol(s).

- It's a network problem.  The three computers are in two different VLANs,
  while the workplace computers running the internal Fedora based distro are in
  a third (the main network that all the normal Windows and Linux workstations
  are connected to).  However, they are on the same switch as the two computers
  running my software.  One argument against this is that the Windows PC that
  runs on the same VLAN does *not* have any problems like this.

One of the other ideas I had was faulty power management, and I did read of
problems of the sort regarding the exact same network card that is in the old
Gentoo machine on an HP support forum (from around 2008).  However, the local
sysadmin said that they have had nothing but good experience with those network
cards. Also: *three* computers with NIC power management problems?  That sounds
a bit far-fetched to me.  Nevertheless, I am not fully discounting the
possibility.

You can imagine how confusing and frustrating this is.

So, has anybody here ever experienced something like this? Any ideas on what
could be the cause?

Greetings
-- 
Marc Joliet
--
"People who think they know everything really annoy those of us who know we
don't" - Bjarne Stroustrup