Re: [gentoo-user] Running HTTP and DNS on same machine

[kashani]

On 8/17/2011 2:43 PM, Alan McKinnon wrote:
>
> I'm just itching to type up the long list of horror stories I've
> stored from people doing their own DNS thinking it was real easy.
>
> But there's this little thing called an NDA and it says I can't :-(

heh, I think I can dredge one up for you that no one will care about 
these days.

	This was at a large ISP in '99 known for their free Internet. Bind 8 
was fresh on the scene and somehow Network Engineering was in charge of 
DNS rather than Systems. My intern and I came up with a plan to have 
ns00.int as the internal master and make the rest of name servers slave 
off of it. All ns00 did was supply the production name servers with zones.

ns00 --> ns01(vip) --> ns01-[01-03]
     \--> ns02(vip) --> ns02-[01-03]
      \-> ns03(vip) --> ns03-[01-03]

Three virtual IPs and three name servers behind each vip.

This way we could have tools deal with updating zones on ns00 on the 
internal network and not have to push to a number of name servers. This 
worked well for a few months and we generally forgot about it. Almost a 
month after a reorganization in the local datacenter DNS went down. Well 
not down down, but our zones weren't working. After a hectic hour of 
freaking out, troubleshooting random things, and bouncing from machine 
to machine by IP address because none of DNS worked we realized our 
mistake. The TTL of the zone itself was set to three weeks. In the move 
Bind had silently died on ns00 which we didn't monitor because it was 
inside the corp network. The slaves dutifully stayed up and working till 
they hit the TTL of the zones and demanded to speak to the master again. 
Restarting Bind on the prod servers did nothing other than remove the 
already expired cache.
	Once restarted Bind on ns00 (and made it part of the runlevel) the prod 
server checked in and all was well.

The lessons:
	Monitor *all* of your DNS infrastructure
	DNS can break even with a large distributed system and it is never pretty.

kashani