1 |
Alan McKinnon writes: |
2 |
|
3 |
> On Saturday 14 November 2009 19:36:06 Alex Schuster wrote: |
4 |
>> Alan McKinnon wrote: |
5 |
|
6 |
>>> clusterssh will let you log into many machines at once and run emerge |
7 |
>>> -avuND world everywhere |
8 |
>> This is way cool. I just started using it on eight Fedora servers I am |
9 |
>> administrating. Nice, now this is an improvement over my 'for $h in |
10 |
>> $HOSTS; do ssh $h "yum install foo"; done' approach. |
11 |
> |
12 |
> I feel your pain :-) |
13 |
> |
14 |
> We used to have the same problem adding new admins to 87 machines. Now |
15 |
> we have a bespoke provisioner that does it all. |
16 |
|
17 |
Sorry, I just do not get 'bespoke provisioner'. Some sort of software, |
18 |
like clusterssh? Or a person, one admin instead of many? |
19 |
|
20 |
|
21 |
>> What do you guys think about using Gentoo for servers? At the institute |
22 |
>> I partially work we chose Fedora. There is no special reason for that - |
23 |
>> we already had some Fedora machines, the setup seemed to work, the |
24 |
>> reputation was good, so we kept it. That was okay for me, why choose |
25 |
>> many different environments and learn everything again. I mentioned |
26 |
>> Gentoo, but did not really suggest to actually use it. Maybe I should |
27 |
>> have. |
28 |
> |
29 |
> I'm a huge fan of Gentoo |
30 |
|
31 |
Now who would have thought of that! |
32 |
|
33 |
> and all my personal machines (except the new netbook have run it for the |
34 |
> last 5 years. |
35 |
> |
36 |
> But I will never install Gentoo on a production server at work. |
37 |
> |
38 |
> Why? |
39 |
> |
40 |
> Because it is too time consuming, because no two machines are set up the |
41 |
> same, because I can't trust that other admins used the flags they should |
42 |
> have. So updates become a case of logging into 80+ machines individually |
43 |
> and doing emerge world by hand. Gentoo allows you to customize things to |
44 |
> the nth degree - that is it's strength - so people WILL use this one |
45 |
> discriminating factor. |
46 |
> |
47 |
> If OTOH I had a server farm of 80+ machines, all identical, I'd put |
48 |
> Gentoo on them in a flash. But I don't have that |
49 |
|
50 |
Of our 8 machines, 7 are essentially the same and differ only in hard |
51 |
drive space and CPU speed. The other machine is Intel, not AMD, and needs |
52 |
different IDE drivers. At the moment it has a different initrd (I set up a |
53 |
minimal fedora install to generate it after the cloned system did not |
54 |
boot), the rest is - apart from some config files - identical. |
55 |
|
56 |
So I would make sure that about everything is exactly the same, well, |
57 |
maybe except for hostnames, udev net-persistent-rules, ssh keys... what |
58 |
more? |
59 |
The last, a little different machine is a problem though. With optimized |
60 |
CFLAGS, this one would have to compile all stuff again, while for the |
61 |
others I could use binpkgs. Updating them all with clusterssh should not |
62 |
be much more work than updating a single one. Well, not completely true, I |
63 |
would have the double work, as I would upgrade one server first to test if |
64 |
there are problems, and then do it for the others. Maybe I could use the |
65 |
special machine to test stuff, and then update all the others. |
66 |
|
67 |
If they would differ, Gentoo would of course be too much work. I already |
68 |
have this problem now... there is my desktop machine, my notebook running |
69 |
a Gentoo VM, a second desktop machine at my other home, the living-room |
70 |
machine of my flat share, the machine of a fried I also administrate, the |
71 |
server of my flat share I need to set up again... and clusterssh is no |
72 |
option here. |
73 |
|
74 |
|
75 |
>> Now I am thinking about a Gentoo installation instead. |
76 |
>> |
77 |
>> Pros: |
78 |
>> - Continuous updates, no downtime for upgrading, only when I decide to |
79 |
>> install a new kernel. This is really really cool. I fear the upgrade |
80 |
>> from Fedora 10 to 12 which has to be done soon. |
81 |
> |
82 |
> Do not upgrade, especially not with a version jump of 2 or more. If you |
83 |
> have a lot of machines, I assume you are a decent shop, and that you |
84 |
> have some form of formal process for upgrades and changes. |
85 |
|
86 |
Not really, I think. We are not very professional I must admit. We have |
87 |
two capable admins, but one is specialized in network stuff and Windows, |
88 |
the other has to do with our big Sun servers, huuge storage systems and |
89 |
such. They do not much about the Linux cluster. Another user sometimes |
90 |
installs a package on a machine, but usually I do this. For me, it is not |
91 |
my main job, I work only about ten hours per week there, mostly being some |
92 |
100 km away. |
93 |
We are a research institute. We do neurological research, PET and MRI |
94 |
tomography. The Linux servers do number crunching, and of course they |
95 |
should work and have good uptimes, but it is not as important as if we |
96 |
were an ISP. |
97 |
|
98 |
> What you do instead is a formal migration - copy the data off, |
99 |
> reinstall, restore data. |
100 |
|
101 |
Advice noted. Yes, this sounds like the better idea, giving a cleaner |
102 |
setup. And if some things break I do not have to wonder if it was some |
103 |
strange side effect from the upgrade process. |
104 |
|
105 |
> If you can't afford to do that every six or twleve months, then |
106 |
> I have to ask - what the hell is the organization doing using a distro |
107 |
> that is unsupported after 12 months? |
108 |
|
109 |
Well, I do not think this was considered much. One machine was set up with |
110 |
Fedora for no specific reason, and we kept this distro then. This does not |
111 |
sound too professional, I know. BTW, what distro would you suggest? |
112 |
|
113 |
>> - Some improvement in speed. Those machines do A LOT of |
114 |
>> numbercrunching, which jobs often lasting for days, so even small |
115 |
>> improvements would be nice. |
116 |
> |
117 |
> Don't fool yourself. Unless you need what Google needs, there is very |
118 |
> little speed difference between Gentoo and Fedora. I/O improvements you |
119 |
> need can be easily gotten by fiddling the kernel tuning knobs. |
120 |
|
121 |
I know the difference will not be huge, I see this as a little bonus - |
122 |
nice if is there, but nothing really important. But in the comparison with |
123 |
Ubuntu that came in a thread a few weeks ago, for some applications the |
124 |
speed increase was about 30 percent. Although I would not necessarily |
125 |
expect the difference to be noticeable, I would also not be surprised too |
126 |
much if it were noticeable for some number-crunching applications if they |
127 |
were optimized for the CPU. |
128 |
|
129 |
>> - Easier debugging. When things do not work, I think it's easier to |
130 |
>> dig into the problem. No fancy, but sometimes buggy GUIs hiding basic |
131 |
>> functionality. |
132 |
> |
133 |
> Errrrrrrrrrrrrrrrmmmmmmmmmmmmmm, Fedora does not require a GUI :-) |
134 |
|
135 |
Right, and now that I think of it I do not use it anyway... Well, I did do |
136 |
some things with netsetup (or whatever it's called), now that I know the |
137 |
system a little better I edit things directly in /etc/sysconfig. |
138 |
But the installer is a GUI, right? And if I remember this correctly, I |
139 |
cannot even switch to a text console and do stuff there while installing. |
140 |
Or I could, but did not have utilities like LVM. Something like that. I |
141 |
have to use the installer and its capabilities. |
142 |
|
143 |
>> - Heck, Gentoo is _cooler_ than typical distributions. And emerging |
144 |
>> with distcc on about 8*4 cores would be fun :) |
145 |
> |
146 |
> Can't argue with that. |
147 |
> |
148 |
> But that is your ego talking and the machines do not belong to you but |
149 |
> to the institute. Your ego has no place in that. |
150 |
|
151 |
You're right, thanks for the reminder. But also note the smiley. I know my |
152 |
boss (who is also into geeky things) would also like this - as long as it |
153 |
would work. |
154 |
|
155 |
>> - I am probably the only one who can administrate them. |
156 |
> |
157 |
> This is not a benefit. It is a severe liability. |
158 |
|
159 |
That's why I listed it also on the contra side. Forgot to add a smiley |
160 |
here, it was not meant seriously. |
161 |
But when I think about it... the others also do not know much about |
162 |
Fedora. Not even I do this well. There you use 'yum install <package>', |
163 |
with Gentoo it's 'emerge <package>'. Daily work would be similar. |
164 |
Upgrades would be a different thing, though. Gentoo's portage blockers |
165 |
would not be understood easily, they would prefer to take the servers down |
166 |
and just install the current Fedora distro. Which hopefully would work. |
167 |
|
168 |
|
169 |
>> Cons: |
170 |
>> - If something will not work with this not so common |
171 |
>> (meta)distribution, people will say "always trouble with your Gentoo |
172 |
>> Schmentoo, it works fine in Fedora". Fedora is more mainstream, if |
173 |
>> something does not work there, then it's okay for the people to accept |
174 |
>> it. |
175 |
> |
176 |
> Those same people are likely to say the same about linux vs windows. |
177 |
|
178 |
Right, but we already have Linux, and we need it for our software. Gentoo |
179 |
would not really be needed. |
180 |
|
181 |
>> - I am probably the only one who can administrate them. I think Gentoo |
182 |
>> is easier to maintain in the long run, but only when you take the time |
183 |
>> to learn it. With Fedora, you do not need much more than the 'yum |
184 |
>> install' command. There is no need to read complicated X.org upgrade |
185 |
>> guides and such. |
186 |
>> |
187 |
>> I think I already made my decision, but I am still interested in your |
188 |
>> opinions, maybe some of you are in a similar position and like to share |
189 |
>> your experiences. Whether I will be allowed to use Gentoo is another |
190 |
>> question, I guess my boss will not like my idea at first, and I am not |
191 |
>> even sure if he is right. But maybe I can test-install Gentoo on one |
192 |
>> machine in a chroot, and see if things work fine. |
193 |
> |
194 |
> Depends how critical these machines are. If you want to change them just |
195 |
> because you feel like it, then I do not see how that can possibly be a |
196 |
> valid reason. |
197 |
> |
198 |
> Remember, the institute's needs and desires trump yours every time |
199 |
|
200 |
No, it's not just because I feel like it. The main advantages would be: |
201 |
- No downtime between upgrades. Our jobs run for several days, every |
202 |
downtime has to be planned in advance. People understand this, but they do |
203 |
not like it. They would be very happy if this were not longer necessary. |
204 |
And I would not fear that during the upgrade something breaks, and it |
205 |
would take me long to fix it. |
206 |
- I know this distro well, and this is not at all true about Fedora. I |
207 |
know how to fix problems, I know how things work here. I would feel better |
208 |
with Gentoo, more competent. It just does not feel so well to administrate |
209 |
Fedora. |
210 |
|
211 |
Thanks for your opinions, Alan. As always. |
212 |
|
213 |
Wonko |