After a frustrating experience with a Linksys WRT54GL, I've decided to
stick with Gentoo routers.  This increases the number of Gentoo
systems I'm responsible for and they're nearing double-digits.  What
can be done to make the management of multiple Gentoo systems easier?
I think identical hardware in each system would help a lot but I'm not
sure that's practical.  I need to put together a bunch of new
workstations and I'm thinking some sort of server/client arrangement
with the only Gentoo install being on the server could be appropriate.

- Grant

If I had a dozen or so rigs, I would want them to be identical at least
as far as mobos and such.  Drives and such could vary depending on other
things tho.  If you use the same USE flags, you could update one and
then copy /etc/portage/* to the other rigs then the updates could just
be binary installs from the first rig updated.

The only thing that might bite you is if you have different uses for
each rig.  If you have different packages installed, then you may have
to compile for that.  Most likely tho, you could use binaries for at
least the system packages and save some time.

I think Alan has some rigs and may be able to share more ideas.  Maybe
some others could comment that have dealt with the same thing.

Dale

:-)  :-)

In the case of editing identical files on different machines (Gentoo or
not), I like clusterssh. It opens an xterm for every destination host
and an additional command window. What you type here happens in all windows.

    Wonko

Out of curiosity, could you tell us more about this experience?

The WRT54G(L) is quite dated, and the OpenWRT devs recommend againsttrying to do anything fancy on it.

In another post you mentioned that you have a TP-Link TL-WR1043ND, whichis a bunch newer, I think, and should run OpenWRT quite well.

Even if you were to use an Atom-based Gentoo system, I'm sure it woulduse more power than an ARM- or MIPS-based commodity router - perhapstwice as much.

Stroller.

Grant (Sat, 2 Jul 2011 15:14:38 -0700):

I used to do it the rsync way. Dozens of machines with varying
hardware, although not profoundly. All machines had all the software any
single machine could need. CFLAGS, kernel config and such were filled
with the greatest common divisor of all the boxes. There was one
'reference box' which did all the compiling and from which all the
other machines would rsync /, minus selected variables, such
as /dev, /etc/mtab, /mnt, /proc, /var/log, etc.

Another set of things excluded from rsync were things that do not
handle themselves locally, but are different accross computers. I had
groups of computers that I wanted to, for example, run different set of
boot-up services. That means that /etc/runlevels was excluded from
rsync'ing, but what was being rsynced were the dirs
/etc/runlevels-group{1,2,3,...} and every machine had a local
symlink /etc/runlevels -> one of the runlevels-groupX. Administrative
tasks are still limited to working with the single reference box and
its single filesystem (plus a few more runlevels-group's and similar
exceptions).

That way you can have specialized runlevel layouts, specialized fstabs
and other configs, etc. Even specialized kernels for every group of
machines, in theory.

There was also a script being distributed that was called by local crons
and allowed for batch rebooting at midnight, or whatever you may need
to run locally.

You may want to check out lsyncd for keeping dirs in sync in a smart
way on-the-fly, but rsyncing from a cronjob at a safe moment may be the
recommended option, depending on your environment. By 'safe' I mean
when there are no users logged in, for example.

The most tricky part was fine-tuning the set of stuff to exclude
from rsyncing. But if sure can be done in reasonable environments.

HTH,
-rz

I've been managing 3 very very different gentoo systems for more than 12
months now.  I used to have 2 more machines but I decomissioned them for
they were old and noisy and it's summer and it's hot enough like that! haha

I do not have a lazy-proof way of administrating this whole stuff, but I've
come up with some kind of methodology that helps...

I found it really helps to have the same profile on all machines, same
/etc/portage everything.  The /etc/make.conf can vary, but I would make it
vary in the most minimal way.  For example, all my kernels and
/etc/make.conf are compiled for an i686, global use flags are exactly the
same... in fact the only difference is my host has extra make option "-j5"
and their distfiles mirrors will point to my own computers.  Basically,
downgrade all computers to the lowest common denominator of all your
computers.

I don't use binpkgs but I'm working on that...  and also for the world file,
I have 2 world file "styles", one has "everything" in it and the other is
just the minimal stuff.  I used to maintain the world files directly but
I've opted for creating 2 ebuilds in my own portage overlay which are
basically called sys-apps/everything and another called
sys-apps/minimalist.  Maintenance of world files moved to the maintenance of
these ebuilds.  Also, it helps to have your faster computer install all
packages that would be installed on the others.  If there's any trouble
doing the emerge on that one, you'll have the same problems on the others.
If the solution was to add some keywords in /etc/portage or something like
that, then you can copy the change over to your other PCs and their emerge
should be fine.

To reduce network load, I sync my master server (my server online) against
official sync server, then I sync my main computer at home against my
server, then I sync any other computers at home against the main computer at
home.

But the activity of making your systems up-to-date will vary in intensity
depending on how well it goes.  When things are fine, it used to take 30
minutes to get all my systems updated, with the slowest ones taking 2-3
times that but requiring no intervention until the end.

I'm currently in the process of writing a script that fully automates the
update process.  It basically tries things like "emerge -uDN --keep-going
world" and checks for errors, if there are, it then fires a few safe
"responses" which have helped in the past.  For example, if the above emerge
fails, it then tries a revdep-rebuild, an lafilefixer, python-updater and a
few more, and then it restart from scratch trying the emerge again, over and
over.  Rarely, this will not work and will require that I mask something,
add some keyword or update my ebuild to get rid of some package.  The script
is clearly not intelligent and is wasting a lot of time, but it eventually
gets it done.  Upgrades now take at least 1 hour, but are fully unattended.

Also, I think Wonko's suggestion is pretty good in combination with my
suggestions, I haven't tried but it does seem very attractive.

Simon

You can do much the same, but in a simpler way, by using sets.

--
Neil Bothwick

Master of all I survey (at the moment, empty pizza boxes)

I maintain multiple Gentoo we mostly use as KVM hosts systems (and
coming embedded routers). As KVM hosts, some of them are very sensible.
Due to the contracts to our customers, I have to do with various update
strategies on top of various hardware.

I've set up a private Gentoo mirror in order to follow updates nicely
(all customers want to update slowly). Well, it's not a true mirror. To
be able to upgrade old systems, I do "private" releases of Gentoo
approximately once a month. A full mirror of all releases would be too
much data. So, I only fetch portage tree and packages from a list I
maintain manually (emerge sucks at that game, by the way). Data is
stored on a nilfs filesystem to improve snapshots size on disk.

--
Nicolas Sebrecht

Thanks to everyone for some very juicy tidbits.  I'm rearranging my
thinking on all of this.  I think the key for me may be to combine
systems with separate functions in the same physical location into a
single system.  Does the KVM thing work well?  Running a bunch of
workstations as nothing more than wireless KVM setups on the same
system?  I should be able to cut my Gentoo systems down to just a few.
Basically one at each physical location.

- Grant

Sure, I was using the stock firmware and I didn't like that you
couldn't specify a source IP address when punching a hole in the
firewall for a particular port, and I also couldn't coax "Remote
Access" into working no matter what I tried.

I chose the WRT54GL because it has the best ratings on newegg.com.  I
looked into OpenWRT once but decided against it after I decided
installation and possibly management was not nearly as trivial as I
had imagined.

I got rid of the TL-WR1043ND a while back because I couldn't get
packet shaping to work with the stock firmware no matter how I tried.

At this point I've sworn off mystery boxes.  I even had a Dlink router
die on me recently.  If I'm not using mystery boxes for greater
hardware reliability, why am I using them?  Power consumption would be
a good reason but it's not worth it IMO.

- Grant

KVM itself works very well here, even with advanced features such as KSM
pages sharing.

The difficulties come with Microsoft products for both good integration
and perfomance (I would recommend RAW format, iSCSI or plain physical
have all working well for XP, NT2003 and 2008 servers.

I use libvirt on top of KVM which is in the way to become very good AFA
you don't rely on libvirt's API which tend to move a lot.

I would be much sceptical for both workstations and wireless guests than
for servers:

1) For workstations, things are currently changing with the very recent
and "not much usable with Gentoo, yet" spice software. I expect a lot of
improvments in the coming months for this use case. I would say it's not
ready for production, yet.

2) About wireless virtualization it's highly depending on what you aim
to do, especially if you intend to use the PCI passthrough feature to
give your wireless card to a guest. For this to work, you MUST have your
hardware (CPU, motherboard and PCI card) VT-d compatible which is
currently NOT a piece of cake, today. It relies on industry and
manufacturers moving not as fast as software. I would expect more widely
VT-d cards in the coming _years_.

Now, if you intend to use the wireless card from you hosts and share
networks using bridge utilities it _MAY_ be OK: Linux bridging does not
always work with all wireless cards (see  http://tinyurl.com/ylcutwv  for
more information).

In a more general approach, when I hear "routers" and "wireless" I'm
more thinking _embedded_. KVM/qemu would only help you to build your
target systems.

For embedded (or tiny, at least) systems, I would not use LXC.

The drawback with Gentoo is that the current official uclibc stage3 for
embedded/tiny systems is obsolete and marked as experimental. In facts,
it's very _hard_ if not impossible to use it these days. Making your own
cross-compilation environment is not a piece of cake (too), even with
dedicated tools such as crossdev. This topic would ask its own book.
So, if you want to try Gentoo embedded save your time by working on
unofficial stage3.

--
Nicolas Sebrecht

I think I'm guilty of assumption regarding your original reference to
KVM.  I assumed you mean keyboard-video-mouse:

http://en.wikipedia.org/wiki/KVM_switch

but now I think you meant Kernel-based Virtual Machine:

http://www.linux-kvm.org/page/Main_Page

And now that I look more closely at KVM switches, it looks like they
provide a method of controlling multiple computers via a single
keyboard, monitor, and mouse.  I need sort of the inverse.  I'd like
to control a single Gentoo computer via multiple sets of keyboards,
monitors, and mice simultaneously.  It would basically be a way to
have the functionality of multiple workstations but the administration
hassle of only a single system.  Wireless communication between the
computer and each keyboard-monitor-mouse would be most convenient, but
that may not be possible so wired would be fine.  Does something like
this exist?

- Grant

You may want to look at something like a config management system. I'm
using Puppet these days, but Gentoo support isn't spectacular. It would
be a bit complex to have Puppet install the packages with the correct
USE flags. However you could use Puppet to manage all the text files and
then manage the packages somewhat manually.

Here's a snippet of a template for nrpe.cfg

<% if processorcount.to_i >= 12 then -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
%>/check_load -w 35,25,25 -c 35,25,25
<% elsif fqdn =~ /(.*)stage|demo(.*)/ then -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
%>/check_load -w 10,10,10 -c 10,10,10
<% else -%>
command[check_load]=<%= scope.lookupvar('nrpe::params::pluginsdir')
%>/check_load -w 10,7,5 -c 10,7,5
<% end -%>

If you were managing a make.conf you could set -j<%= processorcount*2 %>
or whatever as well as pass in your own settings etc. Once you have
things working it's pretty good at keeping your servers in sync and
doing minor customization per server based on OS, hardware, IP,
hostname, etc.

kashani

On Thursday 07 July 2011 11:23:15 kashani did opine thusly:

Give chef a try.

It overcomes a lot of the issue puppet ran into, and of course makes
new ones all of it's won, but by and large chef is more flexible.

alan dot mckinnon at gmail dot com

Too late. I've already put a year in with Puppet and have too much
working code to switch. Also I'm not much of a programmer so I get a bit
more out of the DSL though my templates are getting fairly fancy these
days. For anyone else interested in what we're talking about, here's a
fairly balanced and up to date link talking about some of the differences.

http://redbluemagenta.com/2011/05/21/puppet-vs-chef/

kashani

On Thursday 07 July 2011 14:01:55 kashani did opine thusly:

At least with puppet you can still work around shortcomings as you
find them (no black box tricks in puttet)

But regardless of it's quality, it's still 1,000,000's of times better
than doing it all manually!

--
alan dot mckinnon at gmail dot com

Does this fantasy-arrangement of mine exist?  I guess what I'm after
is a series of dumb terminals to connect to a local Gentoo system so I
don't need to manage a series of Gentoo systems.

- Grant

Have you considered using PXE to network boot your systems? you can
have various configurations set up based on mac addresses to address
different hardware issues. I recommend trying out SystemRescueCD to
experiment with PXE booting for the client and server.
--
No trees were harmed in the sending of this message. However, a large
number of electrons were terribly inconvenienced.

Hello Grant,

You have similar goals as I do. In addition to what you are doing
I'm planning on managing thousands of embedded devices, remotely,
for controls purposes.

The new ARM-15 chip is suppose to be an "Intel Killer" in both
the server space and workstation space. It is also is going
to be the chip for 3D video and multi-head devices, such as
you purport to building in your other emails.

TI is very aggressive on the ARM-15 chips based mother boards.
Embedded Gentoo runs on the panda board, thanks to Armin76!

http://www.gentoo.org/proj/en/base/embedded/handbook/?part=4[..]

I'm not sure you can wait a few more months, but, in my research
the ARM-15 based devices are going to make significant inroads
into many areas.

http://www.slashgear.com/ti-omap-5-outed-twin-cortex-a15-cor[..]

hth,
James

That sounds like exactly what I need.  So, I could set up a Gentoo
server and a bunch of completely diskless clients which would all PXE
boot from the server?  Would the clients basically each control a
different virtual terminal on the server?

That's hilarious. :)

- Grant

Thanks James.  Would ARM-15 machines be a good match for PXE booting?
I'm thinking I just need something minimal so the ARM-15 might be a
great choice if I understand it correctly.  It wouldn't matter that it
runs Gentoo since my clients would be diskless, right?  I'm still
trying to get my mind around this.

- Grant

Each machine can pull a copy of the master boot image to make updates
a lot simpler. The SystemRescueCD PXE boot mechanism just pushes out a
copy of the CD to all the machines to boot them. to update the boot
image just update the files in one location to update all machines.
the machines act as separate fully functioning machine. Check out http://www.sysresccd.org/Sysresccd-manual-en_PXE_network_boo[..]  to
see how to setup the PXE boot environment.

Thanks. A friend shared that with me.

--
No trees were harmed in the sending of this message. However, a large
number of electrons were terribly inconvenienced.

I post some of Neil's sig lines to my wall on facebook.  A lot of them
are pretty neat.  What gets me tho is when one of them applies to the
topic he is replying too.  I know on a couple occasion Neil has even
mentioned it himself.

Keep em coming Neil.

Dale

:-)  :-)

I think I get it now and it sounds great, exactly what I'm looking for.

Everything can be done in RAM, no disks required?

Can PXE boot be done wirelessly?  Maybe only if the wireless is
onboard?  I tried to Google this but the info returned is terribly
outdated for some reason.

Do you think SystemRescueCD is the best boot image for clients that
only need a browser?

What sort of machine would work well as a client?  Should I just put
together a bunch of motherboards with onboard video and ethernet,
CPUs, RAM, PSUs, and small cases?  Is there a prebuilt system that
works well for this?  Maybe an ARM-15 system as "Tampa Bay" James
referenced, although I think that isn't released yet.

- Grant

Well, the first thing you need to decide is whether you want each
client running that browser locally, or whether you want each client
to merely provide an interface to the server, and every user's browser
(and every other application) running on the server itself. If your
clients boot, then run all their own software locally, your server's
under only under load during boot-time and your clients need to be
able to handle that work (not much, but it's more than nothing, just
try running a modern Firefox on 64MB of ram). On the other hand, if
your clients merely boot into a remote connection to the server, a la
VNC or NX, the client does *very* little locally, can run on next to
nothing hardware-wise (a true 'thin client'), and the entirety of the
workload is offloaded to the server. If you want responsive 'eye
candy', 3D graphics work/play, or any form of particularly 'smooth'
animation, you will want that work to be handled on hardware closer to
the user (requiring a far faster processor, more ram, a capable video
device, and likely local storage for swap at the least), while serving
up a simple browser to the user is far more forgiving.

As for wired vs wireless, true hardware PXE booting is generally
limited to wired scenarios, but it would be entirely possible (though
not truly 'diskless') to deploy a minimal kernel initramfs that
handles initial booting, joining WiFi, pulling down of the system
'image' from your server, and handing control off to that in the same
way your run of the mill kernel initramfs loads hardware drivers until
it can find the harddrive, attaches to the root partition, and hands
off control to init from there. Changes to the wireless configuration
would require directly visiting each client, and client-side kernel or
initramfs updates easily could as well, if things don't go as planned
(but, since all the user-side software is either run on the server or
loaded from it at boot-time, changes to the client's "loader"
shouldn't be frequent).

There's also the option of pre-made hardware thin clients that
typically boot from internal flash and simply provide a remote
interface to a central server (though most are geared towards RDP or
Citrix), and some are even WiFi capable.

--
Poison [BLX]
Joshua M. Murphy

After reading this, my first reaction was to run the browser on the
server and have each client connect via VNC/NX.  Now that I think
about it, I may be better off running the browser locally for
simplicity's sake.  I always try to keep the number of layers I'm
dealing with to a minimum and VNC/NX is one layer I could do without
if I beef up the clients a bit.  How different would the client
hardware requirements be between running the browser locally and
running it via VNC/NX?

I suppose I could also do without the PXE layer and all of its
requirements if I install some sort of minimal storage device (flash
drive, SD card, USB key, etc.) into each workstation for the boot
image.  I could still push updates to the boot image over the network
almost as easily as updating the single boot image on the server.

What is the benefit of loading SystemRescueCD instead of another
monolithic "just work" distro like Ubuntu?

It sounds like I should stick with ethernet for simplicity's sake.

A pre-made thin client could be the way to go.  Do you know of any
that are geared toward open protocols?

- Grant

Yeah, PXE on the wire is the place to start if you want to boot across
the network. Start simple. Just get a handful of similar NICs and you
should be set.

Quick query of "the oracle" yields:

http://www.thinlabs.com/products/thin-clients/aden

I have used AXEL thin client terminals and those require a VNC server
instance on your server per thin client, for the scenario that it sounds
like you're envisioning. It does RDP/VNC but you can get it to do
ssh/telnet on a green screen, with several sessions per seat.

We've been using Neoware thin clients for a few years now.  They run some HP
cooked debian version.

http://www.hp.com/sbso/busproducts_thinclient.html
--
Regards,
Mick