Re: [gentoo-dev] Gentoo LTS or: proper backward compatibility?

[Alec Warner]

On Mon, Jan 2, 2023 at 4:55 PM m1027 <m1027@××××××.net> wrote:
>
> Many thanks for your detailed thoughs for sharing the rich
> experiences on this! See below:
>
> antarus:
>
> > On Mon, Jan 2, 2023 at 4:48 AM m1027 <m1027@××××××.net> wrote:
> > >
> > > Hi and happy new year.
> > >
> > > When we create apps on Gentoo they become easily incompatible for
> > > older Gentoo systems in production where unattended remote world
> > > updates are risky. This is due to new glibc, openssl-3 etc.
> >
> > I wrote a very long reply, but I've removed most of it: I basically
> > have a few questions, and then some comments:
> >
> > I don't quite grasp your problem statement, so I will repeat what I
> > think it is and you can confirm / deny.
> >
> >   - Your devs build using gentoo synced against some recent tree, they
> > have recent packages, and they build some software that you deploy to
> > prod.
>
> Yes.
>
> >   - Your prod machines are running gentoo synced against some recent
> > tree, but not upgraded (maybe only glsa-check runs) and so they are
> > running 'old' packages because you are afraid to update them[0]
>
> Well, we did sync (without updading packages) in the past but today we
> even fear to sync against recent trees. Without going into details,
> as a rule of thumb, weekly or monthly sync + package updates work
> near to perfect. (It's cool to see what a good job emerge does on our
> own internal production systems.) Updating systems older than 12
> months or so may, however, be a hugh task. And too risky for remote
> production systems of customers.

My primary risk I think is that even if you ship your app in a
container you still need somewhere to run the containers. Currently
that is a fleet of different hardware and gentoo configurations, and
while containers certainly simplify your life there, they won't fix
all your problems. Now instead of worrying that upgrading your Gentoo
OS will break your app, it will instead break your container runtime.
It is likely a smaller surface area, but it is not zero. Not saying
don't use containers, just that there is no free lunch here
necessarily.

>
>
> >   - Your software builds OK in dev, but when you deploy it in prod it
> > breaks, because prod is really old, and your developments are using
> > packages that are too new.
>
> Exactly.
>
>
> > My main feedback here is:
> >  - Your "build" environment should be like prod. You said you didn't
> > want to build "developer VMs" but I am unsure why. For example I run
> > Ubuntu and I do all my gentoo development (admittedly very little
> > these days)
> >    in a systemd-nspawn container, and I have a few shell scripts to
> > mount everything and set it up (so it has a tree snapshot, some git
> > repos, some writable space etc.)
>
> Okay, yes. That is way (1) I mentioned in my OP. It works indeed but
> has the mentioned drawbacks: VMs and maintenance pile up, and for
> each developer. And you don't know when there is the moment to
> create a new VM. But yes it seems to me one of the ways to go:
> *Before* creating a production system you need to freeze portage,
> create dev VMs, and prevent updates on the VMs, too. (Freezing aka
> not updating has many disadvantages, of course.)

Oh sorry, I failed to understand you were doing that already. I agree
it's challenging, I think if you don't have a great method to simplify
here, it might not be a great avenue going forward.
 - Trying to figure out when you can make a new VM.
 - Trying to figure out when you can take a build and deploy it to a
customer safely.

I've seen folks try to group customers in some way to reduce the
number of prod artifacts required, but if you cannot it might be

The benefit of containers here is that you can basically deploy your
app at whatever rate you want, and only the OS upgrades remain risky
(because they might break the container runtime.)
Depending on your business needs, it might be advantageous to go that route.

>
>
> >  - Your "prod" environment is too risky to upgrade, and you have
> > difficulty crafting builds that run in every prod environment. I think
> > this is fixable by making a build environment more like the prod
> > environment.
> >     The challenge here is that if you have not done that (kept the
> > copies of ebuilds around, the distfiles, etc) it can be challenging to
> > "recreate" the existing older prod environments.
> >     But if you do the above thing (where devs build in a container)
> > and you can make that container like the prod environments, then you
> > can enable devs to build for the prod environment (in a container on
> > their local machine) and get the outcome you want.
>
> Not sure I got your point here. But yes, it comes down to what was
> said above.
>
>
> >  - Understand that not upgrading prod is like, to use a finance term,
> > picking up pennies in front of a steamroller. It's a great strategy,
> > but eventually you will actually *need* to upgrade something. Maybe
> > for a critical security issue, maybe for a feature. Having a build
> > environment that matches prod is good practice, you should do it, but
> > you should also really schedule maintenance for these prod nodes to
> > get them upgraded. (For physical machines, I've often seen businesses
> > just eat the risk and assume the machine will physically fail before
> > the steamroller comes, but this is less true with virtualized
> > environments that have longer real lifetimes.)
>
> Yes, haha, I agree. And yes, I totally ignored backporting security
> here, as well as the need that we might *require* a dependend
> package upgrade (e.g. to fix a known memory leak). I left that out
> for simlicity only.

Ahh my worry is that the easy parts are easy and the edge cases are
what really makes things intractable here.

>
>
> > > So, what we've thought of so far is:
> > >
> > > (1) Keeping outdated developer boxes around and compile there. We
> > > would freeze portage against accidental emerge sync by creating a
> > > git branch in /var/db/repos/gentoo. This feels hacky and requires a
> > > increating number of develper VMs. And sometimes we are hit by a
> > > silent incompatibility we were not aware of.
> >
> > In general when you build binaries for some target, you should build
> > on that target when possible. To me, this is the crux of your issue
> > (that you do not) and one of the main causes of your pain.
> > You will need to figure out a way to either:
> >  - Upgrade the older environments to new packages.
> >  - Build in copies of the older environments.
> >
> > I actually expect the second one to take 1-2 sprints (so like 1 engineer month?)
> >  - One sprint to make some scripts that makes a new production 'container'
> >  - One sprint to sort of integrate that container into your dev
> > workflow, so devs build in the container instead of what they build in
> > now.
> >
> > It might be more or less daunting depending on how many distinct
> > (unique?) prod environments you have (how many containers will you
> > actually need for good build coverage?), how experienced in Gentoo
> > your developers are, and how many artifacts from prod you have.
> >  - A few crazy ideas are like:
> >    - Snapshot an existing prod machine, strip of it machine-specific
> > bits, and use that as your container.
> >    - Use quickpkg to generate a bunch of bin pkgs from a prod machine,
> > use that to bootstrap a container.
> >    - Probably some other exciting ideas on the list ;)
>
> Thanks for the enthusiasm on it. ;-) Well:
>
> We cannot build (develop) on that exact target. Imagine hardware
> being sold to customers. They just want/need a software update of
> our app.
>
> And, unfortunatelly we don't have hardware clones of all the
> different customer's hardware at ours to build, test etc.

Ahh sorry, I meant mostly the software configuration here (my
apologies). It sounds like above you are doing that already and are
finding that keeping numerous software configurations (VMs) around is
too costly.
If that is the case it sounds like containers, flatpak, or snap
packages could be the way to go (the last one only if your prod
environment is systemd compatible.)

>
> So, we come back on the question how to have a solid LTS-like
> software OS / stack onto which newly compiled developer apps can be
> distributed and just work. And all this in Gentoo. :-)
>
>
> > > (2) Using Ubuntu LTS for production and Gentoo for development is
> > > hit by subtile libjpeg incompatibilites and such.
> >
> > I would advise, if possible, to make dev and prod as similar as
> > possible[1]. I'd be curious what blockers you think there are to this
> > pattern.
> > Remember that "dev" is not "whatever your devs are using" but is
> > ideally some maintained environment; segmented from their daily driver
> > computer (somehow).
>
> That is again VMs per "release" and per dev, right? See above "way
> (1)".

At a previous job we built some scripts to build a VM per release; but
in our scheme we only had to build 9 VMs worst case (3 targets, and 3
OS tracks, so 9 total.) We shared the base VM images (per release)
with the entire development team of 9 people. It was feasible with
decent internet (100mbit). We had some shared storage we put signed
images on. But often you would only use 1 image to test locally, then
push to the CI pipeline that would test on the other 8 images (because
it was cheaper / whatever in the datacenter.)

I continue to agree with you in that if you can't get your # of
targets down near that kind of number (10-20ish) it's probably not
going to be a great time for you.


>
>
> > > (3) Distributing apps as VMs or docker: Even those tools advance and
> > > become incompatible, right? And not suitable when for smaller Arm
> > > devices.
> >
> > I think if your apps are small and self-contained and easily rebuilt,
> > your (3) and (4) can be workable.
> >
> > If you need 1000 dependencies at runtime, your containers are going to
> > be expensive to build, expensive to maintain, you are gonna have to
> > build them often (for security issues), it can be challenging to
> > support incremental builds and incremental updates...you generally
> > want a clearer problem statement to adopt this pain. Two problem
> > statements that might be worth it are below ;)
> >
> > If you told me you had 100 different production environments, or
> > needed to support 12 different OSes, I'd tell you to use containers
> > (or similar)
> > If you told me you didn't control your production environment (because
> > users installed the software wherever) I'd tell you use containers (or
> > similar)
> >
> > >
> > > (4) Flatpak: No experience, does it work well?
> >
> > Flatpak is conceptually similar to your (3). I know you are basically
> > asking "does it work" and the answer is "probably", but see the other
> > questions for (3). I suspect it's less about "does it work" and more
> > about "is some container deployment thing really a great idea."
>
> Well thanks for your comments on containers and flatpak. It's
> motivating to investigate that further.
>
> Admittedly, we've been sticking to natively built apps for reasons
> that might not be relevant these days. (Hardware bound apps, bus
> systems etc, performance reasons on IoT like devices, no real
> experience in lean containers yet, only Qemu.)

Depending on your app, you can get pretty lean containers. We have a
golang app (https://gitweb.gentoo.org/sites/soko.git/tree/Dockerfile)
whose docker image is 39MB, but it mostly just has a large statically
compiled go-binary in it.  We run gitlab-ce in a large container that
is 2.5GB, so the sizes can definitely get large if you are not
careful.

Another potential issue for containers is the container runtime shares
a kernel with the host, so if your host kernel is very old, but you
need new kernel features (or syscalls) they may be missing on the host
kernel, so there are still some gotchas (but as mentioned, probably
fewer than you experience with a full OS build.)

Good Luck!

-A

>
>
> > Peter's comment about basically running your own fork of gentoo.git
> > and sort of 'importing the updates' is workable. Google did this for
> > debian testing (called project Rodete)[2]. I can't say it's a
> > particularly cheap solution (significant automation and testing
> > required) but I think as long as you are keeping up (I would advise
> > never falling more than 365d behind time.now() in your fork) then I
> > think it provides some benefits.
> >   - You control when you take updates.
> >   - You want to stay "close" to time.now() in the tree, since a
> > rolling distro is how things are tested.
> >   - This buys you 365d or so to fix any problem you find.
> >   - It nominally requires that you test against ::gentoo and
> > ::your-gentoo-fork, so you find problems in ::gentoo before they are
> > pulled into your fork, giving you a heads up that you need to put work
> > in.
>
> I haven't commented on Peter yet but yes I'll have a look on what he
> added. Something tells me that distributing apps in a container
> might be the cheaper way for us. We'll see.
>
>
> > [0] FWIW this is basically what #gentoo-infra does on our boxes and
> > it's terrible and I would not recommend it to most people in the
> > modern era. Upgrade your stuff regularly.
> > [1] When I was at Google we had a hilarious outage because someone
> > switched login managers (gdm vs kdm) and kdm had a different default
> > umask somehow? Anyway it resulted in a critical component having the
> > wrong permissions and it caused a massive outage (luckily we had
> > sufficient redundancy that it was not user visible) but it was one of
> > the scariest outages I had ever seen. I was in charge of investigating
> > (being on the dev OS team at the time) and it was definitely very
> > difficult to figure out "what changed" to produce the bad build. We
> > stopped building on developer workstations soon after, FWIW.
> > [2] https://cloud.google.com/blog/topics/developers-practitioners/how-google-got-to-rolling-linux-releases-for-desktops
>
> Thanks for sharing! Very interesting insights.
>
> To sum up:
>
> You described interesting ways to create and control own releases of
> Gentoo. So production and developer systems could be aligned on
> that. The effort depends.
>
> Another way is containers.
>
>