Re: [gentoo-dev] Not-Forking 0.3.1 with ebuild

[Dan Shearer]

Gordon,

Le mar. 26 janv. 2021 à 00:40, Gordon Pettey <petteyg359@×××××.com> a écrit :

> A bit off topic to whether or not there's an ebuild/maintainer for it

As to relevance to gentoo-dev:

1. Not-Forking is a package that general Gentoo users may want,
especially those who do embedded development or who maintain a suite
of apps.
    The ebuild supplied with Not-Forking works today, but as a
non-Gentoo dev, I recognise that is not the same as having a
Maintainer or a Proxied Maintainer :-)

2. The problem Not-Forking addresses is one the Gentoo distribution
has at scale, and neither Git or any other other VCS can solve
unassisted and
    in a general way. Part of this is what Debian calls vendoring[0].

> Instead of using the built-in features of Git like submodules and rebasing, depend on an external tool
> to make your repository even more messy?

Git cannot address the Not-Forking class of problems because the
decisions required are typically made by humans doing a
port or reimplementation where multiple upstreams need to be combined.
A patch stream can't describe what needs to be done,
so automating this requires a tangle of fragile one-off code.
Not-Forking makes it possible to write a build system without these
code tangles.

Examples of actions Not-Forking can take:

* check for new versions of all upstreams, doing comparisons of the
human-readable release numbers rather than repo checkins or tags,
  where human-readable version numbers vary widely in their construction
* replace foo.c with bar.c in all cases (perhaps because we want to
replace a library that has an identical API with a safer
implementation)
* apply this patch to main.c of Upstream 0, but only in the case where
we are also pulling in upstream1.c, but not if we are also using
upstream2.c
* apply these non-patch changes to Upstream 0 main.c in the style of
`sed` rather than `patch`, making it possible to merge trees that a
VCS says are unmergable
* build with upstream1.c version 2, and upstream3.c version 3, both of
which are ported to upstream 0's main.c version 5
* track changes in all upstreams, which may use arbitrary release
mechanisms (Git, tarball, Fossil, other)
* cache all versions of all upstreams, so that a build system can step
through a large matrix of versions of code quickly, perhaps for
test/benchmark

I can best illustrate this with LumoSQL [1]. The point of LumoSQL is
to take the SQLite source code in one hand and up to five (currently
two)
existing key-value pair storage backends on the other, and make them
work together. LumoSQL has not developed a new version of SQLite,
and doesn't fork SQLite or the upstream key-value storage systems.
This is despite the fact SQlite was never intended to have pluggable
backends,
and none of the key-value pair storage systems were designed to be
compatible or to work with SQLite.

Thanks to Not-Forking, LumoSQL mostly consists of a build system[2] .
A single `make` command can generate a binary with:

* several versions of SQLite
* any or all of up to twenty versions of LMDB
* one version of BDB

in many combinations, with some combinations skipped because they
can't exist. Now that the LumoSQL build system exists, we can focus on
expanding the
range of versions and backends, confident that any manual porting work
can be turned into Not-Forking configuration items.

Now, you might ask "why doesn't SQLite have a backend storage API", to
which there are two answers:

a. Since it is probably the most-deployed software in the world with
billions of human end users and many billions more instances, SQLite
can do whatever it likes and other software just needs to accommodate
that. As it happens SQLite chooses to be extremely careful and
conservative about user-visible changes, especially the C API and
on-disk storage format. There are various issues with that, and SQLite
features and internals do change quite a bit, however, SQLite is very
unlikely to adopt a funky new architecture because it would be neater.
SQLite is intended to be forwards-compatible for decades, more or
less.

b. Despite (a) above, LumoSQL is developing a generic backend for
key-value stores to SQLite, and Richard Hipp has indicated he will
consider minimally-invasive patches to that end if they seem to be
generally helpful. So LumoSQL has committed to definitely not forking
SQLite (there are already countless SQLite forks in the world).

And that's why Not-Forking exists.

-- 
Dan Shearer
dan@×××××××.org

[0] https://lwn.net/Articles/836911/
                                  Debian vendoring discussion
[1] https://lumosql.org/src/lumosql
                                 LumoSQL home page
[2] https://lumosql.org/src/lumosql/doc/trunk/doc/lumo-build-benchmark.md
   Detailed description of LumoSQL meta-build system