[gentoo-dev] verifying commits via server-side git pre-receive hook

[gentoo-dev] verifying commits via server-side git pre-receive hook

[Tim Harder]

Hi all,

I've been working for a while on pkgcraft[1] and related tooling that
could have the potential to run various CI checks server-side as a
pre-receive git hook verifying commits. Previously, this wasn't at all
possible due to portage/repoman and pkgcore/pkgcheck being too slow to
feasibly run and give feedback within a second or two of interactively
pushing.

For more information on benchmarks, see the related sections in the
discussions of pkgcraft's metadata generation[2] and pkgcruft[3]
comparing them to similar functionality from portage and pkgcore. Also,
note that a majority of commits only involve scanning a limited number
of packages which is much faster than scanning the entire tree.

However, even with all that making it semi-feasible for most commits
(except perhaps those doing tree-wide changes or altering eclasses with
extensive usage) much work has to be done on the infra and git-hook
service side. First off, the server-side git repo is probably hosted in
a bare format so the git hook would need to notify a scanning service
that runs remotely on a relatively powerful machine.

In terms of service design, my high-level thoughts are as follows: the
scanning service tracks repo changes using some copy-on-write solution
that supports snapshots such as lvm2 or btrfs on top of a fast, backing
I/O solution (direct NVME or tmpfs). This would be leveraged to keep
track of the most recent tree state with metadata layered in. As commits
come in they would be put on a queue, from which the most recent would
be pulled, checked out on top of the most recent snapshot, tested, and
merged back into the most recent snapshot when scanning passes.

Alternatively, Gentoo could officially move to a merge queue approach
which would alleviate a lot of this complexity, but also bring up
questions of how merge commits and signing would work among other
issues.

 From all this, my main questions to the dev community are the following:

- Do you see value in running a service that can reject commits due to
   issues like invalid metadata during `git push`? 

- Would the project be open to moving to a merge queue model?

- Are there others who would be interested in helping with the
   development, testing, and maintenance of a git-hook service if that is
   the chosen path?

Thanks,
Tim

[1]: https://github.com/pkgcraft
[2]: https://pkgcraft.github.io/posts/metadata-cache-generation/
[3]: https://pkgcraft.github.io/posts/evolving-qa-tooling/

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1557 bytes --]

On 11/25/24 12:15 PM, Tim Harder wrote:
> Hi all,
> 
> I've been working for a while on pkgcraft[1] and related tooling that
> could have the potential to run various CI checks server-side as a
> pre-receive git hook verifying commits. Previously, this wasn't at all
> possible due to portage/repoman and pkgcore/pkgcheck being too slow to
> feasibly run and give feedback within a second or two of interactively
> pushing.
> 
> [...]
> 
> From all this, my main questions to the dev community are the following:
> 
> - Do you see value in running a service that can reject commits due to
>   issues like invalid metadata during `git push`?


Yes please! This would be fantastic to have. Already, pushing is not
quite instantaneous since the hooks wait on e.g. pushing again to
mirrors, and I don't think waiting a couple seconds for significantly
greater peace of mind is at all a bad thing.


> - Would the project be open to moving to a merge queue model?


At least for me, no. I don't think it's necessary as we can already see
huge reliability benefits without it, and the merge commit / signing
issues are what I'd personally consider to be an instant dealbreaker.

So I'd say there's a lot to lose and not much to gain.


> - Are there others who would be interested in helping with the
>   development, testing, and maintenance of a git-hook service if that is
>   the chosen path?


I'd be happy at minimum to help test it, although in principle I'm not
uninterested in the rest.


-- 
Eli Schwartz

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Matt Jolly]

Hi,

Sorry for the delayed reply, I was reminded of this email and
realised that I never did respond to it while we were discussing
git hooks today.

On 13/3/25 06:42, Eli Schwartz wrote:
> On 11/25/24 12:15 PM, Tim Harder wrote:
>>  From all this, my main questions to the dev community are the following:
>>
>> - Do you see value in running a service that can reject commits due to
>>    issues like invalid metadata during `git push`?
> 
> 
> Yes please! This would be fantastic to have. Already, pushing is not
> quite instantaneous since the hooks wait on e.g. pushing again to
> mirrors, and I don't think waiting a couple seconds for significantly
> greater peace of mind is at all a bad thing.
I spent quite a bit of time fixing various QA issues (including tons of
whitespace) reported by pkgcruft today, and the only thing better than
fixing QA issues is having them not make it into the repos in the first
place!

Would you be open to trialling this on GURU? It seems like a reasonable
target where it'd seriously reduce the amount of developer / trusted
contributor overhead when merging commits.

If so, what would that process look like, and how would we go about
getting the implementation started?

Thanks for your work on QA tools so far!

Cheers,

Matt

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Tim Harder]

For anyone interested,

I've hacked up an initial, rough implementation of a pkgcruft-git
service that enables verifying ebuild commits during git's pre-receive
hook phase.

Currently it comes with a simple script that runs a local demo instance
targeting the gentoo repo by default.

You can find the code and demo instructions in the pkgcruft-git crate
directory of the pkgcraft repo [1].

If I can find the time, I'll try to write up a dev blog post about the
service design in more detail as well as next steps if there is interest
in trying to use it in any official fashion in the future.

Thanks,
Tim

[1]: https://github.com/pkgcraft/pkgcraft/tree/main/crates/pkgcruft-git

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Tim Harder]

On 2025-05-21 Wed 03:57, Matt Jolly wrote:
>Would you be open to trialling this on GURU? It seems like a reasonable
>target where it'd seriously reduce the amount of developer / trusted
>contributor overhead when merging commits.

It depends on how the infra side of GURU is set up. If it's semi-similar
to how the gentoo repo is handled I imagine it could be done, but note
that this service requires a decent amount of power behind it so the
latency between pushing commits and getting them accepted is minimal.
While it is quite a bit faster than repoman or pkgcheck, it still relies
a lot on bash which isn't known for speed in general. I'm not sure what
infra is able to provide in terms of power to make it feasible or not.

>If so, what would that process look like, and how would we go about
>getting the implementation started?

I've already hacked up a rough proof-of-concept (see my other recent
reply to this thread) that can be found at [1].

Thanks,
Tim

[1]: https://github.com/pkgcraft/pkgcraft/tree/main/crates/pkgcruft-git

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Luca Barbato]

On 30/05/25 10:46, Tim Harder wrote:
> For anyone interested,
> 
> I've hacked up an initial, rough implementation of a pkgcruft-git
> service that enables verifying ebuild commits during git's pre-receive
> hook phase.
> 
> Currently it comes with a simple script that runs a local demo instance
> targeting the gentoo repo by default.
> 
> You can find the code and demo instructions in the pkgcruft-git crate
> directory of the pkgcraft repo [1].
> 
> If I can find the time, I'll try to write up a dev blog post about the
> service design in more detail as well as next steps if there is interest
> in trying to use it in any official fashion in the future.
> 
> Thanks,
> Tim
> 
> [1]: https://github.com/pkgcraft/pkgcraft/tree/main/crates/pkgcruft-git
> 

It would be very nice to have it run over the PRs we are already 
receiving on github, personally I prefer a rebase-only workflow so merge 
queue that imply actual git merges with all the problems they bring on 
bisecting and reading the history are an hard no for me.

lu

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Tim Harder]

On 2025-05-30 Fri 09:23, Luca Barbato wrote:
>It would be very nice to have it run over the PRs we are already 
>receiving on github...

This requires different functionality than what is done for pre-receive
hook support more in line with what the pkgcruft github action[1]
already provides.

Of course, if it was integrated with pkgcruft-git it could be more
efficient to support features such as scan diffing, i.e. only showing
new results introduced by a PR, which the action already provides using
cache files.

On the other hand, the core functionality of pkgcraft and pkgcruft are
probably efficient enough to run directly on github CI for that task in
a reasonable amount of time if limited to the target packages assuming
the reports snapshot required for diffing is requested from an external
pkgcruft-git service.

In either case, before adding to the ever-growing mountain of potential
work, I'd prefer to see if the base git hook functionality is useful in
any fashion or what else it needs to become so.

Thanks,
Tim

[1]: https://github.com/pkgcraft/pkgcruft-action

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Sam James]

Tim Harder <radhermit@gentoo.org> writes:

> For anyone interested,
>
> I've hacked up an initial, rough implementation of a pkgcruft-git
> service that enables verifying ebuild commits during git's pre-receive
> hook phase.
>
> Currently it comes with a simple script that runs a local demo instance
> targeting the gentoo repo by default.
>
> You can find the code and demo instructions in the pkgcruft-git crate
> directory of the pkgcraft repo [1].

I've packaged this now as dev-vcs/pkgcruft-git (live-only for now). I'm
using the pre-push support you added (thank you!) locally with the
following in .git/hooks/pre-push:

    pkgcruft-git-pre-push origin git+ssh://git@git.gentoo.org/repo/gentoo.git

and going from there. I encourage others to give it a go and share
feedback too.
>
> If I can find the time, I'll try to write up a dev blog post about the
> service design in more detail as well as next steps if there is interest
> in trying to use it in any official fashion in the future.

Definitely. My hope is we can get this deployed on infra, but I think
it's going to be really valuable even without that locally.

Great work!

>
> Thanks,
> Tim
>
> [1]: https://github.com/pkgcraft/pkgcraft/tree/main/crates/pkgcruft-git

thanks,
sam

Re: [gentoo-dev] verifying commits via server-side git pre-receive hook

[Tim Harder]

On 2025-06-02 Mon 21:57, Sam James wrote:
>I've packaged this now as dev-vcs/pkgcruft-git (live-only for now). I'm
>using the pre-push support you added (thank you!) locally with the
>following in .git/hooks/pre-push:
>
>    pkgcruft-git-pre-push origin git+ssh://git@git.gentoo.org/repo/gentoo.git
>
>and going from there. I encourage others to give it a go and share
>feedback too.

I think it should also be possible to just symlink the binary into the
pre-push hooks dir or as the pre-push file itself since that's how I was
testing it.

>> If I can find the time, I'll try to write up a dev blog post about the
>> service design in more detail as well as next steps if there is interest
>> in trying to use it in any official fashion in the future.
>
>Definitely. My hope is we can get this deployed on infra, but I think
>it's going to be really valuable even without that locally.

If infra (or anyone else) needs help, has questions, or requests... feel
free to ping me or open discussions/issues upstream. I won't always have
as much availability as I currently do for the next couple weeks, but
I'll try to respond when I can.

Thanks,
Tim