* [gentoo-dev] [RFC] New categories for LLVM
@ 2024-12-07 16:07 Michał Górny
2024-12-08 4:11 ` Sam James
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: Michał Górny @ 2024-12-07 16:07 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]
Hello,
Given that the number of LLVM packages is growing, and probably will
grow again (I'm introducing "offload" right now, expect at least MLIR
soon, there are open requests for flang, polly...), I'd like to propose
creating dedicated categories for these packages and moving them there.
If not anything else, this will help consistently applying flags
and keywords to these packages (`/etc/portage/package.*` accept
wildcards).
My initial idea would be to use two categories: one for the toolchain
packages, another for runtimes, e.g.:
llvm-core/
clang
clang-common
clang-runtime
clang-toolchain-symlinks
lld
lld-toolchain-symlinks
lldb
llvm
llvm-common
llvm-toolchain-symlinks
llvmgold
llvm-runtimes/
compiler-rt
compiler-rt-sanitizers
libclc
libcxx
libcxxabi
libomp (-> openmp?)
llvm-offload (-> offload)
llvm-unwind (-> unwind?)
clang-python, lit and llvm-ocaml would remain in their language
categories.
WDYT?
--
Best regards,
Michał Górny
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] [RFC] New categories for LLVM
2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny
@ 2024-12-08 4:11 ` Sam James
2024-12-08 13:02 ` Michał Górny
2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James
2024-12-08 15:44 ` [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories Michał Górny
2 siblings, 1 reply; 11+ messages in thread
From: Sam James @ 2024-12-08 4:11 UTC (permalink / raw
To: Michał Górny; +Cc: gentoo-dev
Michał Górny <mgorny@gentoo.org> writes:
> Hello,
>
> Given that the number of LLVM packages is growing, and probably will
> grow again (I'm introducing "offload" right now, expect at least MLIR
> soon, there are open requests for flang, polly...), I'd like to propose
> creating dedicated categories for these packages and moving them there.
>
> If not anything else, this will help consistently applying flags
> and keywords to these packages (`/etc/portage/package.*` accept
> wildcards).
I quite like the idea of having a category to ease keywording and testing.
>
> My initial idea would be to use two categories: one for the toolchain
> packages, another for runtimes, e.g.:
>
> llvm-core/
> clang
> clang-common
> clang-runtime
> clang-toolchain-symlinks
> lld
> lld-toolchain-symlinks
> lldb
> llvm
> llvm-common
> llvm-toolchain-symlinks
> llvmgold
>
> llvm-runtimes/
> compiler-rt
> compiler-rt-sanitizers
> libclc
> libcxx
> libcxxabi
> libomp (-> openmp?)
> llvm-offload (-> offload)
> llvm-unwind (-> unwind?)
>
I'm not sure if I'm sold on *two*. What happens for stuff like mlir
where it's not a runtime but it's arguably more of one than core?
It just doesn't feel like the division works great. Or maybe it's just
because I feel like llvm-core will keep growing and llvm-runtimes won't.
> clang-python, lit and llvm-ocaml would remain in their language
> categories.
>
> WDYT?
I'm going to start another reply for a subthread.
thanks,
sam
^ permalink raw reply [flat|nested] 11+ messages in thread
* LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM)
2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny
2024-12-08 4:11 ` Sam James
@ 2024-12-08 4:53 ` Sam James
2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James
` (2 more replies)
2024-12-08 15:44 ` [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories Michał Górny
2 siblings, 3 replies; 11+ messages in thread
From: Sam James @ 2024-12-08 4:53 UTC (permalink / raw
To: Michał Górny; +Cc: gentoo-dev
Michał Górny <mgorny@gentoo.org> writes:
> Hello,
>
> Given that the number of LLVM packages is growing, and probably will
> grow again (I'm introducing "offload" right now, expect at least MLIR
> soon, there are open requests for flang, polly...), I'd like to propose
> creating dedicated categories for these packages and moving them there.
>
> [...]
>
> WDYT?
I fear this sort of assumes we won't switch to monobuild any time soon.
I keep thinking [0] about how sustainable our current setup is:
* Fedora moved away from it for >=18 [1].
* As we saw with offload, it broke a few times in just a week. So it's
only Gentoo who cares about this configuration AFAIK.
* It's not working great if we're already not able to easily package
mlir, flang, or polly.
Violet attempted working on a merge before [2].
The trade-offs I see are:
* Increased disk space requirement for building LLVM
I think that's a legitimate concern although perhaps not that big of a
deal given it is, after all, a whole toolchain. GCC takes quite a bit
to build too.
* Build time
Build time for mono LLVM would increase as we're building more
components at least for some users.
But the time added by
building more components (see below) should be balanced out by ccache if
doing development and also, importantly, not needing to force on all
targets anymore (they keep growing).
The cumulative time should be the same (although maybe a bit less
given the targets change) given that most people need the
same set of components because of mesa, firefox, or other things which
need libclang.
* Moving to an upstream supported-configuration
We may have a bit of pain in moving to it and getting used to it, but
we're no longer swimming upstream and being the only people caring
about our choice of build configuration (and regularly having to
explain and justify it to upstream).
Upstream also recommend doing a bootstrap build. We don't and can't do
that right now.
Folks upstream state at every opportunity that they don't care about
standalone builds,
e.g. https://discourse.llvm.org/t/rfc-do-something-with-the-subproject-tarballs-in-the-release-page/75024/2.
* Better support for LLVM as a system toolchain
The current setup doesn't work well for people using LLVM as a system
toolchain (because some of the components *must* be upgraded together),
it doesn't work well for people who want to use mlir/flang/polly, and it
doesn't work well for users on constrained hardware because we have to
force on all targets. It also prohibits more optimisation, PGO, and
bootstrapping it to test reliability.
(This is why I'm not too sympathetic to claims that the monobuild is
mostly for binary distributions, because we're actually *more*
vulnerable to issues as a result of it being split when building from
source if using the LLVM toolchain.)
* Maintaining older LLVMs
It's easier for older LLVMs to be either maintained by somebody else
(for e.g. Rust's purposes) or in an overlay if it's a single package
rather than many.
This is also true for e.g. testing old snapshots or keeping binary
packages to downgrade to once they got cleaned up.
At the moment, I fear us choosing the non-recommended path gives us very
little, and causes a bunch of problems in return.
I don't particularly care much personally about using LLVM as the system
toolchain and if that's the only reason to move to monobuild, I'd be far
less worried about it (even if I think using it as a system toolchain is
reasonable for someone to work on).
But if it's getting in the way of packaging things like flang and it's
100% on us to make standalone work - and keep it working - is it really
worth it?
I will note that it seems like the breakage has reduced upstream in the
last few cycles, although the offload case was pretty bad just in the
last week. On the other hand, if you think the problems with mlir and
friends are easily-solvable, then we may be fine as we are?
IIRC, Violet ended up running into issues with getting the CMake to
properly handle both shared and static libraries to install with the
monobuild, so maybe LLVM's CMake is just broken in both directions and
we just have to pick whichever one is least-worst for us (which might be
the standalone builds as we do now)?
[0] https://github.com/gentoo/gentoo/pull/32716#issuecomment-1713689410
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/23QT5ISGGOQUOMTOOLMUBV6MSBB2C2C6/#PDAXH7PZTA7SFSSKF65KIXYLPYHAI3SS
[2] https://github.com/gentoo/gentoo/pull/32716
sam
^ permalink raw reply [flat|nested] 11+ messages in thread
* [gentoo-dev] Re: LLVM build strategy
2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James
@ 2024-12-08 5:05 ` Sam James
2024-12-08 5:06 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Violet Purcell
2024-12-08 13:20 ` Michał Górny
2 siblings, 0 replies; 11+ messages in thread
From: Sam James @ 2024-12-08 5:05 UTC (permalink / raw
To: Michał Górny; +Cc: gentoo-dev
Sam James <sam@gentoo.org> writes:
> Michał Górny <mgorny@gentoo.org> writes:
>
>> Hello,
>>
>> Given that the number of LLVM packages is growing, and probably will
>> grow again (I'm introducing "offload" right now, expect at least MLIR
>> soon, there are open requests for flang, polly...), I'd like to propose
>> creating dedicated categories for these packages and moving them there.
>>
>> [...]
>>
>> WDYT?
>
> I fear this sort of assumes we won't switch to monobuild any time soon.
(I'm not even sure I'm advocating a move, it's more that I'd like to
flesh out the arguments for it so I have it clear in my mind.)
>
> I keep thinking [0] about how sustainable our current setup is:
> * Fedora moved away from it for >=18 [1].
> * As we saw with offload, it broke a few times in just a week. So it's
> only Gentoo who cares about this configuration AFAIK.
> * It's not working great if we're already not able to easily package
> mlir, flang, or polly.
>
> Violet attempted working on a merge before [2].
>
> The trade-offs I see are:
> * Increased disk space requirement for building LLVM
>
> I think that's a legitimate concern although perhaps not that big of a
> deal given it is, after all, a whole toolchain. GCC takes quite a bit
> to build too.
>
> * Build time
>
> Build time for mono LLVM would increase as we're building more
> components at least for some users.
>
> But the time added by
> building more components (see below) should be balanced out by ccache if
> doing development and also, importantly, not needing to force on all
> targets anymore (they keep growing).
>
> The cumulative time should be the same (although maybe a bit less
> given the targets change) given that most people need the
> same set of components because of mesa, firefox, or other things which
> need libclang.
>
> * Moving to an upstream supported-configuration
>
> We may have a bit of pain in moving to it and getting used to it, but
> we're no longer swimming upstream and being the only people caring
> about our choice of build configuration (and regularly having to
> explain and justify it to upstream).
>
> Upstream also recommend doing a bootstrap build. We don't and can't do
> that right now.
>
> Folks upstream state at every opportunity that they don't care about
> standalone builds,
> e.g. https://discourse.llvm.org/t/rfc-do-something-with-the-subproject-tarballs-in-the-release-page/75024/2.
>
> * Better support for LLVM as a system toolchain
>
> The current setup doesn't work well for people using LLVM as a system
> toolchain (because some of the components *must* be upgraded together),
> it doesn't work well for people who want to use mlir/flang/polly, and it
> doesn't work well for users on constrained hardware because we have to
> force on all targets. It also prohibits more optimisation, PGO, and
> bootstrapping it to test reliability.
>
> (This is why I'm not too sympathetic to claims that the monobuild is
> mostly for binary distributions, because we're actually *more*
> vulnerable to issues as a result of it being split when building from
> source if using the LLVM toolchain.)
>
> * Maintaining older LLVMs
>
> It's easier for older LLVMs to be either maintained by somebody else
> (for e.g. Rust's purposes) or in an overlay if it's a single package
> rather than many.
>
> This is also true for e.g. testing old snapshots or keeping binary
> packages to downgrade to once they got cleaned up.
Another issue with monobuild would be handling the case where
USE-depends-on-other-USE and you either have a lot of REQUIRED_USE, or
USE that do nothing. This is more of an issue for the runtimes, I think.
We would also have an issue where we either have to default-enable
Clang, or require many, many users to enable it manually/via autounmask
(as opposed to now where it gets dragged in as a dep).
> [...]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM)
2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James
2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James
@ 2024-12-08 5:06 ` Violet Purcell
2024-12-08 13:20 ` Michał Górny
2 siblings, 0 replies; 11+ messages in thread
From: Violet Purcell @ 2024-12-08 5:06 UTC (permalink / raw
To: gentoo-dev
On Sun, Dec 08, 2024 at 04:53:58AM +0000, Sam James wrote:
> I fear this sort of assumes we won't switch to monobuild any time soon.
>
> I keep thinking [0] about how sustainable our current setup is:
> * Fedora moved away from it for >=18 [1].
> * As we saw with offload, it broke a few times in just a week. So it's
> only Gentoo who cares about this configuration AFAIK.
> * It's not working great if we're already not able to easily package
> mlir, flang, or polly.
>
> Violet attempted working on a merge before [2].
Hey! Thanks for bringing up my work on this. I've also been continuing
this work in an overlay which I use myself on some of my machines:
https://codeberg.org/vimproved/llvm-overlay.
> IIRC, Violet ended up running into issues with getting the CMake to
> properly handle both shared and static libraries to install with the
> monobuild, so maybe LLVM's CMake is just broken in both directions and
> we just have to pick whichever one is least-worst for us (which might be
> the standalone builds as we do now)?
LLVM's CMake is definitely just rather broken in many ways, yeah. The
biggest issue I experienced when working on this was with LLD libraries.
Currently, sys-devel/lld is built with -DBUILD_SHARED_LIBS=ON, which
builds some LLD libraries used by e.g. zig as shared libraries. However,
there's no way to do this in monobuild since passing
-DBUILD_SHARED_LIBS=ON is not desirable. The solution I settled on was
splitting a sys-libs/lld-libs package which builds LLD with
-DBUILD_SHARED_LIBS=ON and then only installs the shared libraries.
As for sys-libs/llvm-libs, this package is a result of me trying to
figure out a way to have a statically linked system clang (for
performance purposes) while still having LLVM functional as a library.
After a lot of sifting through CMake spaghetti I settled on splitting
the LLVM monobuild into two packages: sys-devel/llvm and
sys-libs/llvm-libs. sys-devel/llvm installs the toolchain parts of LLVM
(statically linked), and sys-libs/llvm-libs installs the library and
CMake parts of LLVM.
This overlay is just for my personal use, and I don't think the above
package split would be viable for ::gentoo however.
One more thing that wasn't mentioned here, LLVM monobuild would also
potentially unblock PGO/BOLT optimizations for clang, so overall it
could have some pretty good speed improvements for system LLVM toolchain
users.
- Violet
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] [RFC] New categories for LLVM
2024-12-08 4:11 ` Sam James
@ 2024-12-08 13:02 ` Michał Górny
0 siblings, 0 replies; 11+ messages in thread
From: Michał Górny @ 2024-12-08 13:02 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 987 bytes --]
On Sun, 2024-12-08 at 04:11 +0000, Sam James wrote:
> I'm not sure if I'm sold on *two*. What happens for stuff like mlir
> where it's not a runtime but it's arguably more of one than core?
>
> It just doesn't feel like the division works great. Or maybe it's just
> because I feel like llvm-core will keep growing and llvm-runtimes won't.
Well, upstream has a split between "projects" and "runtimes",
and I think it makes sense to follow that. "compiler-rt" and "libc"
happen to be listed in both, but I suppose it's more of a historical
thing -- I think you're always supposed to be doing the runtimes build
for projects that support it these days.
Right now, we don't package libc, pstl, llvm-libgcc -- so there are
definitely more potential packages to be added.
I suspect that this split will also help with crossdev. FWIU we only
need to cover runtimes there, since the toolchain itself is cross-
capable by design.
--
Best regards,
Michał Górny
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM)
2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James
2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James
2024-12-08 5:06 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Violet Purcell
@ 2024-12-08 13:20 ` Michał Górny
2024-12-08 21:45 ` [gentoo-dev] Re: LLVM build strategy Sam James
2 siblings, 1 reply; 11+ messages in thread
From: Michał Górny @ 2024-12-08 13:20 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 3932 bytes --]
On Sun, 2024-12-08 at 04:53 +0000, Sam James wrote:
> I fear this sort of assumes we won't switch to monobuild any time soon.
I don't see one precluding the other. Categories are cheap. Package
moves not necessarily, but switching to monorepo will be complete pain
whether one more package move is involved or not.
> I keep thinking [0] about how sustainable our current setup is:
> * Fedora moved away from it for >=18 [1].
> * As we saw with offload, it broke a few times in just a week. So it's
> only Gentoo who cares about this configuration AFAIK.
It broke once, and only because the pull request merged preceded my
changes, and the author dealt with merge conflicts wrong.
That said, it's not like I didn't fix the monorepo build as well this
week, because it was broken from day one.
We're on our own either way.
> * Build time
>
> Build time for mono LLVM would increase as we're building more
> components at least for some users.
>
> But the time added by
> building more components (see below) should be balanced out by ccache if
> doing development and also, importantly, not needing to force on all
> targets anymore (they keep growing).
I don't see how we would avoid forcing targets if *external* projects
(wasn't the bug about Rust originally?) can still be broken if you
change targets.
> The cumulative time should be the same (although maybe a bit less
> given the targets change) given that most people need the
> same set of components because of mesa, firefox, or other things which
> need libclang.
So you spend hours building LLVM and Clang. Then you spend hours
building everything again because one more packages needs LLD. Then
more hours because you've decided to try LLDB.
I've been rebuilding three LLVM versions recently because of cpp-httplib
changing subslot multiple times recently. With the proposed change, I'd
be rebuilding everything instead.
In fact, I've already started considering splitting llvm-debuginfod.
> At the moment, I fear us choosing the non-recommended path gives us very
> little, and causes a bunch of problems in return.
If you consider being able to have a really working LLVM package "very
little", so be it.
If you choose to go for monorepo, I'm stepping down, because
I definitely won't be able to deal with this shit without being able to
split it into smaller parts.
I don't like the idea that any minor patch (think of compiler-rt that
regularly needs to be updated for newer glibc) will require spending
hours rebuilding everything. In multiple LLVM versions. For every
single person, including all the people who don't build compiler-rt
at all.
I don't like the idea of not being able to run parts of test suite
without resorting to ugly hacks. I don't like the idea of spending
hours building everything before I'm even able to start running tests,
just to learn that LLVM is broken and there's no point in even starting
to build the rest. Or having the test suite fail after a few hours
because of some minor configuration issue (like the one we'd had with
libcxx, and I've hit that one three times), and having to start
everything over again.
And ccache is not a solution. It's a cheap hack, and a costly one.
Maintaining a cache for this thing requires tons of wasted disk space.
And unless you go out of the way to reconfigure it, building 2-3 LLVM
versions will normally push all previous objects of the cache, so it
won't work for most of the people at all. Provided they go out of their
way to configure it in the first place.
In the end, LLVM is a project primarily maintained by people working for
shitty corporations that only care about being able to build their
shitty browser written in bad C++. It sucks we ended up having to
maintain it, but that's not our choice.
--
Best regards,
Michał Górny
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories
2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny
2024-12-08 4:11 ` Sam James
2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James
@ 2024-12-08 15:44 ` Michał Górny
2 siblings, 0 replies; 11+ messages in thread
From: Michał Górny @ 2024-12-08 15:44 UTC (permalink / raw
To: gentoo-dev; +Cc: Michał Górny
Signed-off-by: Michał Górny <mgorny@gentoo.org>
---
llvm-core/metadata.xml | 14 ++++++++++++++
llvm-runtimes/metadata.xml | 14 ++++++++++++++
2 files changed, 28 insertions(+)
create mode 100644 llvm-core/metadata.xml
create mode 100644 llvm-runtimes/metadata.xml
diff --git a/llvm-core/metadata.xml b/llvm-core/metadata.xml
new file mode 100644
index 000000000000..014d005edb88
--- /dev/null
+++ b/llvm-core/metadata.xml
@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE catmetadata SYSTEM "https://www.gentoo.org/dtd/metadata.dtd">
+<catmetadata>
+ <longdescription lang="en">
+ The llvm-core category contains packages comprising the LLVM
+ toolchain (normally selected among LLVM components
+ via LLVM_ENABLE_PROJECTS).
+ </longdescription>
+ <longdescription lang="pl">
+ Kategoria llvm-core zawiera paczki narzędzi programistycznych
+ LLVM (spośród komponentów LLVM wybierane za pośrednictwem
+ LLVM_ENABLE_PROJECTS).
+ </longdescription>
+</catmetadata>
diff --git a/llvm-runtimes/metadata.xml b/llvm-runtimes/metadata.xml
new file mode 100644
index 000000000000..6b0e1e57d36d
--- /dev/null
+++ b/llvm-runtimes/metadata.xml
@@ -0,0 +1,14 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE catmetadata SYSTEM "https://www.gentoo.org/dtd/metadata.dtd">
+<catmetadata>
+ <longdescription lang="en">
+ The llvm-runtimes category contains packages providing runtimes
+ specific to the LLVM toolchain (normally selected among LLVM
+ components via LLVM_ENABLE_RUNTIMES).
+ </longdescription>
+ <longdescription lang="pl">
+ Kategoria llvm-runtimes zawiera paczki z bibliotekami, używanymi
+ przez programy skompilowane za pomocą narzędzi LLVM (spośród
+ komponentów LLVM wybierane za pośrednictwem LLVM_ENABLE_RUNTIMES).
+ </longdescription>
+</catmetadata>
--
2.47.1
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [gentoo-dev] Re: LLVM build strategy
2024-12-08 13:20 ` Michał Górny
@ 2024-12-08 21:45 ` Sam James
2024-12-08 22:23 ` Eli Schwartz
0 siblings, 1 reply; 11+ messages in thread
From: Sam James @ 2024-12-08 21:45 UTC (permalink / raw
To: Michał Górny; +Cc: gentoo-dev
Michał Górny <mgorny@gentoo.org> writes:
> On Sun, 2024-12-08 at 04:53 +0000, Sam James wrote:
>> I fear this sort of assumes we won't switch to monobuild any time soon.
>
> I don't see one precluding the other. Categories are cheap. Package
> moves not necessarily, but switching to monorepo will be complete pain
> whether one more package move is involved or not.
>
>> I keep thinking [0] about how sustainable our current setup is:
>> * Fedora moved away from it for >=18 [1].
>> * As we saw with offload, it broke a few times in just a week. So it's
>> only Gentoo who cares about this configuration AFAIK.
>
> It broke once, and only because the pull request merged preceded my
> changes, and the author dealt with merge conflicts wrong.
>
> That said, it's not like I didn't fix the monorepo build as well this
> week, because it was broken from day one.
>
> We're on our own either way.
>
>
>> * Build time
>>
>> Build time for mono LLVM would increase as we're building more
>> components at least for some users.
>>
>> But the time added by
>> building more components (see below) should be balanced out by ccache if
>> doing development and also, importantly, not needing to force on all
>> targets anymore (they keep growing).
>
> I don't see how we would avoid forcing targets if *external* projects
> (wasn't the bug about Rust originally?) can still be broken if you
> change targets.
We'd have LLVM be internally consistent so we wouldn't have to worry
about issues with LLD (where it came up a lot, IIRC). But yeah, Rust
would still be a problem.
>
>> The cumulative time should be the same (although maybe a bit less
>> given the targets change) given that most people need the
>> same set of components because of mesa, firefox, or other things which
>> need libclang.
>
> So you spend hours building LLVM and Clang. Then you spend hours
> building everything again because one more packages needs LLD. Then
> more hours because you've decided to try LLDB.
>
LLD is kind of persusasive given some stuff like FF ends up needing it.
> I've been rebuilding three LLVM versions recently because of cpp-httplib
> changing subslot multiple times recently. With the proposed change, I'd
> be rebuilding everything instead.
>
> In fact, I've already started considering splitting llvm-debuginfod.
>
>> At the moment, I fear us choosing the non-recommended path gives us very
>> little, and causes a bunch of problems in return.
>
> If you consider being able to have a really working LLVM package "very
> little", so be it.
(I think "really working" is ambiguous but I've still been persuaded by
your other points.)
>
> If you choose to go for monorepo, I'm stepping down, because
> I definitely won't be able to deal with this shit without being able to
> split it into smaller parts.
>
> I don't like the idea that any minor patch (think of compiler-rt that
> regularly needs to be updated for newer glibc) will require spending
> hours rebuilding everything. In multiple LLVM versions. For every
> single person, including all the people who don't build compiler-rt
> at all.
... and I find the compiler-rt argument very compelling, given how
brittle the sanitizers are.
>
> I don't like the idea of not being able to run parts of test suite
> without resorting to ugly hacks.
I think this bit is fair.
> I don't like the idea of spending hours building everything before I'm
> even able to start running tests, just to learn that LLVM is broken
> and there's no point in even starting to build the rest.
I don't follow this bit -- you need the new LLVM merged before you can
build Clang's tests, right? And if any of it fails to build, it's not
like we can commit the release or snapshot?
What am I missing on this bit?
> Or having the test suite fail after a few hours
> because of some minor configuration issue (like the one we'd had with
> libcxx, and I've hit that one three times), and having to start
> everything over again.
>
I don't remember which configuration issue that was, although I could
easily imagine (and we've had it before) where the opposite happens
(such a configuration issue only b/c of our build). But yes, it's a fair
point that we don't have a good way to cleanly/easily just run tests
again without hacks (like avoiding clean) and it doesn't even always
work properly.
>
> And ccache is not a solution. It's a cheap hack, and a costly one.
> Maintaining a cache for this thing requires tons of wasted disk space.
> And unless you go out of the way to reconfigure it, building 2-3 LLVM
> versions will normally push all previous objects of the cache, so it
> won't work for most of the people at all. Provided they go out of their
> way to configure it in the first place.
>
> In the end, LLVM is a project primarily maintained by people working for
> shitty corporations that only care about being able to build their
> shitty browser written in bad C++. It sucks we ended up having to
> maintain it, but that's not our choice.
I'm still a bit worried about mlir/flang/clangir but you've mentioned
them on IRC today. I just hope upstream are open to it for flang.
Anyway, like I said, I think you've persuaded me - but FTR, I'd
appreciate some clarification on the points above.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] Re: LLVM build strategy
2024-12-08 21:45 ` [gentoo-dev] Re: LLVM build strategy Sam James
@ 2024-12-08 22:23 ` Eli Schwartz
2024-12-09 4:35 ` Michał Górny
0 siblings, 1 reply; 11+ messages in thread
From: Eli Schwartz @ 2024-12-08 22:23 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1.1: Type: text/plain, Size: 1277 bytes --]
On 12/8/24 4:45 PM, Sam James wrote:
>> I don't like the idea of spending hours building everything before I'm
>> even able to start running tests, just to learn that LLVM is broken
>> and there's no point in even starting to build the rest.
>
> I don't follow this bit -- you need the new LLVM merged before you can
> build Clang's tests, right? And if any of it fails to build, it's not
> like we can commit the release or snapshot?
>
> What am I missing on this bit?
I think the point here is that currently, one can build sys-devel/llvm
with tests enabled, and if it fails, there's no point in also building
sys-devel/clang. But with a monorepo build, you'd have to build llvm,
clang (and various others) first, and then launch tests for llvm and
clang together, only to get a test failure in the llvm tests that
indicates everything else is broken too. Depending on cmake test
ordering, you may also run half the clang tests before hitting the llvm
failures, even.
In theory this could be solved by building monorepo-llvm with
FEATURES=test USE="-clang" to start running tests, and then if that
passes, rebuild llvm again but this time with clang etc. enabled. Not
sure this is actually solving the stated objection...
--
Eli Schwartz
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] Re: LLVM build strategy
2024-12-08 22:23 ` Eli Schwartz
@ 2024-12-09 4:35 ` Michał Górny
0 siblings, 0 replies; 11+ messages in thread
From: Michał Górny @ 2024-12-09 4:35 UTC (permalink / raw
To: gentoo-dev
[-- Attachment #1: Type: text/plain, Size: 2183 bytes --]
On Sun, 2024-12-08 at 17:23 -0500, Eli Schwartz wrote:
> On 12/8/24 4:45 PM, Sam James wrote:
> > > I don't like the idea of spending hours building everything before I'm
> > > even able to start running tests, just to learn that LLVM is broken
> > > and there's no point in even starting to build the rest.
> >
> > I don't follow this bit -- you need the new LLVM merged before you can
> > build Clang's tests, right? And if any of it fails to build, it's not
> > like we can commit the release or snapshot?
> >
> > What am I missing on this bit?
>
>
> I think the point here is that currently, one can build sys-devel/llvm
> with tests enabled, and if it fails, there's no point in also building
> sys-devel/clang. But with a monorepo build, you'd have to build llvm,
> clang (and various others) first, and then launch tests for llvm and
> clang together, only to get a test failure in the llvm tests that
> indicates everything else is broken too. Depending on cmake test
> ordering, you may also run half the clang tests before hitting the llvm
> failures, even.
Exactly. You have to compile everything first, before you start testing
anything.
And I don't think testing order is predictable either. I mean,
technically we could just issue check-llvm, check-clang, and so on
manually, but that's going to be messy and I'm not even sure if I can
figure out all the correct targets to avoid duplication.
And if you think we could hack this around by stubbing src_compile()
and having tests compile things component after component...
the dependencies are pretty much broken everywhere, and you do need to
compile everything before you run check-* targets.
> In theory this could be solved by building monorepo-llvm with
> FEATURES=test USE="-clang" to start running tests, and then if that
> passes, rebuild llvm again but this time with clang etc. enabled. Not
> sure this is actually solving the stated objection...
ccache will save most of the time on rebuilding the same things (except
for tablegen or sphinx, which are pretty slow too), but you'd still be
rerunning the same tests.
--
Best regards,
Michał Górny
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 512 bytes --]
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-12-09 4:35 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny
2024-12-08 4:11 ` Sam James
2024-12-08 13:02 ` Michał Górny
2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James
2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James
2024-12-08 5:06 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Violet Purcell
2024-12-08 13:20 ` Michał Górny
2024-12-08 21:45 ` [gentoo-dev] Re: LLVM build strategy Sam James
2024-12-08 22:23 ` Eli Schwartz
2024-12-09 4:35 ` Michał Górny
2024-12-08 15:44 ` [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories Michał Górny
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox