* [gentoo-dev] [RFC] New categories for LLVM @ 2024-12-07 16:07 Michał Górny 2024-12-08 4:11 ` Sam James ` (2 more replies) 0 siblings, 3 replies; 11+ messages in thread From: Michał Górny @ 2024-12-07 16:07 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 1043 bytes --] Hello, Given that the number of LLVM packages is growing, and probably will grow again (I'm introducing "offload" right now, expect at least MLIR soon, there are open requests for flang, polly...), I'd like to propose creating dedicated categories for these packages and moving them there. If not anything else, this will help consistently applying flags and keywords to these packages (`/etc/portage/package.*` accept wildcards). My initial idea would be to use two categories: one for the toolchain packages, another for runtimes, e.g.: llvm-core/ clang clang-common clang-runtime clang-toolchain-symlinks lld lld-toolchain-symlinks lldb llvm llvm-common llvm-toolchain-symlinks llvmgold llvm-runtimes/ compiler-rt compiler-rt-sanitizers libclc libcxx libcxxabi libomp (-> openmp?) llvm-offload (-> offload) llvm-unwind (-> unwind?) clang-python, lit and llvm-ocaml would remain in their language categories. WDYT? -- Best regards, Michał Górny [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 512 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] [RFC] New categories for LLVM 2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny @ 2024-12-08 4:11 ` Sam James 2024-12-08 13:02 ` Michał Górny 2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James 2024-12-08 15:44 ` [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories Michał Górny 2 siblings, 1 reply; 11+ messages in thread From: Sam James @ 2024-12-08 4:11 UTC (permalink / raw To: Michał Górny; +Cc: gentoo-dev Michał Górny <mgorny@gentoo.org> writes: > Hello, > > Given that the number of LLVM packages is growing, and probably will > grow again (I'm introducing "offload" right now, expect at least MLIR > soon, there are open requests for flang, polly...), I'd like to propose > creating dedicated categories for these packages and moving them there. > > If not anything else, this will help consistently applying flags > and keywords to these packages (`/etc/portage/package.*` accept > wildcards). I quite like the idea of having a category to ease keywording and testing. > > My initial idea would be to use two categories: one for the toolchain > packages, another for runtimes, e.g.: > > llvm-core/ > clang > clang-common > clang-runtime > clang-toolchain-symlinks > lld > lld-toolchain-symlinks > lldb > llvm > llvm-common > llvm-toolchain-symlinks > llvmgold > > llvm-runtimes/ > compiler-rt > compiler-rt-sanitizers > libclc > libcxx > libcxxabi > libomp (-> openmp?) > llvm-offload (-> offload) > llvm-unwind (-> unwind?) > I'm not sure if I'm sold on *two*. What happens for stuff like mlir where it's not a runtime but it's arguably more of one than core? It just doesn't feel like the division works great. Or maybe it's just because I feel like llvm-core will keep growing and llvm-runtimes won't. > clang-python, lit and llvm-ocaml would remain in their language > categories. > > WDYT? I'm going to start another reply for a subthread. thanks, sam ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] [RFC] New categories for LLVM 2024-12-08 4:11 ` Sam James @ 2024-12-08 13:02 ` Michał Górny 0 siblings, 0 replies; 11+ messages in thread From: Michał Górny @ 2024-12-08 13:02 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 987 bytes --] On Sun, 2024-12-08 at 04:11 +0000, Sam James wrote: > I'm not sure if I'm sold on *two*. What happens for stuff like mlir > where it's not a runtime but it's arguably more of one than core? > > It just doesn't feel like the division works great. Or maybe it's just > because I feel like llvm-core will keep growing and llvm-runtimes won't. Well, upstream has a split between "projects" and "runtimes", and I think it makes sense to follow that. "compiler-rt" and "libc" happen to be listed in both, but I suppose it's more of a historical thing -- I think you're always supposed to be doing the runtimes build for projects that support it these days. Right now, we don't package libc, pstl, llvm-libgcc -- so there are definitely more potential packages to be added. I suspect that this split will also help with crossdev. FWIU we only need to cover runtimes there, since the toolchain itself is cross- capable by design. -- Best regards, Michał Górny [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 512 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) 2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny 2024-12-08 4:11 ` Sam James @ 2024-12-08 4:53 ` Sam James 2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James ` (2 more replies) 2024-12-08 15:44 ` [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories Michał Górny 2 siblings, 3 replies; 11+ messages in thread From: Sam James @ 2024-12-08 4:53 UTC (permalink / raw To: Michał Górny; +Cc: gentoo-dev Michał Górny <mgorny@gentoo.org> writes: > Hello, > > Given that the number of LLVM packages is growing, and probably will > grow again (I'm introducing "offload" right now, expect at least MLIR > soon, there are open requests for flang, polly...), I'd like to propose > creating dedicated categories for these packages and moving them there. > > [...] > > WDYT? I fear this sort of assumes we won't switch to monobuild any time soon. I keep thinking [0] about how sustainable our current setup is: * Fedora moved away from it for >=18 [1]. * As we saw with offload, it broke a few times in just a week. So it's only Gentoo who cares about this configuration AFAIK. * It's not working great if we're already not able to easily package mlir, flang, or polly. Violet attempted working on a merge before [2]. The trade-offs I see are: * Increased disk space requirement for building LLVM I think that's a legitimate concern although perhaps not that big of a deal given it is, after all, a whole toolchain. GCC takes quite a bit to build too. * Build time Build time for mono LLVM would increase as we're building more components at least for some users. But the time added by building more components (see below) should be balanced out by ccache if doing development and also, importantly, not needing to force on all targets anymore (they keep growing). The cumulative time should be the same (although maybe a bit less given the targets change) given that most people need the same set of components because of mesa, firefox, or other things which need libclang. * Moving to an upstream supported-configuration We may have a bit of pain in moving to it and getting used to it, but we're no longer swimming upstream and being the only people caring about our choice of build configuration (and regularly having to explain and justify it to upstream). Upstream also recommend doing a bootstrap build. We don't and can't do that right now. Folks upstream state at every opportunity that they don't care about standalone builds, e.g. https://discourse.llvm.org/t/rfc-do-something-with-the-subproject-tarballs-in-the-release-page/75024/2. * Better support for LLVM as a system toolchain The current setup doesn't work well for people using LLVM as a system toolchain (because some of the components *must* be upgraded together), it doesn't work well for people who want to use mlir/flang/polly, and it doesn't work well for users on constrained hardware because we have to force on all targets. It also prohibits more optimisation, PGO, and bootstrapping it to test reliability. (This is why I'm not too sympathetic to claims that the monobuild is mostly for binary distributions, because we're actually *more* vulnerable to issues as a result of it being split when building from source if using the LLVM toolchain.) * Maintaining older LLVMs It's easier for older LLVMs to be either maintained by somebody else (for e.g. Rust's purposes) or in an overlay if it's a single package rather than many. This is also true for e.g. testing old snapshots or keeping binary packages to downgrade to once they got cleaned up. At the moment, I fear us choosing the non-recommended path gives us very little, and causes a bunch of problems in return. I don't particularly care much personally about using LLVM as the system toolchain and if that's the only reason to move to monobuild, I'd be far less worried about it (even if I think using it as a system toolchain is reasonable for someone to work on). But if it's getting in the way of packaging things like flang and it's 100% on us to make standalone work - and keep it working - is it really worth it? I will note that it seems like the breakage has reduced upstream in the last few cycles, although the offload case was pretty bad just in the last week. On the other hand, if you think the problems with mlir and friends are easily-solvable, then we may be fine as we are? IIRC, Violet ended up running into issues with getting the CMake to properly handle both shared and static libraries to install with the monobuild, so maybe LLVM's CMake is just broken in both directions and we just have to pick whichever one is least-worst for us (which might be the standalone builds as we do now)? [0] https://github.com/gentoo/gentoo/pull/32716#issuecomment-1713689410 [1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/23QT5ISGGOQUOMTOOLMUBV6MSBB2C2C6/#PDAXH7PZTA7SFSSKF65KIXYLPYHAI3SS [2] https://github.com/gentoo/gentoo/pull/32716 sam ^ permalink raw reply [flat|nested] 11+ messages in thread
* [gentoo-dev] Re: LLVM build strategy 2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James @ 2024-12-08 5:05 ` Sam James 2024-12-08 5:06 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Violet Purcell 2024-12-08 13:20 ` Michał Górny 2 siblings, 0 replies; 11+ messages in thread From: Sam James @ 2024-12-08 5:05 UTC (permalink / raw To: Michał Górny; +Cc: gentoo-dev Sam James <sam@gentoo.org> writes: > Michał Górny <mgorny@gentoo.org> writes: > >> Hello, >> >> Given that the number of LLVM packages is growing, and probably will >> grow again (I'm introducing "offload" right now, expect at least MLIR >> soon, there are open requests for flang, polly...), I'd like to propose >> creating dedicated categories for these packages and moving them there. >> >> [...] >> >> WDYT? > > I fear this sort of assumes we won't switch to monobuild any time soon. (I'm not even sure I'm advocating a move, it's more that I'd like to flesh out the arguments for it so I have it clear in my mind.) > > I keep thinking [0] about how sustainable our current setup is: > * Fedora moved away from it for >=18 [1]. > * As we saw with offload, it broke a few times in just a week. So it's > only Gentoo who cares about this configuration AFAIK. > * It's not working great if we're already not able to easily package > mlir, flang, or polly. > > Violet attempted working on a merge before [2]. > > The trade-offs I see are: > * Increased disk space requirement for building LLVM > > I think that's a legitimate concern although perhaps not that big of a > deal given it is, after all, a whole toolchain. GCC takes quite a bit > to build too. > > * Build time > > Build time for mono LLVM would increase as we're building more > components at least for some users. > > But the time added by > building more components (see below) should be balanced out by ccache if > doing development and also, importantly, not needing to force on all > targets anymore (they keep growing). > > The cumulative time should be the same (although maybe a bit less > given the targets change) given that most people need the > same set of components because of mesa, firefox, or other things which > need libclang. > > * Moving to an upstream supported-configuration > > We may have a bit of pain in moving to it and getting used to it, but > we're no longer swimming upstream and being the only people caring > about our choice of build configuration (and regularly having to > explain and justify it to upstream). > > Upstream also recommend doing a bootstrap build. We don't and can't do > that right now. > > Folks upstream state at every opportunity that they don't care about > standalone builds, > e.g. https://discourse.llvm.org/t/rfc-do-something-with-the-subproject-tarballs-in-the-release-page/75024/2. > > * Better support for LLVM as a system toolchain > > The current setup doesn't work well for people using LLVM as a system > toolchain (because some of the components *must* be upgraded together), > it doesn't work well for people who want to use mlir/flang/polly, and it > doesn't work well for users on constrained hardware because we have to > force on all targets. It also prohibits more optimisation, PGO, and > bootstrapping it to test reliability. > > (This is why I'm not too sympathetic to claims that the monobuild is > mostly for binary distributions, because we're actually *more* > vulnerable to issues as a result of it being split when building from > source if using the LLVM toolchain.) > > * Maintaining older LLVMs > > It's easier for older LLVMs to be either maintained by somebody else > (for e.g. Rust's purposes) or in an overlay if it's a single package > rather than many. > > This is also true for e.g. testing old snapshots or keeping binary > packages to downgrade to once they got cleaned up. Another issue with monobuild would be handling the case where USE-depends-on-other-USE and you either have a lot of REQUIRED_USE, or USE that do nothing. This is more of an issue for the runtimes, I think. We would also have an issue where we either have to default-enable Clang, or require many, many users to enable it manually/via autounmask (as opposed to now where it gets dragged in as a dep). > [...] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) 2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James 2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James @ 2024-12-08 5:06 ` Violet Purcell 2024-12-08 13:20 ` Michał Górny 2 siblings, 0 replies; 11+ messages in thread From: Violet Purcell @ 2024-12-08 5:06 UTC (permalink / raw To: gentoo-dev On Sun, Dec 08, 2024 at 04:53:58AM +0000, Sam James wrote: > I fear this sort of assumes we won't switch to monobuild any time soon. > > I keep thinking [0] about how sustainable our current setup is: > * Fedora moved away from it for >=18 [1]. > * As we saw with offload, it broke a few times in just a week. So it's > only Gentoo who cares about this configuration AFAIK. > * It's not working great if we're already not able to easily package > mlir, flang, or polly. > > Violet attempted working on a merge before [2]. Hey! Thanks for bringing up my work on this. I've also been continuing this work in an overlay which I use myself on some of my machines: https://codeberg.org/vimproved/llvm-overlay. > IIRC, Violet ended up running into issues with getting the CMake to > properly handle both shared and static libraries to install with the > monobuild, so maybe LLVM's CMake is just broken in both directions and > we just have to pick whichever one is least-worst for us (which might be > the standalone builds as we do now)? LLVM's CMake is definitely just rather broken in many ways, yeah. The biggest issue I experienced when working on this was with LLD libraries. Currently, sys-devel/lld is built with -DBUILD_SHARED_LIBS=ON, which builds some LLD libraries used by e.g. zig as shared libraries. However, there's no way to do this in monobuild since passing -DBUILD_SHARED_LIBS=ON is not desirable. The solution I settled on was splitting a sys-libs/lld-libs package which builds LLD with -DBUILD_SHARED_LIBS=ON and then only installs the shared libraries. As for sys-libs/llvm-libs, this package is a result of me trying to figure out a way to have a statically linked system clang (for performance purposes) while still having LLVM functional as a library. After a lot of sifting through CMake spaghetti I settled on splitting the LLVM monobuild into two packages: sys-devel/llvm and sys-libs/llvm-libs. sys-devel/llvm installs the toolchain parts of LLVM (statically linked), and sys-libs/llvm-libs installs the library and CMake parts of LLVM. This overlay is just for my personal use, and I don't think the above package split would be viable for ::gentoo however. One more thing that wasn't mentioned here, LLVM monobuild would also potentially unblock PGO/BOLT optimizations for clang, so overall it could have some pretty good speed improvements for system LLVM toolchain users. - Violet ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) 2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James 2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James 2024-12-08 5:06 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Violet Purcell @ 2024-12-08 13:20 ` Michał Górny 2024-12-08 21:45 ` [gentoo-dev] Re: LLVM build strategy Sam James 2 siblings, 1 reply; 11+ messages in thread From: Michał Górny @ 2024-12-08 13:20 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 3932 bytes --] On Sun, 2024-12-08 at 04:53 +0000, Sam James wrote: > I fear this sort of assumes we won't switch to monobuild any time soon. I don't see one precluding the other. Categories are cheap. Package moves not necessarily, but switching to monorepo will be complete pain whether one more package move is involved or not. > I keep thinking [0] about how sustainable our current setup is: > * Fedora moved away from it for >=18 [1]. > * As we saw with offload, it broke a few times in just a week. So it's > only Gentoo who cares about this configuration AFAIK. It broke once, and only because the pull request merged preceded my changes, and the author dealt with merge conflicts wrong. That said, it's not like I didn't fix the monorepo build as well this week, because it was broken from day one. We're on our own either way. > * Build time > > Build time for mono LLVM would increase as we're building more > components at least for some users. > > But the time added by > building more components (see below) should be balanced out by ccache if > doing development and also, importantly, not needing to force on all > targets anymore (they keep growing). I don't see how we would avoid forcing targets if *external* projects (wasn't the bug about Rust originally?) can still be broken if you change targets. > The cumulative time should be the same (although maybe a bit less > given the targets change) given that most people need the > same set of components because of mesa, firefox, or other things which > need libclang. So you spend hours building LLVM and Clang. Then you spend hours building everything again because one more packages needs LLD. Then more hours because you've decided to try LLDB. I've been rebuilding three LLVM versions recently because of cpp-httplib changing subslot multiple times recently. With the proposed change, I'd be rebuilding everything instead. In fact, I've already started considering splitting llvm-debuginfod. > At the moment, I fear us choosing the non-recommended path gives us very > little, and causes a bunch of problems in return. If you consider being able to have a really working LLVM package "very little", so be it. If you choose to go for monorepo, I'm stepping down, because I definitely won't be able to deal with this shit without being able to split it into smaller parts. I don't like the idea that any minor patch (think of compiler-rt that regularly needs to be updated for newer glibc) will require spending hours rebuilding everything. In multiple LLVM versions. For every single person, including all the people who don't build compiler-rt at all. I don't like the idea of not being able to run parts of test suite without resorting to ugly hacks. I don't like the idea of spending hours building everything before I'm even able to start running tests, just to learn that LLVM is broken and there's no point in even starting to build the rest. Or having the test suite fail after a few hours because of some minor configuration issue (like the one we'd had with libcxx, and I've hit that one three times), and having to start everything over again. And ccache is not a solution. It's a cheap hack, and a costly one. Maintaining a cache for this thing requires tons of wasted disk space. And unless you go out of the way to reconfigure it, building 2-3 LLVM versions will normally push all previous objects of the cache, so it won't work for most of the people at all. Provided they go out of their way to configure it in the first place. In the end, LLVM is a project primarily maintained by people working for shitty corporations that only care about being able to build their shitty browser written in bad C++. It sucks we ended up having to maintain it, but that's not our choice. -- Best regards, Michał Górny [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 512 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* [gentoo-dev] Re: LLVM build strategy 2024-12-08 13:20 ` Michał Górny @ 2024-12-08 21:45 ` Sam James 2024-12-08 22:23 ` Eli Schwartz 0 siblings, 1 reply; 11+ messages in thread From: Sam James @ 2024-12-08 21:45 UTC (permalink / raw To: Michał Górny; +Cc: gentoo-dev Michał Górny <mgorny@gentoo.org> writes: > On Sun, 2024-12-08 at 04:53 +0000, Sam James wrote: >> I fear this sort of assumes we won't switch to monobuild any time soon. > > I don't see one precluding the other. Categories are cheap. Package > moves not necessarily, but switching to monorepo will be complete pain > whether one more package move is involved or not. > >> I keep thinking [0] about how sustainable our current setup is: >> * Fedora moved away from it for >=18 [1]. >> * As we saw with offload, it broke a few times in just a week. So it's >> only Gentoo who cares about this configuration AFAIK. > > It broke once, and only because the pull request merged preceded my > changes, and the author dealt with merge conflicts wrong. > > That said, it's not like I didn't fix the monorepo build as well this > week, because it was broken from day one. > > We're on our own either way. > > >> * Build time >> >> Build time for mono LLVM would increase as we're building more >> components at least for some users. >> >> But the time added by >> building more components (see below) should be balanced out by ccache if >> doing development and also, importantly, not needing to force on all >> targets anymore (they keep growing). > > I don't see how we would avoid forcing targets if *external* projects > (wasn't the bug about Rust originally?) can still be broken if you > change targets. We'd have LLVM be internally consistent so we wouldn't have to worry about issues with LLD (where it came up a lot, IIRC). But yeah, Rust would still be a problem. > >> The cumulative time should be the same (although maybe a bit less >> given the targets change) given that most people need the >> same set of components because of mesa, firefox, or other things which >> need libclang. > > So you spend hours building LLVM and Clang. Then you spend hours > building everything again because one more packages needs LLD. Then > more hours because you've decided to try LLDB. > LLD is kind of persusasive given some stuff like FF ends up needing it. > I've been rebuilding three LLVM versions recently because of cpp-httplib > changing subslot multiple times recently. With the proposed change, I'd > be rebuilding everything instead. > > In fact, I've already started considering splitting llvm-debuginfod. > >> At the moment, I fear us choosing the non-recommended path gives us very >> little, and causes a bunch of problems in return. > > If you consider being able to have a really working LLVM package "very > little", so be it. (I think "really working" is ambiguous but I've still been persuaded by your other points.) > > If you choose to go for monorepo, I'm stepping down, because > I definitely won't be able to deal with this shit without being able to > split it into smaller parts. > > I don't like the idea that any minor patch (think of compiler-rt that > regularly needs to be updated for newer glibc) will require spending > hours rebuilding everything. In multiple LLVM versions. For every > single person, including all the people who don't build compiler-rt > at all. ... and I find the compiler-rt argument very compelling, given how brittle the sanitizers are. > > I don't like the idea of not being able to run parts of test suite > without resorting to ugly hacks. I think this bit is fair. > I don't like the idea of spending hours building everything before I'm > even able to start running tests, just to learn that LLVM is broken > and there's no point in even starting to build the rest. I don't follow this bit -- you need the new LLVM merged before you can build Clang's tests, right? And if any of it fails to build, it's not like we can commit the release or snapshot? What am I missing on this bit? > Or having the test suite fail after a few hours > because of some minor configuration issue (like the one we'd had with > libcxx, and I've hit that one three times), and having to start > everything over again. > I don't remember which configuration issue that was, although I could easily imagine (and we've had it before) where the opposite happens (such a configuration issue only b/c of our build). But yes, it's a fair point that we don't have a good way to cleanly/easily just run tests again without hacks (like avoiding clean) and it doesn't even always work properly. > > And ccache is not a solution. It's a cheap hack, and a costly one. > Maintaining a cache for this thing requires tons of wasted disk space. > And unless you go out of the way to reconfigure it, building 2-3 LLVM > versions will normally push all previous objects of the cache, so it > won't work for most of the people at all. Provided they go out of their > way to configure it in the first place. > > In the end, LLVM is a project primarily maintained by people working for > shitty corporations that only care about being able to build their > shitty browser written in bad C++. It sucks we ended up having to > maintain it, but that's not our choice. I'm still a bit worried about mlir/flang/clangir but you've mentioned them on IRC today. I just hope upstream are open to it for flang. Anyway, like I said, I think you've persuaded me - but FTR, I'd appreciate some clarification on the points above. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] Re: LLVM build strategy 2024-12-08 21:45 ` [gentoo-dev] Re: LLVM build strategy Sam James @ 2024-12-08 22:23 ` Eli Schwartz 2024-12-09 4:35 ` Michał Górny 0 siblings, 1 reply; 11+ messages in thread From: Eli Schwartz @ 2024-12-08 22:23 UTC (permalink / raw To: gentoo-dev [-- Attachment #1.1: Type: text/plain, Size: 1277 bytes --] On 12/8/24 4:45 PM, Sam James wrote: >> I don't like the idea of spending hours building everything before I'm >> even able to start running tests, just to learn that LLVM is broken >> and there's no point in even starting to build the rest. > > I don't follow this bit -- you need the new LLVM merged before you can > build Clang's tests, right? And if any of it fails to build, it's not > like we can commit the release or snapshot? > > What am I missing on this bit? I think the point here is that currently, one can build sys-devel/llvm with tests enabled, and if it fails, there's no point in also building sys-devel/clang. But with a monorepo build, you'd have to build llvm, clang (and various others) first, and then launch tests for llvm and clang together, only to get a test failure in the llvm tests that indicates everything else is broken too. Depending on cmake test ordering, you may also run half the clang tests before hitting the llvm failures, even. In theory this could be solved by building monorepo-llvm with FEATURES=test USE="-clang" to start running tests, and then if that passes, rebuild llvm again but this time with clang etc. enabled. Not sure this is actually solving the stated objection... -- Eli Schwartz [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 236 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [gentoo-dev] Re: LLVM build strategy 2024-12-08 22:23 ` Eli Schwartz @ 2024-12-09 4:35 ` Michał Górny 0 siblings, 0 replies; 11+ messages in thread From: Michał Górny @ 2024-12-09 4:35 UTC (permalink / raw To: gentoo-dev [-- Attachment #1: Type: text/plain, Size: 2183 bytes --] On Sun, 2024-12-08 at 17:23 -0500, Eli Schwartz wrote: > On 12/8/24 4:45 PM, Sam James wrote: > > > I don't like the idea of spending hours building everything before I'm > > > even able to start running tests, just to learn that LLVM is broken > > > and there's no point in even starting to build the rest. > > > > I don't follow this bit -- you need the new LLVM merged before you can > > build Clang's tests, right? And if any of it fails to build, it's not > > like we can commit the release or snapshot? > > > > What am I missing on this bit? > > > I think the point here is that currently, one can build sys-devel/llvm > with tests enabled, and if it fails, there's no point in also building > sys-devel/clang. But with a monorepo build, you'd have to build llvm, > clang (and various others) first, and then launch tests for llvm and > clang together, only to get a test failure in the llvm tests that > indicates everything else is broken too. Depending on cmake test > ordering, you may also run half the clang tests before hitting the llvm > failures, even. Exactly. You have to compile everything first, before you start testing anything. And I don't think testing order is predictable either. I mean, technically we could just issue check-llvm, check-clang, and so on manually, but that's going to be messy and I'm not even sure if I can figure out all the correct targets to avoid duplication. And if you think we could hack this around by stubbing src_compile() and having tests compile things component after component... the dependencies are pretty much broken everywhere, and you do need to compile everything before you run check-* targets. > In theory this could be solved by building monorepo-llvm with > FEATURES=test USE="-clang" to start running tests, and then if that > passes, rebuild llvm again but this time with clang etc. enabled. Not > sure this is actually solving the stated objection... ccache will save most of the time on rebuilding the same things (except for tablegen or sphinx, which are pretty slow too), but you'd still be rerunning the same tests. -- Best regards, Michał Górny [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 512 bytes --] ^ permalink raw reply [flat|nested] 11+ messages in thread
* [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories 2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny 2024-12-08 4:11 ` Sam James 2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James @ 2024-12-08 15:44 ` Michał Górny 2 siblings, 0 replies; 11+ messages in thread From: Michał Górny @ 2024-12-08 15:44 UTC (permalink / raw To: gentoo-dev; +Cc: Michał Górny Signed-off-by: Michał Górny <mgorny@gentoo.org> --- llvm-core/metadata.xml | 14 ++++++++++++++ llvm-runtimes/metadata.xml | 14 ++++++++++++++ 2 files changed, 28 insertions(+) create mode 100644 llvm-core/metadata.xml create mode 100644 llvm-runtimes/metadata.xml diff --git a/llvm-core/metadata.xml b/llvm-core/metadata.xml new file mode 100644 index 000000000000..014d005edb88 --- /dev/null +++ b/llvm-core/metadata.xml @@ -0,0 +1,14 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE catmetadata SYSTEM "https://www.gentoo.org/dtd/metadata.dtd"> +<catmetadata> + <longdescription lang="en"> + The llvm-core category contains packages comprising the LLVM + toolchain (normally selected among LLVM components + via LLVM_ENABLE_PROJECTS). + </longdescription> + <longdescription lang="pl"> + Kategoria llvm-core zawiera paczki narzędzi programistycznych + LLVM (spośród komponentów LLVM wybierane za pośrednictwem + LLVM_ENABLE_PROJECTS). + </longdescription> +</catmetadata> diff --git a/llvm-runtimes/metadata.xml b/llvm-runtimes/metadata.xml new file mode 100644 index 000000000000..6b0e1e57d36d --- /dev/null +++ b/llvm-runtimes/metadata.xml @@ -0,0 +1,14 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE catmetadata SYSTEM "https://www.gentoo.org/dtd/metadata.dtd"> +<catmetadata> + <longdescription lang="en"> + The llvm-runtimes category contains packages providing runtimes + specific to the LLVM toolchain (normally selected among LLVM + components via LLVM_ENABLE_RUNTIMES). + </longdescription> + <longdescription lang="pl"> + Kategoria llvm-runtimes zawiera paczki z bibliotekami, używanymi + przez programy skompilowane za pomocą narzędzi LLVM (spośród + komponentów LLVM wybierane za pośrednictwem LLVM_ENABLE_RUNTIMES). + </longdescription> +</catmetadata> -- 2.47.1 ^ permalink raw reply related [flat|nested] 11+ messages in thread
end of thread, other threads:[~2024-12-09 4:35 UTC | newest] Thread overview: 11+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-12-07 16:07 [gentoo-dev] [RFC] New categories for LLVM Michał Górny 2024-12-08 4:11 ` Sam James 2024-12-08 13:02 ` Michał Górny 2024-12-08 4:53 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Sam James 2024-12-08 5:05 ` [gentoo-dev] Re: LLVM build strategy Sam James 2024-12-08 5:06 ` LLVM build strategy (was: Re: [gentoo-dev] [RFC] New categories for LLVM) Violet Purcell 2024-12-08 13:20 ` Michał Górny 2024-12-08 21:45 ` [gentoo-dev] Re: LLVM build strategy Sam James 2024-12-08 22:23 ` Eli Schwartz 2024-12-09 4:35 ` Michał Górny 2024-12-08 15:44 ` [gentoo-dev] [PATCH] Introduce llvm-core and llvm-runtimes categories Michał Górny
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox