From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 1E6951581D8 for ; Wed, 4 Dec 2024 01:13:48 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id 2ABA1E07D0; Wed, 4 Dec 2024 01:13:43 +0000 (UTC) Received: from smtp.gentoo.org (woodpecker.gentoo.org [140.211.166.183]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 3F42FE07A5 for ; Wed, 4 Dec 2024 01:13:42 +0000 (UTC) Message-ID: <9baf649f-1368-48a2-bb22-bdc7e2d391b6@gentoo.org> Date: Wed, 4 Dec 2024 11:13:36 +1000 Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-dev@lists.gentoo.org Reply-to: gentoo-dev@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [gentoo-dev] [RFC] Future changes to LLVM eclasses (or how do you use LLVM?) To: gentoo-dev@lists.gentoo.org References: Content-Language: en-US From: Matt Jolly Autocrypt: addr=kangie@gentoo.org; keydata= xsFNBF6BjksBEAC1QaqF3zKOgdunRhkt9nXkdlsL1sriTBk3WSy4De6wYLjiSofGRJY5pAiH EnlD/oW+sxDQ1DQQ3jNW/xlLUKFKYRnWhmkUv7iy0VDFrdj3mZic1pWr5a+sFX9DZNdYxLaa RIVgkstsLbf0ks5EvIqk5d7Ty8B48CgZZL7RXpAP7xrOgmat+JXNovX4djPW7HyNfblcAbzj tsLcQf7/4Q0LfK7wW3MVJmpmNK7dSKKaWgSXY7IICird0tNMF41vuMRJIta0NtIARq+AjCxV 0iEb5odYIbrbamCpzhupJM8M5LPZkbQERIF+OTzGiXhlGVloxlB+vpTzv2oY+fQkLBSSL+NS P2Pv01exA+ezA+5r+KUClA7qUnC8UriwixjJuWfoe03sYRPb91z4nlBKiCP6+jcJpS4+KsgT +//r2PdHBQDC4kVimjWDfbfXoVUhtxbQ3t/Y1vro5WQvOsgNl53hWcuHd1O+zPS1d7x5xj7a otyLJbDwtNDtApHxNYld/w/uj0QpIlz1hj9QX/NthYe6nHYgPVtEmAXjqtdIxrlS6Qx/Gaqf xVhaxtLJVDWqba4GsVA1xlFuQKJ+RTT9OXGhpQNkLPo7rt5FP3C/SUVJa3YewE3x8vmHvfoK PoKGA/wnBCVO68yQw5K3opb5DC4Z9tngrwGXyfhZzE9Uqk+wswARAQABzR5NYXR0IEpvbGx5 IDxrYW5naWVAZ2VudG9vLm9yZz7CwZQEEwEKAD4WIQTdt8K8F88DrbOPnBZQ7FSNUuBRwAUC ZccI4wIbAwUJC5i9UAULCQgHAwUVCgkICwUWAwIBAAIeBQIXgAAKCRBQ7FSNUuBRwCNuEACi 1C6XSSPWfFh/14Tn3TqseLvX0554tmsW7FoxEu1jGQIEA2f6rtmn6P2ZKCTmqxzUC0sb17TL 8/8sP7ignGJ3nuPgEczv/VeNQziMOmfxjf2VLk8pQ0seQbAwNMvrgjgcRhCdqwIlQKHURlzj qUbNvp7czV8FVd2qV3iLcf3i8VNbqfBffsTxeyhV88ks9FmbT8VcO2bV1Snl6LsG8u3QKme/ kCxg438w1IozYIGP/2M/rl7SUPhuat66mptm9009g8xnXgJBiwI39ZsDhJ8x8sSN2tF3prsG 2SvInhROv8FBmN/DbjfCda2I3hSoSLp5Q/+FeYNJflcBcuWmEU77KhqSBFJrdPZg6+aleQsL vI2WPvwKWZzveUMwbn1ahnwrfLIW2Ue0JMIR/nRXUy7d7ngx111P27si27n/XNrtzCqWQnj9 MpcZ+8GO670H5MR1iV0FEG4rz1iEcw/fNgFFrF0HRgHXrUzSx/aKQDjvDrnZBmslnLXQ8uJS pYUCCluOE7xX6D4R2+K4u7KsEWFlCqhTGAeOMZLY5avjhiQF9ZKtsDcVY3fZwrih+uWvvpPF zAKy3hiY+uC3HQqvJYQofogm1Ls/5WYZ9syBjKPdFnpsj33NTTmDDAz+yl+Y1Unp4iU/SnyR JXB2usTR8b1rMywp4YI9B8OBd+RkBOVIGc7BTQRegY5LARAAs/Hifx+R6ItN95PhwcsGr8/V q8dOqTbPd72/BppY8yeJZFRTl0n24ZVWXBJKBSnIfe8uFmNO/f58yY/MJ2ADF5Sqyn8V76Nb t2JS+dqxlnKxkjsXiKJhZiJ0Mp2+oGO1mBbJpjqWGWiVDBp0P0O4DX+ELWI/MfGiavLO8BXl vL2/qlT8we4obExgStAClKqjM9eIKFL93xPrgS8sFmAGSHC4e0wD9YjjxX4AkIdoJ1F2m47y 2QgGKj9w16sqZswt0XOa+TLNIMEgXH24m66kQHUxE1JZfFDWX5HmC2BRFEfIjaQHsDLYBTYJ 8vkIUR0uZm9I/TUnAQMrN20/y2cbFIUynsNKArg03ZpzCxegOxJSPIv6j97Hy4byDyu/Ybj4 Z8d9/buGHHjzwRLkszyKEYcSSRVvk7Z1kSsfya7OmO7RMJoRwLH/CSZUR23q4X/w8oUSUXl/ S8+avuji3eDGk1Fyr+UjRabXqRL63wghBS895JWYAQRGcDL0d2x/cJi+JACT5Mezchgj393n 43yOvVeVotHvbk0ez+YOJ/pUDBFVzD0FPBeS2YTKrWJXRlxtRTei4B6LzbMFddMYclFDjVqq WcW+1d9Ck0H3pIB9BkAOcyZ7OLMH4GFxwPmR0xILBTt8ZnBbKrKDHikVMJoP1f9oEJ2HuQHe 51/w75JFpUkAEQEAAcLBfAQYAQoAJgIbDBYhBN23wrwXzwOts4+cFlDsVI1S4FHABQJld9e9 BQkLmM9yAAoJEFDsVI1S4FHApS8P/2Dovmb516e0PcmYvVoN835cEBgUEQc0lnQAYuNRIPgI CMfNfeZTRdANzyPYYdp+VQCzj95mUt4qlY0PkEGOR1b+2bjeyqp/zAHQShft7FHpsgeRObaW lKE6q9xwUGsOoOcBsWZiwd4xvalIaBG6uczlTrSGJycTF21c35pP6o/eiQs2d/qIVBUMdAtw kefO+pYds1YoyNW2YxGLThWAqNYoThGFsfOqrvLR2HWOcFeneR+PXx2loGdrHE/D6QXpVXxk qRNPig8E68gAVX7q8eRr5+bkJBrG2QvVsc0opiW6AljCJC0gJt5nuHKOObDAmoGx2PuRppT4 +wGd1yCElUZ602Opuf2gqS2nxSypf568EIcoSMEzbJFLrAoy3MReTMoltAJVJU8sA9OXYSOg AjHg7aU/nmO8KFHIZMiW24XJ577iU48UV+u8Rpmv4y16uAhp8AI4wvkk6xa5tNcCWU1Ati6Y yrdk2hf9dD29ePUkHKKSfbewr+qZ9dRL1dUXsaGa64oOXmUALxGNNzj7V828PbWa16x3JbYi gwBNqcgATaAZgDrXMV9ynB4aU1eSNNs/1R0Ic6B7c9yWaI568V7XKldus1QclpELQ4Lu+r/F H065qJMw4LqNs4gEDgd3JzNiNC0Jy+I8mD73YSeo87mE1KOCY0f9DNaBJ+v32E+Z In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Archives-Salt: 8fa43fda-b629-4b5b-bddc-d1432b9b2393 X-Archives-Hash: 017e844b4811d14316eebf1256735a8a Hi Michał, I use llvm-r1 in a few packages, and for the intended purpose of consistently selecting and depending on a specific LLVM I've had no major issues. Overall things work well, and the addition of LLVM_SLOT USE_EXPAND for -r1 has made influencing the selection as an end-user (and developer) so much more straightforward. I don't think that the Rust eclass could work properly without llvm-r1 given how tightly coupled dev-lang/rust is its vendored LLVM version and the issues that we've encountered mixing those. I'm not opposed to any of the options you've presented; they seem reasonable and an improvement over the current situation. At a high level: - option 1: seems to put a lot of the burden on package maintainers to ensure that their build system is set up to support this and may require upstream changes. - option 2: Seems "fine" for CMake based projects, but I have concerns about how other build systems will be catered to; is this something you could elaborate on - how might a non-CMake build system consume more generic variables? Are these widely used/supported and I'm just unaware of it? - option 3: Seems quite straightforward, and I can see this being quite flexible in terms of being called within an ebuild if necessary (though consuming LLVM_SLOT might get ebuilds most of the way there?). Overall perhaps some combination of options 2 and 3 might be the easiest thing for eclass consumers to use flexibly at the cost of additional eclass complexity. I'm interested in how others feel about this. I wonder if there's some space for catering to those packages which (ab)use LLVM_COMPAT as a proxy for 'Only these Clang versions are supported' - usually to get `llvm_gen_dep` for appropriate toolchain components. For www-client/chromium, where we force `CC=clang` because it's the only supported path upstream (and I simply don't have it in me to maintain and GCC patches for three channels a week), I have been stung a few times re: PATH manipulation where, for example, on an ~arch system with multiple LLVM slots installed, and LTO enabled: 1. `CC=clang` is set, then `llvm-r1_pkg_setup` is called. 2. first llvm-r1 fixes CC=clang to CC=clang-19 because that's the latest in PATH. 3. llvm-r1 uses LLVM_SLOT from the profile and does PATH manipulation 4. Compilation proceeds normally, however at link time `lld` is called from the prefixed `/usr/lib/llvm/18/bin` resulting in an error like: '... (Producer: 'LLVM19.1.4' Reader: 'LLVM 18.1.8')` I suspect that this may come up on other systems where `CC=clang` is set via make.conf and LTO is enabled (which is a good argument for avoiding PATH manipulation by default). I've worked around this in Chromium where we now call `llvm-r1_pkg_setup` _then_ set CC and friends to include `LLVM_SLOT` to enable consistent selection of tooling via `llvm_slot_x` USE. I see some value in providing eclass consumers with a mechanism to select appropriate Clang toolchain components consistently, be it an additional variable or some manually-called `clang_setup` function that follows much of the existing LLVM path prefix logic. To play devil's advocate, I admit that Chromium (and maybe Firefox) are probably the only packages to have a _need_ to force a Clang toolchain (due to overheads and the need to get security updates for web browsers to users quickly), and both can continue to do this outside the eclass - it's the "LLVM eclass" not "Clang eclass" after all. I don't really have strong opinions for packages that I maintain; I actually need to go prod an upstream because they still only support LLVM >14, so thanks for the reminder! I'm interested in seeing how others use LLVM in packages and their opinions. Hopefully some of this was useful! Cheers, Matt On 4/12/24 01:32, Michał Górny wrote: > Hello, > > TL;DR: the way llvm/llvm-r1 eclasses currently mangle PATH is broken, > and I'd like to replace that with something better (possibly in llvm- > r2.eclass, given how fragile this thing is). So I'd like to discuss > potential "better" solutions -- and particularly ask you what your LLVM- > using packages need. > > > Background > ========== > > The current logic goes way back to llvm.eclass, and EAPIs that did not > have native cross-build support. Back then, prepending the slotted LLVM > bindir to PATH was the obvious way of getting software to find the right > LLVM version. > > When I added EAPI 7 support, I went for prepending the following thing > to PATH: > > ${ESYSROOT}/usr/lib/llvm/.../bin > > People doing cross will clearly notice the mistake here -- it's using > binaries from ESYSROOT rather than BROOT! Except it's not a mistake, > but an ugly hack. What we're doing here is: > > 1. Relying on a fancy CMake behavior of locating CMake files relative to > PATH, and > > 2. Relying on the package either not caring about LLVM executables or > the system not being able to execute the executables in ESYSROOT > and gracefully falling back to other locations in PATH. > > So what we're really doing is implicitly telling CMake to use: > > ${ESYSROOT}/usr/lib/llvm/.../lib*/cmake > > Yes, it's awful. And yes, it already did backfire in the past, so I've > ended up adding quite a complex logic to prevent these path > manipulations from overriding the toolchain set by user. For example, > if the user has CC=clang, that normally evalutes to clang-19, we now > adjust CC so that it suddenly doesn't switch to clang-17 because > the package uses libLLVM-17. Meh. > > When working on llvm-r1, I've focused on the more immediate problem of > horribly complex and broken package dependencies, and forgot about this. > I've only recalled the problem during the initial rust.eclass reviews, > since it happened to copy that incorrect logic. > > > Future options > ============== > > Some of the options that already popped up during discussions include: > > 1. Stopping to export pkg_setup() entirely, and expecting people to > explicitly pass the LLVM path to the build system, e.g. something like: > > -DLLVM_CMAKE_PATH="$(get_llvm_prefix -d)" > > 2. Setting specific environment variables (such as LLVM_ROOT, CLANG_ROOT > and so on for CMake, or perhaps CMAKE_PREFIX_PATH). > > 3. Creating a minimal llvm-config wrapper in ${T}, and adding it to > ${PATH} instead of the whole LLVM tree. Note that we'd need to write > our own since llvm-config is an executable, so we can't run the one from > ESYSROOT, and we can't rely on BROOT having a match (or don't want to > force a second copy of LLVM unnecessarily). > > Any other ideas? How does your package select LLVM version, and which > of these options would work best for you? > >