Gentoo Archives: gentoo-dev

From: William Hubbs <williamh@g.o>
To: gentoo-dev@l.g.o
Cc: William Hubbs <williamh@g.o>
Subject: [gentoo-dev] [PATCH v2] go-module.eclass: deprecate EGO_SUM and call ego instead of go
Date: Sun, 27 Feb 2022 04:39:13
Message-Id: 20220227043833.31274-1-williamh@gentoo.org
1 EGO_SUM can be thousands of lines long in ebuilds, and it leads to
2 creating Manifest files that are thousands of lines long.
3 It has been determined that vendor tarballs are a better solution if
4 upstream doesn't vendor their dependencies.
5
6 Also, call the ego helper function instead of calling go directly.
7
8 Signed-off-by: William Hubbs <williamh@g.o>
9 ---
10 eclass/go-module.eclass | 144 +++++++++++-----------------------------
11 1 file changed, 38 insertions(+), 106 deletions(-)
12
13 diff --git a/eclass/go-module.eclass b/eclass/go-module.eclass
14 index 66fe52c9ad7..b9686ad555a 100644
15 --- a/eclass/go-module.eclass
16 +++ b/eclass/go-module.eclass
17 @@ -25,9 +25,26 @@
18 #
19 # If the software has a directory named vendor in its
20 # top level directory, the only thing you need to do is inherit the
21 -# eclass. Otherwise, you need to also populate
22 -# EGO_SUM and call go-module_set_globals as discussed below.
23 +# eclass. If it doesn't, you need to also create a vendor tarball and
24 +# host it somewhere, for example in your dev space.
25 #
26 +# Here are the commands to create a vendor tarball.
27 +#
28 +# @CODE
29 +#
30 +# $ cd /path/to/project
31 +# $ go mod vendor
32 +# $ tar -acf project-1.0-vendor.tar.xz vendor --xform 's:^":project-1.0/:'
33 +#
34 +# @CODE
35 +#
36 +# Note that 'go mod vendor' drops files from the dependency tree, and
37 +# this causes errors with some builds. If your build crashes with errors
38 +# like the ones mentioned in the below issue, you will need to emerge
39 +# dev-go/vend and replace 'go mod vendor' above with 'vend'.
40 +#
41 +# https://github.com/golang/go/issues/26366
42 +#
43 # Since Go programs are statically linked, it is important that your ebuild's
44 # LICENSE= setting includes the licenses of all statically linked
45 # dependencies. So please make sure it is accurate.
46 @@ -40,15 +57,9 @@
47 #
48 # inherit go-module
49 #
50 -# EGO_SUM=(
51 -# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59/go.mod"
52 -# "github.com/aybabtme/rgbterm v0.0.0-20170906152045-cc83f3b3ce59"
53 -# )
54 -#
55 -# go-module_set_globals
56 -#
57 -# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz
58 -# ${EGO_SUM_SRC_URI}"
59 +# SRC_URI="https://github.com/example/${PN}/archive/v${PV}.tar.gz -> ${P}.tar.gz"
60 +# Do not add the next line if you don't have a vendor tarball
61 +# SRC_URI+="${P}-vendor.tar.gz"
62 #
63 # @CODE
64
65 @@ -100,72 +111,6 @@ QA_FLAGS_IGNORED='.*'
66 # Go packages should not be stripped with strip(1).
67 RESTRICT+=" strip"
68
69 -# @ECLASS-VARIABLE: EGO_SUM
70 -# @DESCRIPTION:
71 -# This array is based on the contents of the go.sum file from the top
72 -# level directory of the software you are packaging. Each entry must be
73 -# quoted and contain the first two fields of a line from go.sum.
74 -#
75 -# You can use some combination of sed/awk/cut to extract the
76 -# contents of EGO_SUM or use the dev-go/get-ego-vendor tool.
77 -#
78 -# One manual way to do this is the following:
79 -#
80 -# @CODE
81 -#
82 -# cat go.sum | cut -d" " -f1,2 | awk '{print "\t\"" $0 "\""}'
83 -#
84 -# @CODE
85 -#
86 -# The format of go.sum is described upstream here:
87 -# https://go.dev/ref/mod#go-sum-files
88 -#
89 -# For inclusion in EGO_SUM, the h1: value and other future extensions SHOULD be
90 -# omitted at this time. The EGO_SUM parser will accept them for ease of ebuild
91 -# creation.
92 -#
93 -# h1:<hash> is the Hash1 structure used by upstream Go
94 -# The Hash1 is MORE stable than Gentoo distfile hashing, and upstream warns
95 -# that it's conceptually possible for the Hash1 value to remain stable while
96 -# the upstream zipfiles change. Here are examples that do NOT change the h1:
97 -# hash, but do change a regular checksum over all bytes of the file:
98 -# - Differing mtimes within zipfile
99 -# - Differing filename ordering with the zipfile
100 -# - Differing zipfile compression parameters
101 -# - Differing zipfile extra fields
102 -#
103 -# For Gentoo usage, the authors of this eclass feel that the h1: hash should
104 -# NOT be included in the EGO_SUM at this time in order to reduce size of the
105 -# ebuilds. This position will be reconsidered in future when a Go module
106 -# distfile collision comes to light, where the Hash1 value of two distfiles is
107 -# the same, but checksums over the file as a byte stream consider the files to
108 -# be different.
109 -#
110 -# This decision does NOT weaken Go module security, as Go will verify the
111 -# go.sum copy of the Hash1 values during building of the package.
112 -
113 -# @ECLASS-VARIABLE: _GOMODULE_GOPROXY_BASEURI
114 -# @DESCRIPTION:
115 -# Golang module proxy service to fetch module files from. Note that the module
116 -# proxy generally verifies modules via the Hash1 code.
117 -#
118 -# Users in China may find some mirrors in the default list blocked, and should
119 -# explicitly set an entry in /etc/portage/mirrors for goproxy to
120 -# https://goproxy.cn/ or another mirror that is not blocked in China.
121 -# See https://arslan.io/2019/08/02/why-you-should-use-a-go-module-proxy/ for
122 -# further details
123 -#
124 -# This variable is NOT intended for user-level configuration of mirrors, but
125 -# rather to cover go modules that might exist only on specific Goproxy
126 -# servers for non-technical reasons.
127 -#
128 -# This variable should NOT be present in user-level configuration e.g.
129 -# /etc/portage/make.conf, as it will violate metadata immutability!
130 -#
131 -# I am considering removing this and just hard coding mirror://goproxy
132 -# below, so please do not rely on it.
133 -: "${_GOMODULE_GOPROXY_BASEURI:=mirror://goproxy/}"
134 -
135 # @ECLASS-VARIABLE: _GOMODULE_GOSUM_REVERSE_MAP
136 # @DESCRIPTION:
137 # Mapping back from Gentoo distfile name to upstream distfile path.
138 @@ -181,7 +126,7 @@ declare -A -g _GOMODULE_GOSUM_REVERSE_MAP
139 # ebuild will be considered optional. No dependencies will be added and
140 # no phase functions will be exported.
141 #
142 -# If you enable GO_OPTIONAL, you have to set BDEPEND on >=dev-lang/go-1.12
143 +# If you enable GO_OPTIONAL, you have to set BDEPEND on >=dev-lang/go-1.16
144 # for your package and call go-module_src_unpack manually.
145
146 # @FUNCTION: ego
147 @@ -199,11 +144,7 @@ ego() {
148
149 # @FUNCTION: go-module_set_globals
150 # @DESCRIPTION:
151 -# Convert the information in EGO_SUM for other usage in the ebuild.
152 -# - Populates EGO_SUM_SRC_URI that can be added to SRC_URI
153 -# - Exports _GOMODULE_GOSUM_REVERSE_MAP which provides reverse mapping from
154 -# distfile back to the relative part of SRC_URI, as needed for
155 -# GOPROXY=file:///...
156 +# This function is deprecated.
157 go-module_set_globals() {
158 local line exts
159 # for tracking go.sum errors
160 @@ -256,7 +197,7 @@ go-module_set_globals() {
161 # Relative URI within a GOPROXY for a file
162 _reluri="${_dir}/@v/${version}.${_ext}"
163 # SRC_URI: LHS entry
164 - _uri="${_GOMODULE_GOPROXY_BASEURI}/${_reluri}"
165 + _uri="mirror://goproxy/${_reluri}"
166 # _uri="mirror://goproxy/${_reluri}"
167 # SRC_URI: RHS entry, encode any slash in the path as
168 # %2F in the filename
169 @@ -289,9 +230,7 @@ go-module_set_globals() {
170
171 # @FUNCTION: go-module_setup_proxy
172 # @DESCRIPTION:
173 -# If your ebuild redefines src_unpack and uses EGO_SUM you need to call
174 -# this function in src_unpack.
175 -# It sets up the go module proxy in the appropriate location.
176 +# This function is deprecated.
177 go-module_setup_proxy() {
178 # shellcheck disable=SC2120
179 debug-print-function "${FUNCNAME}" "$@"
180 @@ -331,11 +270,15 @@ go-module_setup_proxy() {
181 # @FUNCTION: go-module_src_unpack
182 # @DESCRIPTION:
183 # If EGO_SUM is set, unpack the base tarball(s) and set up the
184 -# local go proxy.
185 +# local go proxy. Also warn that this usage is deprecated.
186 # - Otherwise, if EGO_VENDOR is set, bail out.
187 -# - Otherwise do a normal unpack.
188 +# - Otherwise do a normal unpack and move the vendor directory into
189 +# place if appropriate.
190 go-module_src_unpack() {
191 if [[ "${#EGO_SUM[@]}" -gt 0 ]]; then
192 + eqawarn "This ebuild uses EGO_SUM which is deprecated"
193 + eqawarn "Please migrate to a vendor tarball"
194 + eqawarn "This will become a fatal error in the future"
195 _go-module_src_unpack_gosum
196 elif [[ "${#EGO_VENDOR[@]}" -gt 0 ]]; then
197 eerror "${EBUILD} is using EGO_VENDOR which is no longer supported"
198 @@ -347,11 +290,7 @@ go-module_src_unpack() {
199
200 # @FUNCTION: _go-module_src_unpack_gosum
201 # @DESCRIPTION:
202 -# Populate a GOPROXY directory hierarchy with distfiles from EGO_SUM and
203 -# unpack the base distfiles.
204 -#
205 -# Exports GOPROXY environment variable so that Go calls will source the
206 -# directory correctly.
207 +# This function is deprecated.
208 _go-module_src_unpack_gosum() {
209 # shellcheck disable=SC2120
210 debug-print-function "${FUNCNAME}" "$@"
211 @@ -392,10 +331,7 @@ _go-module_src_unpack_gosum() {
212
213 # @FUNCTION: _go-module_gosum_synthesize_files
214 # @DESCRIPTION:
215 -# Given a path & version, populate all Goproxy metadata files which aren't
216 -# needed to be downloaded directly.
217 -# - .../@v/${version}.info
218 -# - .../@v/list
219 +# This function is deprecated.
220 _go-module_gosum_synthesize_files() {
221 local target=$1
222 local version=$2
223 @@ -419,8 +355,7 @@ _go-module_gosum_synthesize_files() {
224
225 # @FUNCTION: _go-module_src_unpack_verify_gosum
226 # @DESCRIPTION:
227 -# Validate the Go modules declared by EGO_SUM are sufficient to cover building
228 -# the package, without actually building it yet.
229 +# This function is deprecated.
230 _go-module_src_unpack_verify_gosum() {
231 # shellcheck disable=SC2120
232 debug-print-function "${FUNCNAME}" "$@"
233 @@ -435,7 +370,7 @@ _go-module_src_unpack_verify_gosum() {
234 # This will print 'downloading' messages, but it's accessing content from
235 # the $GOPROXY file:/// URL!
236 einfo "Tidying go.mod/go.sum"
237 - go mod tidy >/dev/null
238 + ego mod tidy >/dev/null
239
240 # This used to call 'go get' to verify by fetching everything from the main
241 # go.mod. However 'go get' also turns out to recursively try to fetch
242 @@ -461,16 +396,13 @@ go-module_live_vendor() {
243 die "${FUNCNAME} only allowed when upstream isn't vendoring"
244
245 pushd "${S}" >& /dev/null || die
246 - go mod vendor || die
247 + ego mod vendor
248 popd >& /dev/null || die
249 }
250
251 # @FUNCTION: _go-module_gomod_encode
252 # @DESCRIPTION:
253 -# Encode the name(path) of a Golang module in the format expected by Goproxy.
254 -#
255 -# Upper letters are replaced by their lowercase version with a '!' prefix.
256 -#
257 +# This function is deprecated.
258 _go-module_gomod_encode() {
259 ## Python:
260 # return re.sub('([A-Z]{1})', r'!\1', s).lower()
261 --
262 2.34.1

Replies