Gentoo Archives: gentoo-dev

From: "Andreas K. Hüttel" <dilfridge@g.o>
To: gentoo-dev@l.g.o
Subject: [gentoo-dev] Fwd: [JuliaLang] Pkg downtime incident
Date: Tue, 04 Aug 2020 22:11:57
Message-Id: 1806686.b9uPGUboIS@farino
1 And this is why one should't depend on external services for critical stuff.
2 Beautiful example.
4 ---------- Weitergeleitete Nachricht ----------
6 Betreff: [JuliaLang] Pkg downtime incident
7 Datum: Mittwoch, 5. August 2020, 01:08:13 EEST
8 Von: Keno Fischer via JuliaLang <julialang@×××××××××××××.com>
9 An: huettel@×××××.com
14 Earlier today, several users started seeing issues installing packages. This
15 post seeks to collect all the information related to this incident.
17 # Impact
19 The issue caused installation of incorrect versions (latest master when a
20 prior version was requested) of packages.
21 - Versions of Julia prior to 1.4 will silently install the wrong version
22 - Windows versions of Julia 1.4.x will also silently install the wrong version
23 - Non-Windows versions of Julia 1.4.x will issue a warning and fall back to
24 git to obtain the correct version
25 - Julia 1.5 is unaffected when using the pkg server (which is the default),
26 otherwise matches 1.4 behavior
28 The issue has since been mitigated in the registry, so you were only affected
29 if you were attempting package operations on an affected version between
30 approximately 2pm Eastern and 3:43pm Eastern when the mitigation went into
31 effect.
33 # Symptoms
35 Installing the wrong version of a Julia package can cause incorrect behavior
36 in several different ways. Perhaps the most common will be inscrutable package
37 dependency errors, but more subtle behaviors are possible. If you performed a
38 package operation today, you may want to see the mitigation section below as a
39 precaution.
41 # Mitigation
43 If an incorrect package version was installed, it will be locally cached until
44 removed. As such, if you believe you were affected, it is advisable to clear
45 your package cache by deleting `.julia/packages`. Note that your list of
46 installed packages will not be affected and you may re-download all installed
47 packages in your current environment by using `Pkg.instantiate()`.
49 # Root cause
51 The root cause of this change was an unannounced serverside change by GitHub,
52 which broke download of tarballs by git-tree-hash, e.g. previously https://
54 2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 would give the tarball for that tree-
55 hash, while it now gives the tarball for master instead. We do not yet know
56 whether this change was intentional or not. The reason this change broke Pkg
57 is that Pkg includes a heuristic where it will use the tarball download
58 feature instead of a full git checkout as faster way to download a requested
59 version (since it no longer needs to download the full repository with all its
60 history). This was special cased for and does not affect packages
61 hosted elsewhere (though the vast majority of packages are currently hosted on
62 GitHub).
64 # Registry workaround
66 The above mentioned workaround was
67 pull/18991/files, which changes the URL for all registered packages from
68 `` to ``. This breaks above mentioned heuristic and will
69 force older versions of Julia to fall back to a full git checkout instead.
70 This method is slower, but should yield the correct package version. Note that
71 Julia 1.5+ is unaffected and downloads via the Pkg server will continue to be
72 fast.
74 # Additional considerations/General registry updates paused
76 We have contacted GitHub to find out whether this change was intentional and
77 is likely to persist. If so, we will need to update Registrator and the
78 validation CI to force packages registered at GitHub to use the same
79 `` workaround we manually applied to the registry. If not, the
80 workaround will be reverted as soon as GitHub restores the original behavior
81 (to get back to faster package download speeds on older versions). In the
82 meantime changes (new packages/version bumps) to the General registry are
83 paused. They will be resumed once either of the two options have been
84 completed.
86 # Future considerations
88 As noted, Julia versions 1.5+ are not affected due to the Pkg server work
89 (which was partly motivated by a desire to avoid incidents like this once).
90 However, such Julia versions will still fall back to raw GitHub downloads if
91 the package server is unavailable for some reason (broken, blocked by
92 corporate firewall, we forgot to pay our bills, etc.). In the near future, the
93 validation currently present on non-Windows versions, will be extended to
94 Windows version, such that even with a broken package server, the fall back
95 path would itself fallback to Git if it is being served incorrect tarballs
96 (the same verification will of course extend to the package server also). This
97 change has been planned for some time and the requisite support is already
98 available in Tar.jl, but has not yet been wired up in Pkg.
104 ---
105 [Visit Topic](
106 or reply to this email to respond.
108 To unsubscribe from these emails, [click here](https://
110 5dcc8fe0a5dab8380516e5d33481407163067880c0e37e4db5d9c1772dabf1d2).
112 -------------------------------------------------------------
113 --
114 Andreas K. Hüttel
115 dilfridge@g.o
116 Gentoo Linux developer
117 (council, qa, toolchain, base-system, perl, libreoffice)


File name MIME type
signature.asc application/pgp-signature