1 |
And this is why one should't depend on external services for critical stuff. |
2 |
Beautiful example. |
3 |
|
4 |
---------- Weitergeleitete Nachricht ---------- |
5 |
|
6 |
Betreff: [JuliaLang] Pkg downtime incident |
7 |
Datum: Mittwoch, 5. August 2020, 01:08:13 EEST |
8 |
Von: Keno Fischer via JuliaLang <julialang@×××××××××××××.com> |
9 |
An: huettel@×××××.com |
10 |
|
11 |
|
12 |
|
13 |
|
14 |
Earlier today, several users started seeing issues installing packages. This |
15 |
post seeks to collect all the information related to this incident. |
16 |
|
17 |
# Impact |
18 |
|
19 |
The issue caused installation of incorrect versions (latest master when a |
20 |
prior version was requested) of packages. |
21 |
- Versions of Julia prior to 1.4 will silently install the wrong version |
22 |
- Windows versions of Julia 1.4.x will also silently install the wrong version |
23 |
- Non-Windows versions of Julia 1.4.x will issue a warning and fall back to |
24 |
git to obtain the correct version |
25 |
- Julia 1.5 is unaffected when using the pkg server (which is the default), |
26 |
otherwise matches 1.4 behavior |
27 |
|
28 |
The issue has since been mitigated in the registry, so you were only affected |
29 |
if you were attempting package operations on an affected version between |
30 |
approximately 2pm Eastern and 3:43pm Eastern when the mitigation went into |
31 |
effect. |
32 |
|
33 |
# Symptoms |
34 |
|
35 |
Installing the wrong version of a Julia package can cause incorrect behavior |
36 |
in several different ways. Perhaps the most common will be inscrutable package |
37 |
dependency errors, but more subtle behaviors are possible. If you performed a |
38 |
package operation today, you may want to see the mitigation section below as a |
39 |
precaution. |
40 |
|
41 |
# Mitigation |
42 |
|
43 |
If an incorrect package version was installed, it will be locally cached until |
44 |
removed. As such, if you believe you were affected, it is advisable to clear |
45 |
your package cache by deleting `.julia/packages`. Note that your list of |
46 |
installed packages will not be affected and you may re-download all installed |
47 |
packages in your current environment by using `Pkg.instantiate()`. |
48 |
|
49 |
# Root cause |
50 |
|
51 |
The root cause of this change was an unannounced serverside change by GitHub, |
52 |
which broke download of tarballs by git-tree-hash, e.g. previously https:// |
53 |
api.github.com/repos/JuliaLang/MbedTLS.jl/tarball/ |
54 |
2d94286a9c2f52c63a16146bb86fd6cdfbf677c6 would give the tarball for that tree- |
55 |
hash, while it now gives the tarball for master instead. We do not yet know |
56 |
whether this change was intentional or not. The reason this change broke Pkg |
57 |
is that Pkg includes a heuristic where it will use the tarball download |
58 |
feature instead of a full git checkout as faster way to download a requested |
59 |
version (since it no longer needs to download the full repository with all its |
60 |
history). This was special cased for github.com and does not affect packages |
61 |
hosted elsewhere (though the vast majority of packages are currently hosted on |
62 |
GitHub). |
63 |
|
64 |
# Registry workaround |
65 |
|
66 |
The above mentioned workaround was https://github.com/JuliaRegistries/General/ |
67 |
pull/18991/files, which changes the URL for all registered packages from |
68 |
`github.com` to `GitHub.com`. This breaks above mentioned heuristic and will |
69 |
force older versions of Julia to fall back to a full git checkout instead. |
70 |
This method is slower, but should yield the correct package version. Note that |
71 |
Julia 1.5+ is unaffected and downloads via the Pkg server will continue to be |
72 |
fast. |
73 |
|
74 |
# Additional considerations/General registry updates paused |
75 |
|
76 |
We have contacted GitHub to find out whether this change was intentional and |
77 |
is likely to persist. If so, we will need to update Registrator and the |
78 |
validation CI to force packages registered at GitHub to use the same |
79 |
`GitHub.com` workaround we manually applied to the registry. If not, the |
80 |
workaround will be reverted as soon as GitHub restores the original behavior |
81 |
(to get back to faster package download speeds on older versions). In the |
82 |
meantime changes (new packages/version bumps) to the General registry are |
83 |
paused. They will be resumed once either of the two options have been |
84 |
completed. |
85 |
|
86 |
# Future considerations |
87 |
|
88 |
As noted, Julia versions 1.5+ are not affected due to the Pkg server work |
89 |
(which was partly motivated by a desire to avoid incidents like this once). |
90 |
However, such Julia versions will still fall back to raw GitHub downloads if |
91 |
the package server is unavailable for some reason (broken, blocked by |
92 |
corporate firewall, we forgot to pay our bills, etc.). In the near future, the |
93 |
validation currently present on non-Windows versions, will be extended to |
94 |
Windows version, such that even with a broken package server, the fall back |
95 |
path would itself fallback to Git if it is being served incorrect tarballs |
96 |
(the same verification will of course extend to the package server also). This |
97 |
change has been planned for some time and the requisite support is already |
98 |
available in Tar.jl, but has not yet been wired up in Pkg. |
99 |
|
100 |
|
101 |
|
102 |
|
103 |
|
104 |
--- |
105 |
[Visit Topic](https://discourse.julialang.org/t/pkg-downtime-incident/44288/1) |
106 |
or reply to this email to respond. |
107 |
|
108 |
To unsubscribe from these emails, [click here](https:// |
109 |
discourse.julialang.org/email/unsubscribe/ |
110 |
5dcc8fe0a5dab8380516e5d33481407163067880c0e37e4db5d9c1772dabf1d2). |
111 |
|
112 |
------------------------------------------------------------- |
113 |
-- |
114 |
Andreas K. Hüttel |
115 |
dilfridge@g.o |
116 |
Gentoo Linux developer |
117 |
(council, qa, toolchain, base-system, perl, libreoffice) |