Re: [gentoo-dev] Need GitHub snapshot hash verification failure samples

[Sergei Trofimovich]

On Wed, 05 Jul 2017 21:48:12 +0200
Michał Górny <mgorny@g.o> wrote:

> Hi, everyone.
>
> I've seen multiple bugs related to hash verification failures for GitHub
> snapshots lately. However, none of the maintainers have been so far able
> to provide me with a sample of the old and new snapshot for comparison,
> so we still have no clue what's happening exactly.
>
> if you see your package failing or get a report for it, then *please*
> save the original tarball before replacing it with the new one and send
> me both for comparison. Thank you.

Sounds easy to verify.
1. grab all the github tarballs (should be a better way to do it with proper USE expansiion):
    $ egrep -R 'SRC_URI.*github.com' metadata/ | grep -o '[^/ ]*$' | sort -u > github_distfiles.list
2. grab all manifest files that look like defining these files and remove them locally:
    $ git grep -l -F -f ./github_distfiles.list | grep -F /Manifest | xargs rm -v
3. Refetch distfiles from internets:
    $ mkdir /tmp/fresh
    $ GENTOO_MIRRORS= DISTDIR=/tmp/fresh repoman manifest

As a result each 'git diff' report is your potential candidate.
You have new file in /tmp/fresh/<file>
and old one on http://distfiles.gentoo.org/distfiles/<file>

A few samples:
--- a/app-admin/qtpass/Manifest
+++ b/app-admin/qtpass/Manifest
@@ -1,4 +1,4 @@
-DIST qtpass-1.0.5.tar.gz 636461 SHA256 0c07bd1eb9e5336c0225f891e5b9a9df103f218619cf7ec6311edf654e8db281
-DIST qtpass-1.1.0.tar.gz 671525 SHA256 60b458062f54184057e55dbd9c93958a8bf845244ffd70b9cb31bf58697f0dc6
+DIST qtpass-1.0.5.tar.gz 636457 SHA256 b9f1c1ecf4afbe716915792ff692e7114568de5bd8c47750d5c8404aa28699e7
+DIST qtpass-1.1.0.tar.gz 671537 SHA256 f2fff7922902c4c118e04164c078ca80e9a28221320b4253d3117d885e8417b6

diffoscope reports case change only in root dir name:

$ diffoscope old/qtpass-1.1.0.tar.gz new/qtpass-1.1.0.tar.gz
│   │ @@ -1,83 +1,83 @@
│   │ -drwxrwxr-x   0 root         (0) root         (0)        0 2016-01-25 09:58:18.000000 qtpass-1.1.0/
│   │ +drwxrwxr-x   0 root         (0) root         (0)        0 2016-01-25 09:58:18.000000 QtPass-1.1.0/
...

I guess somebody decided to rename github repo slightly.

Both files are at:

http://dev.gentoo.org/~slyfox/unstable_tarballs/old/qtpass-1.1.0.tar.gz
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/qtpass-1.1.0.tar.gz

--- a/app-crypt/acme/Manifest
+++ b/app-crypt/acme/Manifest
@@ -1,3 +1,3 @@
 DIST certbot-0.14.1.tar.gz 851705 SHA256 7992fced742649e7b7668e4db7685de12248a4ffba66810cb336e9b6412e3567
 DIST certbot-0.15.0.tar.gz 942788 SHA256 87d306b1c013b472b8f548b38ccc476c125816435bb3b99e932fed09ac777296
-DIST letsencrypt-0.1.0.tar.gz 524821 SHA256 1c1ac7b41e5e0fc0e41a7ef159ac9147a4aafff54453d57b519eb05bf52ade14
+DIST letsencrypt-0.1.0.tar.gz 524854 SHA256 3ba1add217fc1665ad1d3c4812c0de60590f406cb83d6514332898ab60b26f62

$ diffoscope old/letsencrypt-0.1.0.tar.gz new/letsencrypt-0.1.0.tar.gz
│   │ @@ -1,579 +1,579 @@
│   │ -drwxrwxr-x   0 root         (0) root         (0)        0 2015-12-02 23:55:43.000000 letsencrypt-0.1.0/
│   │ +drwxrwxr-x   0 root         (0) root         (0)        0 2015-12-02 23:55:43.000000 certbot-0.1.0/

Same thing.


http://dev.gentoo.org/~slyfox/unstable_tarballs/old/letsencrypt-0.1.0.tar.gz
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/letsencrypt-0.1.0.tar.gz

Zip file!

--- a/app-crypt/etcd-ca/Manifest
+++ b/app-crypt/etcd-ca/Manifest
@@ -1,2 +1,2 @@
-DIST etcd-ca-0_p20140903.zip 1178338 SHA256 5da9f7afad6dd373d96c5d36dd30e9f43cfc8fc2359bbf2d0c6a864fff139f81
+DIST etcd-ca-0_p20140903.zip 1178338 SHA256 7ef6b7f34324bd4b48b369990a7eb70e30809240f3c3d97b7d56d021af3f43f3

$ diffoscope old/etcd-ca-0_p20140903.zip new/etcd-ca-0_p20140903.zip
│  drwx---     0.0 fat        0 bx stor 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/
│ --rw----     0.0 fat       24 bx stor 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/.gitconfig
│ --rw----     0.0 fat     3924 bx defN 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/CONTRIBUTING.md
│ +-rw----     0.0 fat       24 tx stor 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/.gitconfig
│ +-rw----     0.0 fat     3924 tx defN 14-Sep-03 21:30 etcd-ca-812f3626796be16d9db052720ce9c54f5a40bb26/CONTRIBUTING.md

Here contents didn't change but zip compressor decided to pick different file type (bx/tx is binary/text).

http://dev.gentoo.org/~slyfox/unstable_tarballs/old/etcd-ca-0_p20140903.zip
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/etcd-ca-0_p20140903.zip

--- a/app-emacs/lua-mode/Manifest
+++ b/app-emacs/lua-mode/Manifest
@@ -1 +1 @@
-DIST lua-mode-20130419.tar.gz 26236 SHA256 75c1696421983fbb58946ea649d2917f0deefc8b4f1dbc16b819e0cd603e396a
+DIST lua-mode-20130419.tar.gz 26242 SHA256 7a5e1a21e53aeab6e7cad8c616f6b026fd32f414bc6a32371e04d4e7424800c7

This one is different. Tag expansion changed (on GitHub's side?):

$ diffoscope old/lua-mode-20130419.tar.gz new/lua-mode-20130419.tar.gz  | lv

│   ├── lua-mode-rel-20130419/lua-mode.el
│   │ @@ -31,15 +31,15 @@
│   │  ;; Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
│   │  ;; MA 02110-1301, USA.
│   │  
│   │  ;; Keywords: languages, processes, tools
│   │  
│   │  ;; This field is expanded to commit SHA, date & associated heads/tags during
│   │  ;; archive creation.
│   │ -;; Revision: 040bc8f (Fri, 19 Apr 2013 11:27:32 +0400  (rel-20130419))
│   │ +;; Revision: 040bc8f (Fri, 19 Apr 2013 11:27:32 +0400  (tag: rel-20130419))
│   │  ;;
│   │  
│   │  ;;; Commentary:
│   │  
│   │  ;; Thanks to d87 <github.com/d87> for an idea of highlighting lua
│   │  ;; builtins/numbers
│   │  
│   ╵

http://dev.gentoo.org/~slyfox/unstable_tarballs/old/lua-mode-20130419.tar.gz
http://dev.gentoo.org/~slyfox/unstable_tarballs/new/lua-mode-20130419.tar.gz

--- a/app-emulation/docker/Manifest
+++ b/app-emulation/docker/Manifest
@@ -1,3 +1,3 @@
-DIST docker-17.03.1.tar.gz 7773296 SHA256 a8f1eefadf3966885ad0579facfc2017cca7dd3a0b20d086dfd798168716cb83
+DIST docker-17.03.1.tar.gz 7773988 SHA256 411e32ee388ad6d99479b97a3937c851bd84dacf4267be9d5501665e468e148e

$ diffoscope old/docker-17.03.1.tar.gz new/docker-17.03.1.tar.gz


-- 

  Sergei