Gentoo Archives: gentoo-user

From: Miroslav Rovis <miro.rovis@××××××××××××××.hr>
To: gentoo-user@l.g.o
Subject: [gentoo-user] Inconsistent behavior in my Gentoo OS instance
Date: Mon, 01 May 2017 16:18:19
Message-Id: 20170501161736.GA28651@g0n.xdwgrp
1 Hi!
2
3 I have tried hard to make this message to the point. But that wasn't easy, as
4 what I figured out about gzip --only after hours of work-- I still do need
5 confirmation about, even though all appears correct.
6
7 gzip apparently inconsistent behavior occupies the most part of the report on
8 inconsistencies here (esp. the script make_gzip_archives_consistent.sh).
9
10 Another part is actually on Wireshark mailing list. Pls. see:
11
12 Filtering on (negated) frame.time_relative filters out wrong frame.number
13 https://www.wireshark.org/lists/wireshark-users/201704/msg00037.html
14 as well as my study at:
15 https://www.croatiafidelis.hr/foss/cap/cap-170313-git-devuan-mail/git-devuan-mail-4.php
16 (and the previous ones there, but I gave the last as it is simplest/fastest to check)
17
18 There is information that any advanced reader can easily provide by retracing
19 some of my steps there, and which would clear some uncertainties here.
20
21 The third part is about Bash apparent misbehavior.
22
23 And the fourth part is about eix complaining it has a bug, it's near the bottom part
24 of this email.
25
26 That is my starting point, and at this stage, after having spent quite a few
27 hours on this issue below (but I have other issues not-directly-related to my system
28 that haven't allowed me to finish and send this yet)...
29
30 This issue below I thought at first could be related to my Wireshark woes, and
31 now I don't... and at this stage (the proofreading time) I'm actually asking for
32 any confirmation/denial on my findings about it...
33
34 miro@g0n ~ $ ls -lh wireshark-users_miroR_q_bug_q.txt
35 -rw-r--r-- 1 miro miro 802 2017-04-30 17:16 wireshark-users_miroR_q_bug_q.txt
36 miro@g0n ~ $ ls -l --block-size=K wireshark-users_miroR_q_bug_q.txt
37 -rw-r--r-- 1 miro miro 1K 2017-04-30 17:16 wireshark-users_miroR_q_bug_q.txt
38 miro@g0n ~ $ mv -vi wireshark-users_miroR_q_bug_q.txt gzip_buggy.txt
39 'wireshark-users_miroR_q_bug_q.txt' -> 'gzip_buggy.txt'
40
41 miro@g0n ~ $ tar czf gzip_buggy.txt.tar.gz gzip_buggy.txt
42 miro@g0n ~ $ tar czf gzip_buggy.txt_1.tar.gz gzip_buggy.txt
43 miro@g0n ~ $ tar czf gzip_buggy.txt_2.tar.gz gzip_buggy.txt
44 miro@g0n ~ $ sha256sum gzip_buggy.txt*.tar.gz
45 26bcad15b4a81a3606859fdd4560af0e2c4cd875f1271808afb4569c50bfb5ec gzip_buggy.txt_1.tar.gz
46 bd8f51bcd3274bbee4f73fa7ba9e1a2acd5db5d27f5643f72de10c52e224c22a gzip_buggy.txt_2.tar.gz
47 251a57e83ebaf4f07179f4a6500be80612a39ac6d2e97e482b30fb0a4de66f56 gzip_buggy.txt.tar.gz
48 miro@g0n ~ $
49
50 You may check attachments:
51 gzip_buggy.txt_1.tar.gz
52 gzip_buggy.txt_2.tar.gz
53 gzip_buggy.txt.tar.gz
54
55 I think this is not like it should be. But the important question, really, is:
56
57 is this the case in my machine only, or is it Gentoo-userdom wide?
58
59 I fear it's the former, but like I asked in that Wireshark thread
60 (
61 And it was amazing to see how little people care... there are always plenty of
62 readers of Wireshark ML, and while not all can understand that thread --it
63 took me years since I first used Wireshark to get to be able to minimally
64 contribute there... lots of readers there such--, and of course those that can
65 not really understand, need not reply... but lots of readers of Wireshark ML
66 can easily repeat my steps that show in which exact fashion my Wireshark
67 misbehaves...
68
69 I'll be thankful is some Gentoo reader fills in this huge gap of
70 carelessness..
71 )
72 [but like I asked in that Wireshark thread,] I ask here on Gentoo users ML (or
73 readers from elsewhere).
74
75 Pls. somebody confirm if this is, or is not, the case in their Gentoo
76 machines. By simply decompressing the tar-gzip'd archive are do the tar-gzip
77 compression as I did, and checking the sums of them.
78
79 If you don't like my text file, you can use any file (like the ASCII file
80 below, which I hope should be simple and fine enough), and reproduce what I
81 tried with it.
82
83 So let's try /usr/bin/eix-installed-after
84 It's ASCII, it's Gentoo, everybody has it on this ML, but for visitors, it's in
85
86 You may check attachments:
87
88 eix-installed-after_1.tar.gz
89 eix-installed-after_2.tar.gz
90 eix-installed-after.tar.gz
91
92 In my Gentoo, I copied it in my homedir and...
93
94 miro@g0n ~ $ tar czf eix-installed-after.tar.gz eix-installed-after
95 miro@g0n ~ $ tar czf eix-installed-after_1.tar.gz eix-installed-after
96 miro@g0n ~ $ tar czf eix-installed-after_2.tar.gz eix-installed-after
97 miro@g0n ~ $ sha256sum eix-installed-after*.tar.gz
98 80d004b1ecc6e6c2aec7cd453d05c7b4e4cc4092b9cef6b7c3fb31d148eb2310 eix-installed-after_1.tar.gz
99 f4c30c6d4ccc90d1ec2cc6df2ac10cae2545d04a85c7ff39207d93129cd0ec78 eix-installed-after_2.tar.gz
100 a8161c5771b661476966250af31d2b8bd80581a3e0c52d535b401524ee541ae7 eix-installed-after.tar.gz
101 miro@g0n ~ $
102
103 And let me go further. Inspecting those...:
104
105 miro@g0n ~ $ ls -l eix-installed-after*.tar.gz
106 -rw-r--r-- 1 miro miro 3771 2017-04-30 17:39 eix-installed-after_1.tar.gz
107 -rw-r--r-- 1 miro miro 3771 2017-04-30 17:39 eix-installed-after_2.tar.gz
108 -rw-r--r-- 1 miro miro 3771 2017-04-30 17:39 eix-installed-after.tar.gz
109 miro@g0n ~ $ ls -l --block-size=K eix-installed-after*.tar.gz
110 -rw-r--r-- 1 miro miro 4K 2017-04-30 17:39 eix-installed-after_1.tar.gz
111 -rw-r--r-- 1 miro miro 4K 2017-04-30 17:39 eix-installed-after_2.tar.gz
112 -rw-r--r-- 1 miro miro 4K 2017-04-30 17:39 eix-installed-after.tar.gz
113 miro@g0n ~ $
114
115 ...those tiny files, I find that...:
116
117 miro@g0n ~ $ hexdump -C eix-installed-after.tar.gz | head -c38
118 00000000 1f 8b 08 00 25 05 06 59 00 miro@g0n ~ $ hexdump -C eix-installed-after_1.tar.gz | head -c38
119 00000000 1f 8b 08 00 27 05 06 59 00 miro@g0n ~ $ hexdump -C eix-installed-after_2.tar.gz | head -c38
120 00000000 1f 8b 08 00 29 05 06 59 00 miro@g0n ~ $
121 miro@g0n ~ $
122
123 ...it's always the... ah well, for clarity, let's head just the first 25 bytes
124 of the hex representation, whihc will later be 5 bytes for dd commands...:
125
126 miro@g0n ~ $ hexdump -C eix-installed-after.tar.gz | head -c25
127 00000000 1f 8b 08 00 25 miro@g0n ~ $ hexdump -C eix-installed-after_1.tar.gz | head -c25
128 00000000 1f 8b 08 00 27 miro@g0n ~ $ hexdump -C eix-installed-after_2.tar.gz | head -c25
129 00000000 1f 8b 08 00 29 miro@g0n ~ $
130
131 ...that it's always (or almost always, see below about PART3 sometimes being
132 different) just the 5th byte that is different, in the way that doing the
133 procedure below will get those files which differed at creation time...:
134
135 80d004b1ecc6e6c2aec7cd453d05c7b4e4cc4092b9cef6b7c3fb31d148eb2310 eix-installed-after_1.tar.gz
136 f4c30c6d4ccc90d1ec2cc6df2ac10cae2545d04a85c7ff39207d93129cd0ec78 eix-installed-after_2.tar.gz
137 a8161c5771b661476966250af31d2b8bd80581a3e0c52d535b401524ee541ae7 eix-installed-after.tar.gz
138
139 (
140 Notice: the hash is always new. You won't get any of those hashes above
141 if you tar-gzip it. This is about the non-identical results.
142 )
143
144 ...to become identical. This procedure, that I'll put in the script:
145
146 Pls see attachment:
147 make_gzip_archives_consistent.sh
148
149 The fact is, I have, hours ago (this is slow work for me...) made a backup
150 (and spent all this time, and more to write that script), and noticed how my
151 gzip behaves, or should I say the gzip program *in my machine*... And...
152
153 And I can tell that I get different instances of archives consistently fixed
154 with that script.
155 I only can't get identical archives when PART3 is different in some (usually
156 only one of the three) of the archives.
157 NOTE (at proofreading time): But I'm not investigating that anymore, find how
158 I confirmed that the archives upon decompression are all of verifiably same
159 content, and that suffices.
160
161 I really have tried only on my backup, on multiple series of three different
162 tar-gzip'ed instances each of which in the series was done on the same
163 directory with the script above.
164
165 The script is just a proof of concept, and a demonstration of the apparent
166 issue. Because this previously does not seem to me to have been the case, but
167 maybe I misremember?
168
169 I checked a few of the tar-gzip'd archives of my backup, and so far they have
170 always, upon decompression, shown identical in all the different instances.
171
172 The finding
173 ===========
174
175 I've typically compared series of three instances of same directories that I
176 backed up, such as /home/miro, /root and /var/log, and they always were
177 identical upon decompression.
178 ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^
179 | | | | | | | | | | | | | | | | | | | | | | | | | | | |
180 That means *probably* all is fine with gzip, and there is no issue.
181
182 (But this result was not trivial to get by... and since I'm unsure in my
183 conclusion, I'm still posting this, asking for confirmation or denial.)
184
185 So maybe I've been chasing shadows here... And for long hours...
186
187 And I'm sorry if that is so, but look at that issue that I have with
188 Wireshark! Look at that! That's not a shadow. That's a serious bug or a
189 serious malfunction in my Gentoo, the latter being most likely...
190
191 And if it is the latter, it can only be one or the other way. One: the cause
192 is in some Gentoo packge. Two: it is an attack by some unknown means.
193
194 (
195 If Air-Gapped is some info, I did try and editcap (and the whole
196 Wireshark) behave in the same wrong way in my Air-Gapped too.
197
198 Same holds for the gzip inconsistency --or "inconsistency".
199 )
200
201 And looking for solution to that Wireshark issue, I found the above issue
202 which appeared to me to be some kind of inconsistency...
203
204 And I have other inconsistencies, bogus or true that they might be, such as
205 with bash.
206
207 One is very quick to report. Here it goes...
208
209 Thanks for any report on how this issue, and the one with Wireshark, shows,
210 completely in order or just like in my machine, in any other Gentoo users'
211 systems.
212
213 This is one of a series of commands that I used to check one of the backups,
214 in three different instances of tar-gzip'd archive I checked (such as the
215 /root directory tar-gzip'd today), and which showed faultless upon
216 decompression in all the three instances, despite the three instances of
217 tar-gzip'd archives not being identical (as their SHA256 sums show):
218
219 # for i in $(ls -1d root_170430_g0n*.d/); do sum=$(echo $i|sed 's/\.d\//\.sum/'); echo $sum ; read FAKE ; j=$(echo $i | sed 's/\.d\//\.tar.gz/'); ls -l $j $i ; cd $i; pwd ; read FAKE ; for file in $(find ./ -name '*'); do if [ -f "$file" ]; then sha256sum $file >> ../$sum ; fi; done ; cd - ; done ;
220
221 Now if I just place the cursor, by moving with Alt-F (skipping "words") and Ctrl-F (skipping 1 char) to just after:
222
223 "for i in $(ls -1d root_170430_g0n*.d/" in that command,
224
225 and if I then hit Tab for completion on the experssion there, I get (and I'm
226 sorry for the mess, but that's what I get):
227
228 g0n ~ # for i in $(ls -1d root_170430_g0n*.dbash: unexpected EOF while looking for matching `)'bash: syntax error: unexpected end of file//\.tar.gz/'); ls -l $j $i ; cd $i; pwd ; read FAKE ; for file in $(find ./ -name '*'); do if [ -f "$file" ]; then sha256sum $file >> ../$sum ; fi; done ; cd - ; done ;
229
230 NOTE (at proofreading time): rechecked, I do get that same behavior the day
231 after (wrote most of this yesterday, still to send this morning).
232
233 [[
234 NOTE (before delayed sending): In fact, it is only this clone that exibits the
235 above Bash malfunctioning. I just checked the same for loop command (some six
236 paragraphs above) in my Air-Gapped master [1] (never any internet it sees,
237 longer workaround/detailed checking before updating it with stuff from
238 internet, sneakernet or optical media), and it is just fine. That line, simply
239 gave what it should:
240
241 # for i in $(ls -1d root_170430_g0n*.d/); do sum=$(echo $i|sed 's/\.d\//\.sum/'); echo $sum ; read FAKE ; j=$(echo $i | sed 's/\.d\//\.tar.gz/'); ls -l $j $i ; cd $i; pwd ; read FAKE ; for file in $(find ./ -name '*'); do if [ -f "$file" ]; then sha256sum $file >> ../$sum ; fi; done ; cd - ; done
242 root_170430_g0n_1.d// root_170430_g0n_2.d// root_170430_g0n.d//
243 # [[and the same command line was back here]]
244
245 under exact same conditions/circumstances as the clone of my Air-Gapped. And
246 it's similar with some other completion issues: they seem non-existent in my
247 Air-Gapped.
248 ]]
249
250 IOW, first, Bash sullied the entire line, which is not very considerate of
251 Her, and second that's not some usual error. Just for clarity, it wrote this:
252
253 bash: unexpected EOF while looking for matching `)'bash: syntax error: unexpected end of file
254
255 (and it wrote it by overwriting, which I never used to see in Bash)
256
257 What's going on there?... Ah... Importantly:
258
259 do any of you other users get some erratic unusual behavior like this with Bash?
260
261 Of course, I can move to the start of the line with Ctrl-A and then issue
262 Ctrl-K to clear and capture to the entire line and then issue Ctrl-Y to paste
263 it back, and no disorderly message remains, but Bash isn't behaving...
264
265 I'll try and send this soon, but I first need to finish my backup...
266
267 Backup is done. Just, I guess if the reader has this bash version installed:
268 $ bash --version
269 GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu)
270 Copyright (C) 2016 Free Software Foundation, Inc.
271 License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
272
273 This is free software; you are free to change and redistribute it.
274 There is NO WARRANTY, to the extent permitted by law.
275 $
276 they might be able to reproduce such kind of misbehavior.
277
278 And finally, and this is what eix throws on any package that I would check:
279
280 g0n ~ # eix memtest86+
281 * sys-apps/memtest86
282 Available versions: 4.3.7 (~)4.3.7-r1 {serial}
283 Homepage: http://www.memtest86.com/
284 Description: A stand alone memory test for x86 computers
285
286 * sys-apps/memtest86+
287 Available versions: 2.01^t 4.00^t 4.20-r1 (~)4.20-r3 5.01-r2 (~)5.01-r3 {floppy iso serial}
288 Homepage: http://www.memtest.org/
289 Description: Memory tester based on memtest86
290
291 Found 2 matches
292 Received SIGSEGV - you probably found a bug in eix.
293 Please proceed with the following few instructions and help us find the bug:
294 * install gdb (sys-devel/gdb)
295 * reemerge eix with FEATURES="nostrip" CXXFLAGS="-g -ggdb3" LDFLAGS=""
296 * enter gdb with "gdb --args eix your_arguments_for_eix"
297 * type "run" and wait for the segfault to happen
298 * type "bt" to get a backtrace (this helps us a lot)
299 * post a bugreport and be sure to include the output from gdb.
300
301 Sorry for the inconvenience and thanks in advance!
302 g0n ~ #
303
304 Too many inconsistencies. Where do I start searching for the causes?
305
306 (As far as the fourth "inconsistency", I was thinking about trying memtest as
307 per:
308
309 Message-ID: <LO1P123MB067395FD4E9010B549743C9280120@××××××××××××××××××××××××××××××××××.COM>
310
311 How to get memtest onto a USB drive
312 https://lists.gt.net/gentoo/user/325837#325837
313
314 , but that's just for lack of other ideas, these issues don't look
315 like bad memory. I might still try it, but when I go to sleep, not sooner.
316 )
317
318 Regards!
319 ---
320 [1] My methods are still these:
321 Air-Gapped Gentoo Install, Tentative
322 https://forums.gentoo.org/viewtopic-t-987268.html
323
324 and
325
326 Postfix smtp/TLS, Bkp/Cloning Mthd, Censorship/Intrusion
327 https://forums.gentoo.org/viewtopic-t-999436.html#7613044
328
329 --
330 Miroslav Rovis
331 Zagreb, Croatia
332 https://www.CroatiaFidelis.hr

Attachments

File name MIME type
gzip_buggy.txt.tar.gz application/octet-stream
gzip_buggy.txt_1.tar.gz application/octet-stream
gzip_buggy.txt_2.tar.gz application/octet-stream
make_gzip_archives_consistent.sh application/x-sh
gzip_buggy.txt text/plain
eix-installed-after text/plain
eix-installed-after.tar.gz application/octet-stream
eix-installed-after_1.tar.gz application/octet-stream
eix-installed-after_2.tar.gz application/octet-stream
signature.asc application/pgp-signature

Replies

Subject Author
Re: [gentoo-user] Inconsistent behavior in my Gentoo OS instance Raffaele Belardi <raffaele.belardi@××.com>
Re: [gentoo-user] Inconsistent behavior in my Gentoo OS instance Miroslav Rovis <miro.rovis@××××××××××××××.hr>
[gentoo-user] eix bug (was: Inconsistent behavior in my Gentoo OS instance) Martin Vaeth <martin@×××××.de>