From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from lists.gentoo.org (pigeon.gentoo.org [208.92.234.80]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by finch.gentoo.org (Postfix) with ESMTPS id 65CEE158083 for ; Fri, 6 Sep 2024 21:41:51 +0000 (UTC) Received: from pigeon.gentoo.org (localhost [127.0.0.1]) by pigeon.gentoo.org (Postfix) with SMTP id A1A55E2A48; Fri, 6 Sep 2024 21:41:44 +0000 (UTC) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (prime256v1) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by pigeon.gentoo.org (Postfix) with ESMTPS id 11F9CE2A08 for ; Fri, 6 Sep 2024 21:41:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmx.de; s=s31663417; t=1725658896; x=1726263696; i=warp_7@gmx.de; bh=j5uapIrtBQiy746+y1Zm2KiAWwdNlXKiQK2Crba/nT0=; h=X-UI-Sender-Class:Date:From:To:Subject:Message-ID:References: MIME-Version:Content-Type:In-Reply-To:cc: content-transfer-encoding:content-type:date:from:message-id: mime-version:reply-to:subject:to; b=A32s06JW6fSTbSgXgwO8N3dyd/cy1EKjKHaYVNN+KDabKMgsTscJCZBs7MJwUf6N offb5Kx4ikYA2J/CSukDk4ryc4cu7BTnAxfjEwCytjPg9AIrKsoAz8mC16EC/ooIK yrve9mzk1DzGC2ZLvqkhfBcUurk5bjA5sOMEn+5muMMxJ/ykAUz+xCkI9NUOALxa5 k0xEtB9FGSbSgOqWBJCf33Vqq7ru8uk/ns772SYDmPLWjcWTGwW6Wcu7dz9urRs3L 0Zlh410octUDBV5BLK5akoz4mQjLzZGXgYaiYzTbKdtNk/ri1aDrWycx+Azest9MA myfR6NyZrfbPf3ZC8A== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from tp ([87.122.66.247]) by mail.gmx.net (mrgmx004 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MkHQh-1sJvTd0x7B-00fE7u for ; Fri, 06 Sep 2024 23:41:36 +0200 Date: Fri, 6 Sep 2024 23:41:33 +0200 From: Frank Steinmetzger To: gentoo-user@lists.gentoo.org Subject: Re: [gentoo-user] Re: Package compile failures with "internal compiler error: Segmentation fault". Message-ID: Mail-Followup-To: gentoo-user@lists.gentoo.org References: <8c26be16-d033-ea3f-06e1-a9ce84cbbafb@gmail.com> <7866172.lvqk35OSZv@rogueboard> <15276605.tv2OnDr8pf@rogueboard> Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-Id: Gentoo Linux mail X-BeenThere: gentoo-user@lists.gentoo.org Reply-to: gentoo-user@lists.gentoo.org X-Auto-Response-Suppress: DR, RN, NRN, OOF, AutoReply MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7ioDbKeoC09+ytw5" Content-Disposition: inline In-Reply-To: <15276605.tv2OnDr8pf@rogueboard> User-Agent: Mutt 2.2.13; VIM - Vi IMproved 9.1 X-Provags-ID: V03:K1:7Px+E24LtVHC3f+z3YHSSwC8H5/57RxBQqEDGWn1Z6LtwVmbZ+s m6Ktm24YwZ6USPmXSywDXMLt85cY4wp8zG2sSz6aVTnsq20V1SgqjgO2ppZKlVw3yfHK9Xq ljxFfWdpL81xbwIj0IKx+aEA5jR4lEkUevTP+RSAg8KHV/gspeQua6TNZonEVJU6UXf/t5w TLRctStxnszZmNrP6gvGg== X-Spam-Flag: NO UI-OutboundReport: notjunk:1;M01:P0:5zvtXJILkRg=;v8YB9ANdhlIadZhAqqCBW/pF6Nb JhxI0Q+avoeCVwFegX3Sp0JXh5L/zo6P/eZLVdNfA16mgw2BDc9ee5cTjQX0iCVSSMocQijFw EOX/RhskRmuEDCLGGxji8nXidsQeqfkvV7BHxSR6pKxOQFBGFWEdS2MACIccQi/6wV0prLsu4 gBape6DS27vZ3VvX5sSVH40MIa103u5AmTdkYxYxy3Zvx1k1q9eQG0Tb/BUUNXEtBHdDwQrJt T25P4KVPHJF2kYehgRhCS7fzSpUjv77I3SzGINpOilxD2IbzdE9rPJJ3oTw0NoWcBCFhoxmPP orEAkjP11tmSwJNi9H5AmUbvgwx1sSpw2lh6lWl1oVc1UbO4Pafl+ut/z/T5WTMHsv+pKM8sm RWmGXYzGiUoCAbtUUrMv7o4qGiEUnRmOG7cTyVLu3PdplG2DlSla35oIFLKtItUcfj2aKjjKw n8aLuBtv6ZOhUJfr27z+n99MopuuXOvvQJ2otrDiY+VCQpDJmOIjWMtvywy11ZOOpYC9yO/wt r0WSOmsGLx0+x01jvrCXHYGVCz2yYrzmAztWYHWoC6Lv6VlUJ/30r31TH/HfnF1ULjNwJqcD3 gzun0Fjzv37uJaZdsCENgMNaZG/kzX6q1g38n4kRkWa3pIaidn5d0wO8cwlO3/kYrrqXYGB+j GzccZiNVEF1WNkBNkWplhPKivIk2dB21/M+mZ4pmg/mApiPinFRx+hPtFZd4/aIgeIvccdx6d Z7tfFLnfGS8hBX8yOgthXpS4A7G9ytwXqtH9i2rABi0VO73UMHndYqadIHbggGAe6bvHtRnzd t1JBbYRbScx8qeewawFcmyoQ== X-Archives-Salt: f2533d24-6525-4165-a5f4-e0012ef29ac4 X-Archives-Hash: c0baeaef1bd1cbb90065b82051a0e372 --7ioDbKeoC09+ytw5 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Am Fri, Sep 06, 2024 at 01:21:20PM +0100 schrieb Michael: > > > find path-to-directory/ -type f | xargs md5sum > digest.log > > >=20 > > > then to compare with a backup of the same directory you could run: > > >=20 > > > md5sum -c digest.log | grep FAILED I had a quick look at the manpage: with md5sum --quiet you can omit the gre= p=20 part. > > > Someone more knowledgeable should be able to knock out some clever py= thon > > > script to do the same at speed. And that is exactly what I have written for myself over the last 11 years. = I=20 call it dh (short for dirhash). As I described in the previous mail, I use= =20 it to create one hash files per directory. But it also supports one hash=20 file per data file and =E2=80=93 a rather new feature =E2=80=93 one hash fi= le at the root of=20 a tree. Have a look here: https://github.com/felf/dh Clone the repo or simply download the one file and put it into your path. > > I'll be honest here, on two points. I'd really like to be able to do > > this but I have no idea where to or how to even start. My setup for > > series type videos. In a parent directory, where I'd like a tool to > > start, is about 600 directories. On a few occasions, there is another > > directory inside that one. That directory under the parent is the name > > of the series. In its default, my tool ignores directories which have subdirectories. It= =20 only hashes files in dirs that have no subdirs (leaves in the tree). But=20 this can be overridden with the -f option. My tool also has an option to skip a number of directories and to process= =20 only a certain number of directories. > > Sometimes I have a sub directory that has temp files; > > new files I have yet to rename, considering replacing in the main series > > directory etc. I wouldn't mind having a file with a checksum for each > > video in the top directory, and even one in the sub directory. As a > > example. > >=20 > > TV_Series/ > >=20 > > =E2=94=9C=E2=94=80=E2=94=80 77 Sunset Strip (1958) > > =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 torrent > > =E2=94=9C=E2=94=80=E2=94=80 Adam-12 (1968) > > =E2=94=9C=E2=94=80=E2=94=80 Airwolf (1984) So with my tool you would do $ dh -f -F all TV_Series `-F all` causes a checksum file to be created for each data file. > > What > > I'd like, a program that would generate checksums for each file under > > say 77 Sunset and it could skip or include the directory under it. Unfortunately I don=E2=80=99t have a skip feature yet that skips specific= =20 directories. I could add a feature that looks for a marker file and then=20 skips that directory (and its subdirs). > > Might be best if I could switch it on or off. Obviously, I may not want > > to do this for my whole system. I'd like to be able to target > > directories. I have another large directory, lets say not a series but > > sometimes has remakes, that I'd also like to do. It is kinda set up > > like the above, parent directory with a directory underneath and on > > occasion one more under that.=20 >=20 > As an example, let's assume you have the following fs tree: >=20 > VIDEO > =E2=94=9C=E2=94=80=E2=94=80TV_Series/ > | =E2=94=9C=E2=94=80=E2=94=80 77 Sunset Strip (1958) > | =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 torrent > | =E2=94=9C=E2=94=80=E2=94=80 Adam-12 (1968) > | =E2=94=9C=E2=94=80=E2=94=80 Airwolf (1984) > | > =E2=94=9C=E2=94=80=E2=94=80Documentaries > =E2=94=9C=E2=94=80=E2=94=80Films > =E2=94=9C=E2=94=80=E2=94=80etc. >=20 > You could run: >=20 > $ find VIDEO -type f | xargs md5sum > digest.log >=20 > The file digest.log will contain md5sum hashes of each of your files with= in=20 > the VIDEO directory and its subdirectories. >=20 > To check if any of these files have changed, become corrupted, etc. you c= an=20 > run: >=20 > $ md5sum -c digest.log | grep FAILED >=20 > If you want to compare the contents of the same VIDEO directory on a back= up,=20 > you can copy the same digest file with its hashes over to the backup top= =20 > directory and run again: >=20 > $ md5sum -c digest.log | grep FAILED My tool does this as well. ;-) In check mode, it recurses, looks for hash files and if it finds them,=20 checks all hashes. There is also an option to only check paths and=20 filenames, not hashes. This allows to quickly find files that have been=20 renamed or deleted since the hash file was created. > > One thing I worry about is not just memory problems, drive failure but > > also just some random error or even bit rot. Some of these files are > > rarely changed or even touched. I'd like a way to detect problems and > > there may even be a software tool that does this with some setup, > > reminds me of Kbackup where you can select what to backup or leave out > > on a directory or even individual file level.=20 Well that could be covered with ZFS, especially with a redundant pool so it= =20 can repair itself. Otherwise it will only identify the bitrot, but not be= =20 able to fix it. > > Right now, I suspect my backup copy is likely better than my main copy.= =20 The problem is: if they differ, how do you know which one is good apart fro= m=20 watching one from start to finish? You could use vbindiff to first find the= =20 part that changed. That will at least tell you where the difference is, so= =20 you could seek to the area of the position in the video. > This should work in rsync terms: >=20 > rsync -v --checksum --delete --recursive --dry-run SOURCE/ DESTINATION >=20 > It will output a list of files which have been deleted from the SOURCE an= d=20 > will need to be deleted at the DESTINATION directory. If you look at changed *and* deleted files in one run, better use -i instea= d=20 of -v. --=20 Gr=C3=BC=C3=9Fe | Greetings | Salut | Qapla=E2=80=99 Please do not share anything from, with or about me on any social network. If two processes are running concurrently, the less important will take processor time away from the more important on= e. --7ioDbKeoC09+ytw5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEVbE9o2D2lE5fhoVsizG+tUDUMMoFAmbbdwoACgkQizG+tUDU MMpwahAAx4iULwvOmIpbw4IG30yMiNnkYK9Fq5yVGgL6dfM3Izint9mvHpEmOKl4 wg1gJQf8V8dDJFAI2al/ihP11+phbbVmnxwW/lked68JcRu7Te84tUsqLdi0mPp8 jK4zycqXRt9UzcUEyBW2+u75CHmAI4qRUOHhPvpoUeaUBmHmOm32GSWX79fjKtIe O47YvdUzIL9Y/RJFKg1eK1LxOvu01vGXkmPiwjk6HAGNtMq/+7T3SHpnkAPbStQn 7X7YixB3yyxuxYE0mn90hnIGBu+vzNCkyxo5E0MiDWhf6XnAD+SMQK+wcfazt+Zo OHaI+aOhpral6Xt5CfUyNKEbJNolG4oHE8LePfcU0MP5XDArsghWQW0wOjwVgFba U5bYOoFmmGmyBU0Ox/9Z9o1EGGuPZ2+q5ZZJmnNEFFwttZcjhbORIKvA6A0UBIuy Knc8IX+m0vGFdM5FFbAFcRWWt/FOZ6WGbLeXnpwKawK9+gOryqn2tvEJ22ZndppQ gqrmRV/HWeYu/eNfYzC8iofSNqTZk9fq/sg6+Q0fj9UbVCKRGxhJ7ZSyUlZd7mcr qFhikTEAH46u9gKbzT5W7RZJ4CBityaAEcuWckEq6Ar8oBIvgaruYZ/WmBZJCXW7 euvYVqBTOFzGodxUu/nLNStUJxUMk1nq36BfUdgz0Kl2cNcPnIE= =KN2G -----END PGP SIGNATURE----- --7ioDbKeoC09+ytw5--