Re: [gentoo-alt] Something is wrong with `file`

[Elias Pipping]

On Sat, Jul 21, 2007 at 12:42:25PM +0200, Elias Pipping wrote:
> Hi,
>
> I use file(1) on a regular basis and I've come across something really
> strange.
>
> I noticed running `file` on info pages takes *ages*.
>
> What's confusing, is this: that's not the case with /usr/bin/file.
>
>   % pwd
>   /usr/share/info
>   % time /usr/bin/file * > /dev/null
>   0.201u 0.018s 0:00.22 95.4%     0+0k 0+0io 0pf+0w
>   % time ~/Gentoo/usr/bin/file * > /dev/null
>   28.371u 0.078s 0:28.56 99.5%    0+0k 0+4io 0pf+0w
>
> That is a *HUGE* difference. I don't know why that is.
>
>   % /usr/bin/file --version
>   file-4.10
>   magic file from /usr/share/file/magic
>   % ~/Gentoo/usr/bin/file --version
>   file-4.21
>   magic file from /Users/pipping/Gentoo/usr/share/misc/file/magic
>
> I've also compared some more filetypes:
>
>   % pwd
>   /usr/share/man/man7
>   % time /usr/bin/file * > /dev/null
>   0.034u 0.011s 0:00.04 100.0%    0+0k 0+0io 0pf+0w
>   % time ~/Gentoo/usr/bin/file * > /dev/null
>   1.423u 0.010s 0:01.44 99.3%     0+0k 0+0io 0pf+0w
>
>   % pwd
>   /usr/share/ri/1.8/system/YAML
>   % time /usr/bin/file * > /dev/null
>   0.024u 0.009s 0:00.03 66.6%     0+0k 0+0io 0pf+0w
>   % time ~/Gentoo/usr/bin/file * > /dev/null
>   0.049u 0.004s 0:00.05 80.0%     0+0k 0+0io 0pf+0w
>
>   % pwd
>   /usr/include
>   % time /usr/bin/file * > /dev/null
>   0.176u 0.014s 0:00.19 94.7%     0+0k 0+0io 0pf+0w
>   % time ~/Gentoo/usr/bin/file * > /dev/null
>   3.874u 0.026s 0:03.91 99.4%     0+0k 0+0io 0pf+0w
>
> == what i compiled `file` with ==
>
>   % emerge --info | egrep C\(XX\)\?FLAGS
>   CFLAGS="-O3 -march=nocona -fomit-frame-pointer -pipe"
>   CXXFLAGS="-O3 -march=nocona -fomit-frame-pointer -pipe"
>
>
> Kind regards,
>
> Elias
>
> PS: I ran the above commands multiple times so that time's output
> stabilized.
>
> PPS: don't be tempted to benchmark in /usr/bin or /usr/lib, because
> /usr/bin/file does a lot more work there, splitting up universal
> binaries (on x86), hence the benchmarking results become useless.
>
> PPPS: I've had a look at Apple's patches to file[1], but I couldn't find
> one that explains the above behavior.
>
> PPPPS: I'm attaching the output of
> `diff -u /usr/share/file/magic ~/Gentoo/usr/share/misc/file/magic`
>
> [1] http://www.opensource.apple.com/darwinsource/Current/file-6.2/patches/

Apparently the issue has *only* to do with the magic file.  I compiled
file from within the prefix but without portage with the prefix
/usr/local.  `./src/file` would not work because there's no
/usr/local/share/file/magic ( or at least on my machine thee isn't ),
so i made a link there to ~/Gentoo/usr/share/misc/file/magic.  It worked
and it was slow.  Now I replaced the link with one to
/usr/share/file/magic and all of a sudden it was fast.  Switched it
back, slow again and so on ( I ran the commands a couple of times to
make sure the differences in performance weren't caching-related.

I still don't quite get the enormous difference but at least I know
where to dig.

  % du -h /usr/share/file/magic | cut -f 1 
  352K
  % du -h ~/Gentoo/usr/share/misc/file/magic | cut -f 1 
  464K


-- Elias