Re: [gentoo-user] Speed up `du'

[Stroller]

On 25 May 2008, at 00:24, Willie Wong wrote:

> On Sat, May 24, 2008 at 04:49:09PM -0500, Penguin Lover
> reader@×××××××.com squawked:
>> ...
>> I use Reiserfs with default sizes.  In some situations like a large
>> cache of nntp messages of several GB.  I might wait 5-10 minutes
>> or more
>> for du to get the size of the directory.
>
> I am pretty sure the problem with du is that it actually looks,
> recursively, at every single file and computes the size that way.

What he said.

> Or maybe there is some other tool or technique that can quickly tell
> me the size of a directory or set of directories.

Keep all the files in a honkin' big tarball.
:P
If you need to read these files on the fly then I'm afraid you'll  
have to write a kernel filesystem extension (or find one?) that will  
read them out of the tar file, slowing all read & write actions down.  
But, hey, `du` on the tarball will complete in no time at all!! ;)

In seriousness, another thing to do is keep these files on a separate  
partition, if you can. Basically a user's ~ which includes  
both .maildir and "My HiDef Videos" is non-optimal.

>> Are there other file systems that can return a result of `du' faster?


All filesystems have their advantages & disadvantages.

<http://www.debian-administration.org/articles/388>
Reading the above I _think_ the test most similar in function to  
running `du` on many small files is the "Directory listing and file  
search into the previous file tree" test, at which ResiderFS is fastest.

I need to look into this myself soon, to try & get best speed at a  
3gig corpus of email. I was expecting EXT3 to be best - when you  
create the filesystem you can specify the blocksize. It's possible  
that the author of the filesystems comparison could have chosen  
options when formatting his EXT3 disk that affected the speed of the  
results - a journal would make writes slower, for instance (not sure  
about reads).

Stroller.
-- 
gentoo-user@l.g.o mailing list