Re: [gentoo-user] Fast file system for cache directory with lot's of files

[Florian Philipp]

Am 14.08.2012 15:54, schrieb Daniel Troeder:
> On 14.08.2012 11:46, Neil Bothwick wrote:
>> On Tue, 14 Aug 2012 10:21:54 +0200, Daniel Troeder wrote:
>>
>>> There is also the possibility to write a really small daemon (less than
>>> 50 lines of C) that registers with inotify for the entire fs and
>>> journals the file activity to a sqlite-db.
>>
>> sys-process/incron ?
> Uh... didn't know that one! ... very interesting :)
>
> Have you used it?
> How does it perform if there are lots of modifications going on?
> Does it have a throttle against fork bombing?
> must-read-myself-a-little.....
>
> A incron line
> # sqlite3 /file.sql 'INSERT filename, date INTO table'
> would be inefficient, because it spawn lots of processes, but it would
> be very nice to simply test out the idea. Then a
> # sqlite3 /file.sql 'SELECT filename FROM table SORTBY date < date-30days'
> or something to get the files older than 30 days, and voilá :)
>
>

Maybe inotifywait is better for this kind of batch job.

Collecting events:
inotifywait -rm -e CREATE,DELETE --timefmt '%s' --format \
  "$(printf '%%T\t%%e\t%%w%%f')" /tmp > events.tbl
# the printf is there because inotifywait's format does not
# recognize common escapes like \t
# Output format:
# Seconds since epoch \t CREATE/DELETE \t file name \n

Filtering events:
sort --stable -k3 events.tbl |
awk '
  function update() {
    line=$0; exists= $2=="DELETE" ? 0 : 1; file=$3
  }
  NR==1{ update(); next }
  { if($3!=file && exists==1){ print line } update() }'
# Sorts by file name while preserving temporal order.
# Uses awk to suppress output of files that have been deleted.
# Output: Last CREATE event for each existing file

Retrieving files created 30+ days ago:
awk -v newest=$(date -d -5seconds +%s) '
  $1>newest{ nextfile }
  { print $3 }'

Remarks:

The awk scripts need some improvement if you have to handle whitespaces
in filenames but with the input format, it should be able to work with
everything except newlines.

Inotifywait itself is utterly useless when dealing with newlines in file
names unless you want to put some serious effort into sanitizing the output.

Regards,
Florian Philipp