Re: [gentoo-server][OT] Mirroring/backing-up a large 15Million (6TB) file collection

[jos houtman]

Thank you all for the reply's. let me answer them in one reply.

Björn Gustafsson wrote:

> Now what kind of systems are these? Home-grown arrays or "real" ones?
> In the latter case, are there no vendor-provided approaches to this?

These are what you would call homegrown, we are now looking at more 
serieus systems. but they also cost serious money..

> I'm not sure how this would apply to regular filesystems (no idea
> which one you use though), but in "larger" (not size-wise) systems, a
> bitmap of the filesystem is kept in a separate location separate, and
> disk areas with changed or added files are marked as dirty, and
> transferred to the remote host either immediately (with synchronous
> i/o), as soon as possible (async i/o), or when requested (veeeery
> async i/o ;)). This is rather effective system, with the backup speed
> mainly dependent on the size you would choose for the bitmap (large
> bitmap => smaller blocks => potentially less data) and transfer
> speed.. Restructuring of data on the physical disk would also create a
> major update of blocks to be transferred.

hmm i like that, is it possible to filter the things that are 
transferred? for example i dont want to mirror deletions.

> Marking folders as dirty is another solution, however 50k files is a
> bit big. Implementing dirty files in chunks of say 50 or 100 would be
> a half-way solution, but that'd be dependant on the application [see
> below].

another nice suggestion, if the list of individual edited files is 
getting to big, we can indeed start working with groups..


 From Alex efros:

In your case that mean, for example: it's probably best solution to
backup issue to change a way how files changed so what changed files
isn't really CHANGED, but instead new version is just ADDED to collection.
This way it will be enough for you to just remember which file was
backuped last by previous backup and on next backup continue from that
file (I suppose all your files are numbered: "(1-50000,50001-100000, etc)").

This way backup will not depend on collection size (only on amount of
added files) and will not depend on some "special feature" in application
(like constructing list of changed files) which may have bugs.

I really like this solution. It has several advantages:
- it's really simple
- it requires no interaction with the application.
- it gives a little overhead in diskspace, but thats probably negligable 
to the 20/30% of picture we dont need, but just don't delete. we dont 
want to take the risk of deleting wrong files. and ofc if ever the 
service is misused, we still have evidence.
- no need to construct a list. just a last-backupped-photo-id somewhere.


 From Mikey:

Certainly every file is not linked directly in a web page?  

Why not keep links in a database that point to the correct location on disk
for the images themselves?  Then all you need to do is query the database
for a timestamp field that has changed and you know what you need to back
up, and at any time you can move the underlying files around and update the
links in the database...

You can be fairly sure that atleast 80% of the photo's is accessible on 
the website.
I dont really understand you here, but i think we allready have what you 
mean.
but for completeness, this is abit how the system works.
We allready keep a record of the photo's in the db for 
bookkeeping/userinfo/accessrights/albums/etc... etc...
the actual location of the image-file is determined by the id plus a 
secret. so image 11809373 can be accessed using
http://interval1.rendered.startpda.net/11800001-11850000/11809373_120_120_sfi5.jpeg. 
This allows to do resizing (120_120), provide the content with a simple 
system of apache servers and squids.

jos
-- 
gentoo-server@g.o mailing list