RE: [gentoo-server] Re: [OT] Mirroring/backing-up a large

> In normal circumstances, databases are more efficient at
> handling lookups than filesystems.
>
> In your image application database, use a timestamp field
> that is updated whenever images are added or updated.
>
> Generate your backup jobs based on queries to this database
> instead of requiring rsync to do its differencing thing.  For
> example you can automate a process that queries the database
> for images that have been updated or added since the last
> time it ran and generate a file list or backup job that only
> copies over new or updated images based on the timestamp.
> You would have to somehow map within the database the actual
> physical location of the files if you are not already doing
> it, in addition to using squid/apache to translate to the client.
>
> That is the first step.

MIkey's right here. You cannot expect your filesystem to be able to return
the query results you need. You need to take this "out of band". You could
even store your data base on a separate filesystem so you don't use I/O in
the disk array that you need for the backups.
 
> The second step is to ditch storing everything on a single
> 9TB system that cannot be backed up efficiently.  Distribute
> the storage of the images on clusters or whatever.  For
> example peel of 1TB of images onto a single server, then
> update the database (or apache/squid mapping) to point to the
> new location.  9 1TB boxes would be far less prone to
> catastrophic failure and much easier to
> replicate/mirror/backup than a single 9TB box.  This is what
> I call the "google approach" ;)  Use cheap commodity hardware
> and smart implementation to distribute/scale the load.

Many years ago, I was at a Progress database conference and one very useful
presentation was about the effect on performance of a large data store
without increasing the bandwidth available to that data store. The speaker's
example showed how your performance decreases when you have one large
database but still only a single channel for access. His point was to
increase the number of channels along with the size of the store, otherwise
you actually lose performance. This is tantamount to MIkey's discussion
above. Spread out your disks if possible. Your problem is that you cannot
get more channels into your backup store, so you'll have to think about
either a separate local backup SAN or a provider with more bandwidth.

Bill
-- 
gentoo-server@g.o mailing list