| 1 |
jos houtman wrote: |
| 2 |
|
| 3 |
> I dont really understand you here, but i think we allready have what you |
| 4 |
> mean. |
| 5 |
> but for completeness, this is abit how the system works. |
| 6 |
> We allready keep a record of the photo's in the db for |
| 7 |
> bookkeeping/userinfo/accessrights/albums/etc... etc... |
| 8 |
> the actual location of the image-file is determined by the id plus a |
| 9 |
> secret. so image 11809373 can be accessed using |
| 10 |
> This allows to do resizing (120_120), provide the content with a simple |
| 11 |
> system of apache servers and squids. |
| 12 |
|
| 13 |
Here is what I am thinking... |
| 14 |
|
| 15 |
In normal circumstances, databases are more efficient at handling lookups |
| 16 |
than filesystems. |
| 17 |
|
| 18 |
In your image application database, use a timestamp field that is updated |
| 19 |
whenever images are added or updated. |
| 20 |
|
| 21 |
Generate your backup jobs based on queries to this database instead of |
| 22 |
requiring rsync to do its differencing thing. For example you can automate |
| 23 |
a process that queries the database for images that have been updated or |
| 24 |
added since the last time it ran and generate a file list or backup job |
| 25 |
that only copies over new or updated images based on the timestamp. You |
| 26 |
would have to somehow map within the database the actual physical location |
| 27 |
of the files if you are not already doing it, in addition to using |
| 28 |
squid/apache to translate to the client. |
| 29 |
|
| 30 |
That is the first step. |
| 31 |
|
| 32 |
The second step is to ditch storing everything on a single 9TB system that |
| 33 |
cannot be backed up efficiently. Distribute the storage of the images on |
| 34 |
clusters or whatever. For example peel of 1TB of images onto a single |
| 35 |
server, then update the database (or apache/squid mapping) to point to the |
| 36 |
new location. 9 1TB boxes would be far less prone to catastrophic failure |
| 37 |
and much easier to replicate/mirror/backup than a single 9TB box. This is |
| 38 |
what I call the "google approach" ;) Use cheap commodity hardware and |
| 39 |
smart implementation to distribute/scale the load. |
| 40 |
|
| 41 |
Of course the ultimate solution would some sort of cluster or san |
| 42 |
approach... |
| 43 |
|
| 44 |
-- |
| 45 |
gentoo-server@g.o mailing list |
Archives