Re: [gentoo-user] replacement for ftp?

[Poison BL.]

On Sat, Apr 29, 2017 at 9:11 PM, lee <lee@××××××××.de> wrote:
>
> "Poison BL." <poisonbl@×××××.com> writes:
> > Half petabyte datasets aren't really something I'd personally *ever*
trust
> > ftp with in the first place.
>
> Why not?  (12GB are nowhere close to half a petabyte ...)

Ah... I completely misread that "or over 50k files in 12GB" as 50k files
*at* 12GB each... which works out to 0.6 PB, incidentally.

> The data would come in from suppliers.  There isn't really anything
> going on atm but fetching data once a month which can be like 100MB or
> 12GB or more.  That's because ppl don't use ftp ...

Really, if you're pulling it in from third party suppliers, you tend to be
tied to what they offer as a method of pulling it from them (or them
pushing it out to you), unless you're in the unique position to dictate the
decision for them. From there, assuming you can push your choice of product
on them, it becomes a question of how often the same dataset will need
updated from the same sources, how much it changes between updates, how
secure it needs to be in transit, how much you need to be able to trust
that the source is still legitimately who you think it is, and how much
verification that there wasn't any corruption during the transfer. Generic
FTP has been losing favor over time because it was built in a time that
many of those questions weren't really at the top of the list for concerns.

SFTP (or SCP) (as long as keys are handled properly) allows for pretty
solid certainty that a) both ends of the connection are who they say they
are, b) those two ends are the only ones reading the data in transit, and
c) the data that was sent is the same that was received (simply as a side
benefit of the encryption/decryption). Rsync over SSH gives the same set of
benefits, reduces the bandwidth used for updating the dataset (when it's
the same dataset, at least), and will also verify the data on both ends (as
it exists on disk) matches. If you're particularly lucky, the data might
even hit just the right mark that benefits from the in-line compression you
can turn on with SSH, too, cutting down the actual amount of bandwidth you
burn through for each transfer.

If your suppliers all have *nix based systems available, those are also
standard tools that they'll have on hand. If they're strictly Windows
shops, SCP/SFTP are still readily available, though they aren't built into
the OS... rsync gets a bit trickier.

> > How often does it need moved in/out of your facility, and is there no
way
> > to break up the processing into smaller chunks than a 0.6PB mass of
files?
> > Distribute out the smaller pieces with rsync, scp, or the like, operate
on
> > them, and pull back in the results, rather than trying to shift around
the
> > entire set. There's a reason Amazon will send a physical truck to a
site to
> > import large datasets into glacier... ;)
>
> Amazon has trucks?  Perhaps they do in other countries.  Here, amazon is
> just another web shop.  They might have some delivery vans, but I've
> never seen one, so I doubt it.  And why would anyone give them their
> data?  There's no telling what they would do with it.

Amazon's also one of the best known cloud computing suppliers on the planet
(AWS = Amazon Web Services). They have everything from pure compute
offerings to cloud storage geared towards *large* data archival. The latter
offering is named "glacier", and they offer a service for the import of
data into it (usually the "first pass", incremental changes are generally
done over the wire) that consists of a shipping truck with a rather nifty
storage system in the back of it that they hook right into your network.
You fill it with data, and then they drive it back to one of their data
centers to load it into place.

--
Poison [BLX]
Joshua M. Murphy