1 |
On Sat, Apr 29, 2017 at 3:24 PM, lee <lee@××××××××.de> wrote: |
2 |
|
3 |
> Mick <michaelkintzios@×××××.com> writes: |
4 |
> |
5 |
> > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote: |
6 |
> >> On 25/04/2017 16:29, lee wrote: |
7 |
> >> > Hi, |
8 |
> >> > |
9 |
> >> > since the usage of FTP seems to be declining, what is a replacement |
10 |
> >> > which is at least as good as FTP? |
11 |
> >> > |
12 |
> >> > I'm aware that there's webdav, but that's very awkward to use and |
13 |
> >> > missing features. |
14 |
> >> |
15 |
> >> Why not stick with ftp? |
16 |
> >> Or, put another way, why do you feel you need to use something else? |
17 |
> >> |
18 |
> >> There's always dropbox |
19 |
> > |
20 |
> > |
21 |
> > Invariably all web hosting ISPs offer ftp(s) for file upload/download. |
22 |
> If you |
23 |
> > pay a bit more you should be able to get ssh/scp/sftp too. Indeed, many |
24 |
> ISPs |
25 |
> > throw in scp/sftp access as part of their basic package. |
26 |
> > |
27 |
> > Webdav(s) offers the same basic upload/download functionality, so I am |
28 |
> not |
29 |
> > sure what you find awkward about it, although I'd rather use lftp |
30 |
> instead of |
31 |
> > cadaver any day. ;-) |
32 |
> > |
33 |
> > As Alan mentioned, with JavaScript'ed web pages these days there are many |
34 |
> > webapp'ed ISP offerings like Dropbox and friends. |
35 |
> > |
36 |
> > What is the use case you have in mind? |
37 |
> |
38 |
> transferring large amounts of data and automatization in processing at |
39 |
> least some of it, without involving a 3rd party |
40 |
> |
41 |
> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB, |
42 |
> or even more. The mirror feature of lftp is extremely useful for such |
43 |
> things. |
44 |
> |
45 |
> I wouldn't ever want having to mess around with web pages to figure out |
46 |
> how to do this. Ftp is plain and simple. So you see why I'm explicitly |
47 |
> asking for a replacement which is at least as good as ftp. |
48 |
> |
49 |
> |
50 |
> -- |
51 |
> "Didn't work" is an error. |
52 |
> |
53 |
> |
54 |
Half petabyte datasets aren't really something I'd personally *ever* trust |
55 |
ftp with in the first place. That said, it depends entirely on the network |
56 |
you're working with. Are you pushing this data in/out of the network your |
57 |
machines live in, or are you working primarily internally? If internal, |
58 |
what're the network side capabilities you have? Since you're likely already |
59 |
using something on the order of CEPH or Gluster to back the datasets where |
60 |
they sit, just working with it all across network from that storage would |
61 |
be my first instinct. |
62 |
|
63 |
How often does it need moved in/out of your facility, and is there no way |
64 |
to break up the processing into smaller chunks than a 0.6PB mass of files? |
65 |
Distribute out the smaller pieces with rsync, scp, or the like, operate on |
66 |
them, and pull back in the results, rather than trying to shift around the |
67 |
entire set. There's a reason Amazon will send a physical truck to a site to |
68 |
import large datasets into glacier... ;) |
69 |
|
70 |
-- |
71 |
Poison [BLX] |
72 |
Joshua M. Murphy |