1 |
Greetings, |
2 |
|
3 |
Let me start off by saying I'm quite new to network programming so I |
4 |
might ask some dumb questions and you might have to dumb-down your answers. |
5 |
|
6 |
So let me paint the picture of what I need to accomplish and perhaps the |
7 |
wisdom of all gentoo-server'ers can point me in the right direction. |
8 |
|
9 |
I have a cluster with a handful of quad-opteron blades (running Gentoo). |
10 |
This is the server. |
11 |
|
12 |
I also have clients (mostly windows users) out in internet-land who need |
13 |
to, through a web interface, upload files and/or point some data-stream |
14 |
to be sent (presumably over tcp/ip) to the server. Once it hits the |
15 |
server, an application process gets spawned to process the data and |
16 |
publishes the results. The data sets could be quite large, something |
17 |
around the order of an 8 hour video or something (or a 24/7 real-time |
18 |
video stream). Initially there might be up to 100 simultaneous |
19 |
connections but it needs to scale to the 1000's. I need to be able to |
20 |
process the data as soon as they start uploading/streaming (so clients |
21 |
dont have to wait for the complete upload and then wait for processing |
22 |
to get their report). |
23 |
|
24 |
So I think I've familiarized myself with some basic terminology so I'll |
25 |
toss out a few comments/Q's etc. |
26 |
|
27 |
I've run across some software that claims to be "QoS middleware",the |
28 |
_ADAPTIVE <http://www.cs.wustl.edu/%7Eschmidt/adaptive.html>_ |
29 |
Communication Environment for example. Its important that all data from |
30 |
all streams gets processed in a timely fashion and without error, |
31 |
dropped packets etc. so I imagine this is something I will have to |
32 |
consider? It seems to have a good corporate backing (giving it |
33 |
credibility i suppose). Anyone use it? |
34 |
|
35 |
I've also run into CORBA. I read "optimized for high-performance and |
36 |
real-time systems" which sounds like my scenario, but I also read that |
37 |
its meant for distributed computing environments. I'm assuming an HPC |
38 |
cluster counts as a distributed computing environment but, not 100% |
39 |
sure. Nor am I sure that this is the technology I'm looking for. Perhaps |
40 |
theres other alternatives? Other standards? |
41 |
|
42 |
And even though I've taken a real-time systems course, I'm not convinced |
43 |
my server is "real-time" in the scheduling or embedded sense. I don't |
44 |
think I need a real-time OS although I will need to do load-balancing, |
45 |
memory management, running concurrent process, MPI's etc etc. |
46 |
|
47 |
So, am I barking up the right tree? In an ideal world I'd just hire |
48 |
someone with 10 years experience on the subject but alas resources are |
49 |
scarce and I have to suffice with myself pretending to be a sys admin |
50 |
for the time being. |
51 |
|
52 |
Any and all help is much appreciated - now back to research, reserach, |
53 |
research. |
54 |
|
55 |
Cheers, |
56 |
|
57 |
Chris |
58 |
|
59 |
|
60 |
|
61 |
|
62 |
|
63 |
|
64 |
|
65 |
|
66 |
|
67 |
-- |
68 |
gentoo-server@g.o mailing list |