[gentoo-server] Starting point for HPC client-server interface.

[Chris Fairles]

Greetings,

Let me start off by saying I'm quite new to network programming so I 
might ask some dumb questions and you might have to dumb-down your answers.

So let me paint the picture of what I need to accomplish and perhaps the 
wisdom of all gentoo-server'ers can point me in the right direction.

I have a cluster with a handful of quad-opteron blades (running Gentoo). 
This is the server.

I also have clients (mostly windows users) out in internet-land who need 
to, through a web interface, upload files and/or point some data-stream 
to be sent (presumably over tcp/ip) to the server. Once it hits the 
server, an application process gets spawned to process the data and 
publishes the results. The data sets could be quite large, something 
around the order of an 8 hour video or something (or a 24/7 real-time 
video stream). Initially there might be up to 100 simultaneous 
connections but it needs to scale to the 1000's.  I need to be able to 
process the data as soon as they start uploading/streaming (so clients 
dont have to wait for the complete upload and then wait for processing 
to get their report).

So I think I've familiarized myself with some basic terminology so I'll 
toss out a few comments/Q's etc.

I've run across some software that claims to be "QoS middleware",the  
_ADAPTIVE <http://www.cs.wustl.edu/%7Eschmidt/adaptive.html>_ 
Communication Environment for example. Its important that all data from 
all streams gets processed in a timely fashion and without error, 
dropped packets etc. so I imagine this is something I will have to 
consider? It seems to have a good corporate backing (giving it 
credibility i suppose). Anyone use it?

I've also run into CORBA. I read "optimized for high-performance and 
real-time systems" which sounds like my scenario, but I also read that 
its meant for distributed computing environments. I'm assuming an HPC 
cluster counts as a distributed computing environment but, not 100% 
sure. Nor am I sure that this is the technology I'm looking for. Perhaps 
theres other alternatives? Other standards?

And even though I've taken a real-time systems course, I'm not convinced 
my server is "real-time" in the scheduling  or embedded sense. I don't 
think I need a real-time OS although I will need to do load-balancing, 
memory management, running concurrent process, MPI's etc etc.

So, am I barking up the right tree? In an ideal world I'd just hire 
someone with 10 years experience on the subject but alas resources are 
scarce and I have to suffice with myself pretending to be a sys admin 
for the time being.

Any and all help is much appreciated - now back to research, reserach, 
research.

Cheers,

Chris









-- 
gentoo-server@g.o mailing list