[gentoo-soc] Gsoc 2009: Online Image Builder

[kartik rustagi]

Ok. So this is a rough draft of the steps that will be involved in the
building of an Online Image Builder. There are few doubts in my mind
regarding the final stage of the process, which are listed at the end.
Many Ideas over here have been given by solar. I will request all of
you to kindly go through this and suggest me the many shortcomings in
the process and if possible then may be answer my queries at the end.
Here it goes, step wise:


Steps on the user/client side:

The client will accessing a web page to avail the service. For this
reason the only scripting language that can be used on the client side
is Java Script. Most of the browsers support only Java Script as the
only scripting language. There is some work going on and their might
be an expected release of Iron Python in Mozilla Firefox 4. So at this
stage all the client side work will be done in js. Steps:

1) All the requirements like removing initial dependency, user select
option etc. , the JS script for these tasks will be hard coded in the
html file. This will significantly reduce client-server interaction
which usually slow down the process.
2) Their is no standard implementation of xmlrpc parser in js though
many are available. But I will be making a parser of our own. It wont
be much trouble, simple Data Object Model scripting. Completing this
step gives the ability to encode and decode (messages,parameters) in
the XMLRPC format.
3) Again since their is no standard implementation of xmlrpc in js so
all the requests will be sent and received using XMLHttpRequest
object.
4) There is a mock up of the interface at the client side( curtsy
solar) at http://tinderbox.dev.gentoo.org/image/
5) We can have a basic authentication system (Both http and https
supports this) :http://user:pass@host:port/actual_server_path



Steps on the Server side:

To handle the network traffic we can either go with normal blocking
modes using threads to handle multiple clients. But according to me a
better aproach will be to use asynchronous server, I have the one
developed by twisted in mind:
http://twistedmatrix.com/trac/wiki/TwistedProject
This will effeciently take care of multiple clients. The twisted
server will pass on all the request to a python  object of
SimpleXMLRPCServer.SimpleXMLRPCServer(addr[, requestHandler[,
logRequests[, allow_none[, encoding]]]]). Their onwards all the
processing will be done by this object.
1) All the functions will be added using
SimpleXMLRPCServer.register_function(function[, name]).
2) Whenever a request come appropriate function will be called.

Now once we have the complete configuration from users, comes the task
of actually gathering all the packages and binding them in a user
defined output form (eg:- .iso, .tar.gz etc). I had a detailed
discussion regarding this with solar and this is what I got from what
he suggested (apologies to solar if I misinterpret any of the steps).
Steps:
1) User request will finally be made available in the form of
.conf(regular text file) something like
http://tinderbox.dev.gentoo.org/image/feeds
2) Now each of the package in the above file consists of feeds like
http://tinderbox.dev.gentoo.org/embedded/openmoko/Packages
3) To speed up the process of collection of packages from these feeds,
they can be broken down into sets such that each set can be treated as
a meta package. These sets will be rsync simultaneously into a common
directory.
4) After all the packages have been fetched we do a final packing
producing the archives of package fetched in the user given format
like .iso or .tar.gz (using different utilities).

My Doubts:
1* Solar suggested to break feeds into set such that each set
resembles a meta package. How can this be done without human
interaction (obviously)
2* Is their any step missing between step 3 and step 4 above.

Thanks for going through this. I request you to kindly find the time to reply.

With Regards
Kartik Rustagi