Re: [gentoo-dev] portage + bittorrent anyone?

[Jason Toffaletti]

This email includes my responses to multiple emails, so keep reading....

On Monday 16 February 2004 10:27 am, Fred Van Andel wrote:
> Bittorrent is not a good mechanism for sharing gentoo files for the
> following reasons:
>
> Bitorrent requires a separate instance and a separate open port for every
> file that you are sharing. When some people will be sharing hundreds of
> files this is simply not a workable approach.

This is the work that needs to be done I mentioned. Rather than creating a new 
p2p system, we could slightly adapt bt to our needs. I think the changes 
would probably make it back into the official bt too.

> Bittorent is optimized towards dealing with a small number of large files
> under very high (initial) demand situations. Our needs are different, we
> are dealing with large number of much smaller files under moderate demand
> and without the initial spike in demand.

I am theorizing here without any statistics to back this up, but I think we do 
have an initial spike in demand when new source tarballs are released. I 
think we'd need to study the traffic on gentoo mirrors, but my theory would 
be that most traffic is from what is released in the past 2 or 3 days. We 
also do have a few large files to deal with such as the ut2004-demo. I also 
don't think bt would perform any worse with small files than big files.

> The reason that bittorrent works so well is that it forces everyone to
> share if they want to download, consequently there is a lot of upload
> bandwith available on the network.  Any system that forces sharing would be
> able to work as well.

Many systems force people to share to download, but none work as well as 
bittorrent because they don't use a good algorithm for transfering the file. 
I think you'd be hard pressed to find a better way.

> In theory I am working on a P2P system specific to the needs of gentoo,
> however I must admit that I have not worked on it for a while.  Will
> someone please kick me in the *ss to get me working on it again.

Again, why create something totally new and specific to gentoo when bittorrent 
just needs a few modifications. We'd be contributing to OSS, the entire 
BitTorrent community, and Gentoo.

------------------------------------------------------------------------------------------------------------
On Monday 16 February 2004 05:57 am, purslow@×××××××××.ca wrote:
> Bittorrent has more of a place w binary distros, eg Mandrake,
> where it was welcomed as a big improvement for downloading a new release.
> but there you want to get  3  ISOs all at once, as does everyone else,
> whereas here on Gentoo you download much smaller files
> & everyone is grabbing different pieces at different times,
> so it doesn't seem to have much relevance.

I'd like to see traffic statistics from mirrors to back this claim up. I would 
guess the exact opposite: most people will be downloading whatever has been 
released in the past few days.

------------------------------------------------------------------------------------------------------------
On Monday 16 February 2004 05:36 am, Christian Gut wrote:
> Hi Jason!
> i would not even think of using that without some strong gpg-like
> signing of the source-files.

I'd welcome you to read more about how bittorrent works, but you basically put 
your trust in the source of the torrent file. It contains md5 hashes for 
every 1k block of the file so you are able to verify every block you download 
from a source. If a source is giving out bad blocks, it will get forced off 
the network. I think it is just as secure as the current method of 
downloading tarballs and trusting the md5 in the ebuild.

------------------------------------------------------------------------------------------------------------
On Monday 16 February 2004 01:25 am, Svyatogor wrote:
> How many users you think, will actually do that? I.e. share their
> distfiles dir. I wouldn't do it...
>
> P.S. That not to mention, that for me Bittorrent always gave lower
> speeds than just a descent mirror (ADSL 600 KB/s connection).
>
> Wkr,

I think lots of users would, but it would probably be a good idea to take some 
kind of poll before putting work into the project. You would of course be 
able to limit the amount of bandwidth you'd be sharing. I think lots of users 
would be willing to share 5k/sec or so of bandwidth all the time. The mirrors 
could even run seeding bittorrent clients, so there would actually be more 
total bandwidth available and it'd be distributed better.

------------------------------------------------------------------------------------------------------------
On Monday 16 February 2004 01:25 am, Matt Wilson wrote:
> I like the idea, don't suppose you've had any other positive feedback?
> Could be the start of a new portage branch (well, sort of).

I haven't had much positive feedback from this list as you can see. In the 
#gentoo-amd64 irc channel I got a lot of positive feedback from users who 
would be willing to donate bandwidth and thought that it was just the sort of 
progressive idea gentoo is known for. I wouldn't have even proposed the idea 
here if it wasn't for their encouragement.

--
gentoo-dev@g.o mailing list