Gentoo Archives: gentoo-dev

From: Igor <lanthruster@×××××.com>
To: Naohiro Aota <gentoo-dev@l.g.o>
Cc: Emery Hemingway <emery@×××××××.net>
Subject: Re: [gentoo-dev] Portage QOS
Date: Sun, 12 Jan 2014 10:28:21
Message-Id: 52d26e33.cf38700a.6dcc.2d5f@mx.google.com
In Reply to: Re: [gentoo-dev] Portage QOS by Naohiro Aota
1 Hello Naohiro,
2
3 Saturday, January 11, 2014, 7:00:58 PM, you wrote:
4
5 Thanks! I'll peek into it today.
6
7 Emery Hemingway all so generously offered his help.
8
9 What I plan to do is to share the system design with the people who thinks
10 the same. I won't mind if either of us will implement it when it's
11 partially ready or we all may contribute to the cause when it's ready.
12
13 My personal goal is to help Gentoo a bit as it's free project I don't care much
14 about propriety.
15
16 Let's know each other a bit better.
17
18 I'm good with C++/PERL/PHP/JS/HTML/FASTCGI, hardware, services, like Apache, MYSQL, bind.
19 I've experience of designing systems that handle about 1 million hosts daily.
20
21 The most problematic part in this project apart from a political task of getting it into the portage
22 is going to be the load that it can handle. In our case the load is a killer, if this system is slow or
23 unstable it will be deployed in portage.
24
25 I don't know how many users gentoo has world wide I have to design the system scalable, such systems
26 can hold as many host as there is hardware/bandwidth available.
27
28 Theoretically there might about 100 000 ebuild done each second world wide. The hardware that I have at
29 the moment if we design everything right can handle about 500 failed and 1000 total|second not more.
30
31 But I hope there will be no more than 1000 ebuilds/second load at the beginning.
32
33 Let's register a domain portageqos.org and create a mailgroup on it so we could communicate more easily.
34
35 It's all so possible that before we have PortageQOS ready somebody else will deploy the same and there will
36 be no need in this project any more. These are risk we would all have to accept.
37
38 But it's highly unlikely that there are many people who are willing to implement PQOS service and who has
39 enough experience and hardware to actually launch this project.
40
41 If we ever launch we would need to deploy in phases. First phase - we need to deploy first 1000 systems
42 say, whose name is starting from a-c then we watch how we're doing and how much load we brought. Then we
43 go further with alphabet extending the number of connected hosts.
44
45 If we reach max load on the infrastructure we would have to be ready to extend it.
46
47
48 > LTHR <lanthruster@×××××.com> writes:
49
50 >> Portage QOS
51
52 >> Hi All,
53
54 >> What do you think about implementing this:
55
56 >> http://forums.gentoo.org/viewtopic.php?p=7477494
57
58 >> I've system design in my head and could write it down with the
59 >> implementation details.
60 >> Then may be we could all review it and get to something we all agree
61 >> upon then I could
62 >> try getting a team and implement it.
63
64 >> Just a brief question - does anyone know how many ebuilds are
65 >> assembled world
66 >> wide each second?
67
68 > This is quite impressive for me. I'm one who have been thinking like
69 > this. For the purpose, I started a Web service named GenTwoo on May
70 > 2011: http://gentwoo.elisp.net/?locale=en
71
72 > First a GenTwoo user login with Twitter account to the service and
73 > installs a tiny script [1] on their Gentoo box. Then each time they run
74 > emerge, the script collect what package is emerged, if the emerge is
75 > succeed or failed, its elog output, and its build.log (only if the
76 > emerge failed). These information is sent to my server and tweeted
77 > periodically with "#gentwoo" hash tag so that you can see your friends are
78 > heating their computer :-). [2]
79
80 > You can also browse:
81
82 > - What all GenTwoo users emerge recently
83 > http://gentwoo.elisp.net/emerges?locale=en
84 > - What one user emerge recently
85 > http://gentwoo.elisp.net/users/naota344?locale=en
86 > - How a emerge failed (click "error log" tab)
87 > http://gentwoo.elisp.net/emerges/644259?locale=en
88 > - Recent popular packages
89 > http://gentwoo.elisp.net/poppackage?locale=en
90 > - How long dose it take to emerge a package (and its average)
91 > http://gentwoo.elisp.net/packages/app-text/poppler/?locale=en#0.24.5
92
93 > Since I'm not much advertising the project, there are not so many active
94 > users (and they are mostly Japanese): there are only 54 users who ever
95 > emerged since Dec 2013. I'm not sure my GenTwoo project completely suit
96 > your demand, but there are already many emerge record on my server
97 > (645112 emerge records since the service started, and 22129 emerges
98 > since Dec 2013). So if you are interested, you can start "PortStat" or
99 > "PortStatDEV" implementation immediately with the emerge data I have. Or
100 > you can also join GenTwoo if you feel it's better to start from
101 > scratch. I'd appreciate it if you would join GenTwoo project and improve
102 > it together ;-) There's also on going project to rewrite GenTwoo into a
103 > package stat [3]
104
105 > Anyway, feel free to ask me GenTwoo's implementation detail if you are
106 > interested in it. I think I can help you write your design idea if you
107 > start from scratch with your idea.
108
109 > [1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py
110 > [2] https://twitter.com/search?q=%23gentwoo&f=realtime
111 > [3] https://github.com/gentoo/GenTwoo-backend/
112
113 > Regards,
114
115
116
117 --
118 Best regards,
119 Igor mailto:lanthruster@×××××.com