1 |
Hello Naohiro, |
2 |
|
3 |
Saturday, January 11, 2014, 7:00:58 PM, you wrote: |
4 |
|
5 |
Thanks! I'll peek into it today. |
6 |
|
7 |
Emery Hemingway all so generously offered his help. |
8 |
|
9 |
What I plan to do is to share the system design with the people who thinks |
10 |
the same. I won't mind if either of us will implement it when it's |
11 |
partially ready or we all may contribute to the cause when it's ready. |
12 |
|
13 |
My personal goal is to help Gentoo a bit as it's free project I don't care much |
14 |
about propriety. |
15 |
|
16 |
Let's know each other a bit better. |
17 |
|
18 |
I'm good with C++/PERL/PHP/JS/HTML/FASTCGI, hardware, services, like Apache, MYSQL, bind. |
19 |
I've experience of designing systems that handle about 1 million hosts daily. |
20 |
|
21 |
The most problematic part in this project apart from a political task of getting it into the portage |
22 |
is going to be the load that it can handle. In our case the load is a killer, if this system is slow or |
23 |
unstable it will be deployed in portage. |
24 |
|
25 |
I don't know how many users gentoo has world wide I have to design the system scalable, such systems |
26 |
can hold as many host as there is hardware/bandwidth available. |
27 |
|
28 |
Theoretically there might about 100 000 ebuild done each second world wide. The hardware that I have at |
29 |
the moment if we design everything right can handle about 500 failed and 1000 total|second not more. |
30 |
|
31 |
But I hope there will be no more than 1000 ebuilds/second load at the beginning. |
32 |
|
33 |
Let's register a domain portageqos.org and create a mailgroup on it so we could communicate more easily. |
34 |
|
35 |
It's all so possible that before we have PortageQOS ready somebody else will deploy the same and there will |
36 |
be no need in this project any more. These are risk we would all have to accept. |
37 |
|
38 |
But it's highly unlikely that there are many people who are willing to implement PQOS service and who has |
39 |
enough experience and hardware to actually launch this project. |
40 |
|
41 |
If we ever launch we would need to deploy in phases. First phase - we need to deploy first 1000 systems |
42 |
say, whose name is starting from a-c then we watch how we're doing and how much load we brought. Then we |
43 |
go further with alphabet extending the number of connected hosts. |
44 |
|
45 |
If we reach max load on the infrastructure we would have to be ready to extend it. |
46 |
|
47 |
|
48 |
> LTHR <lanthruster@×××××.com> writes: |
49 |
|
50 |
>> Portage QOS |
51 |
|
52 |
>> Hi All, |
53 |
|
54 |
>> What do you think about implementing this: |
55 |
|
56 |
>> http://forums.gentoo.org/viewtopic.php?p=7477494 |
57 |
|
58 |
>> I've system design in my head and could write it down with the |
59 |
>> implementation details. |
60 |
>> Then may be we could all review it and get to something we all agree |
61 |
>> upon then I could |
62 |
>> try getting a team and implement it. |
63 |
|
64 |
>> Just a brief question - does anyone know how many ebuilds are |
65 |
>> assembled world |
66 |
>> wide each second? |
67 |
|
68 |
> This is quite impressive for me. I'm one who have been thinking like |
69 |
> this. For the purpose, I started a Web service named GenTwoo on May |
70 |
> 2011: http://gentwoo.elisp.net/?locale=en |
71 |
|
72 |
> First a GenTwoo user login with Twitter account to the service and |
73 |
> installs a tiny script [1] on their Gentoo box. Then each time they run |
74 |
> emerge, the script collect what package is emerged, if the emerge is |
75 |
> succeed or failed, its elog output, and its build.log (only if the |
76 |
> emerge failed). These information is sent to my server and tweeted |
77 |
> periodically with "#gentwoo" hash tag so that you can see your friends are |
78 |
> heating their computer :-). [2] |
79 |
|
80 |
> You can also browse: |
81 |
|
82 |
> - What all GenTwoo users emerge recently |
83 |
> http://gentwoo.elisp.net/emerges?locale=en |
84 |
> - What one user emerge recently |
85 |
> http://gentwoo.elisp.net/users/naota344?locale=en |
86 |
> - How a emerge failed (click "error log" tab) |
87 |
> http://gentwoo.elisp.net/emerges/644259?locale=en |
88 |
> - Recent popular packages |
89 |
> http://gentwoo.elisp.net/poppackage?locale=en |
90 |
> - How long dose it take to emerge a package (and its average) |
91 |
> http://gentwoo.elisp.net/packages/app-text/poppler/?locale=en#0.24.5 |
92 |
|
93 |
> Since I'm not much advertising the project, there are not so many active |
94 |
> users (and they are mostly Japanese): there are only 54 users who ever |
95 |
> emerged since Dec 2013. I'm not sure my GenTwoo project completely suit |
96 |
> your demand, but there are already many emerge record on my server |
97 |
> (645112 emerge records since the service started, and 22129 emerges |
98 |
> since Dec 2013). So if you are interested, you can start "PortStat" or |
99 |
> "PortStatDEV" implementation immediately with the emerge data I have. Or |
100 |
> you can also join GenTwoo if you feel it's better to start from |
101 |
> scratch. I'd appreciate it if you would join GenTwoo project and improve |
102 |
> it together ;-) There's also on going project to rewrite GenTwoo into a |
103 |
> package stat [3] |
104 |
|
105 |
> Anyway, feel free to ask me GenTwoo's implementation detail if you are |
106 |
> interested in it. I think I can help you write your design idea if you |
107 |
> start from scratch with your idea. |
108 |
|
109 |
> [1] https://github.com/naota/gentwoo/blob/master/client/gentwoo.py |
110 |
> [2] https://twitter.com/search?q=%23gentwoo&f=realtime |
111 |
> [3] https://github.com/gentoo/GenTwoo-backend/ |
112 |
|
113 |
> Regards, |
114 |
|
115 |
|
116 |
|
117 |
-- |
118 |
Best regards, |
119 |
Igor mailto:lanthruster@×××××.com |