Re: [gentoo-dev] Portage QOS

[Igor]

Hello Alec,

Thursday, January 9, 2014, 12:12:18 PM, you wrote:


On Wed, Jan 8, 2014 at 11:24 PM, LTHR <lanthruster@×××××.com> wrote:
Hi All,


I want to start off by discussing your premise, before embarking on the overall goals.

You wrote:
"I'm with Gentoo for many years. For various reasons many techs were not implemented and now Gentoo is in a kind of stagnation. But we can give Gentoo a new birth with relatively little efforts and bring the distro to the whole new level. "

I don't actually believe your premise of stagnation. But I can put aside my disagreement for now.


True. There is no data to tell what happens with Gentoo (to give that data is one of the goals of the 
project). We only have some formal esteems from unreliable sources. 

According to distro watch: 

http://web.archive.org/web/20130701000000*/http://distrowatch.com/dwres.php?resource=popularity

In February 2012, Gentoo distro was in 19th place. 
In December 2012, Gentoo went to 22nd place. 
In December 2013, Gentoo is down to 32nd place 

According to Linux Counter 
http://web.archive.org/web/20120101000000*/http://linuxcounter.net/distributions/stats.html

In January 2012, Gentoo distro had 5.32%
In January 2012, Gentoo had 4.04%
In November 2013, Gentoo had 4,21% 

And from my experience of Gentoo forums, gentoo.wiki - I vote for Gentoo at least not gaining new users.
If in several years the number of users is not increased - we can tell about stagnation. 

Why new users are not with Gentoo and how to get them - good question, let's find out!


 Lets talk about how the overall goal of 'bringing the distro to a whole new level' and how 'Portage QoS' will help us get there. I don't think you covered these points well in your post (I will talk about the goals more below...)


I'll re-phrase the goals. 

It's all not very well thought after at this stage but immediate goals are like this: 


* Knowledge of the number of Gentoo distros installed world wide - knowing the trend how many users choose 
  Gentoo and where Gentoo is really going down|up|stands still. 
  
  You can then try different features and see how a feature is met - if the number of systems increase 
  then this feature is probably useful. It's a strategical job, somebody at the very top of the project should 
  analyze databases and make conclusions. 

* Knowledge of the ebuild popularity - what ebuilds are popular and what are not - what ebuild to give an extra focus 
  and what ebuild could wait

* Knowledge of ebuild quality. If some ebuilds fail on many systems - something is wrong and ebuild and may be portage 
  need fixing. It's especially useful to make sure that all ebuilds have correct dependencies, missing dependencies, etc.

* A formal esteem of portage quality 
  PortageQ = (the number of successful ebuilds/the number of all ebuild attempts)

  Portage speed efficiency:  
  Average time before build starts (or download starts)

  How many times portage fails itself. 

* Immediate problem detection. If the number of PortageQ went down last day - there is some problem. 
  (then you go to ebuild stats and see what is failing)

* Reducing load on bugtracker folks - the build problems will be detected automatically and solved according 
  to their importance. There will be no need to supply bug tracker with ebuild logs and emerge --info if 
  somebody wants to report a problem. 

* Team efficiency esteem. The stats will tell what ebuilds are failing most often. 

* Team automated info. When failure rate of a certain ebuild increase the maintainer is automatically 
  informed and he can login in web-interface and see details how exactly ebuild failed. 
  The same for the portage itself. Next day a maintainer could push a new ebuild in the portage and the 
  problem might be solved.

  It's not possible not to make mistakes. But it's possible to react on their consequences fast. 

* Knowledge what kernels are used by Gentoo users, how often they update their systems, what flags 
  are used 

2nd turn goals: 

* to integrate forums.gentoo.org and bug tracker. People are offering great workarounds and solutions. But 
  they're not known to the majority of Gentoo users. 

  If a e-build fails - may be there is already a solution - and we can offer the solutions automatically from 
  portage. Like:

  There might be some work-arounds on this problem: 
  [Gentoo Forum - qt-core ebuild fails - SOLVED] 
  htpp://forums.gentoo.org/..... 

  There is a known bug on this ebuild: 
  [Gentoo Bug - qt-core ebuild fails] 
  htpp://forums.gentoo.org/..... 

* to make Bug Tracker almost unmanned. We can use gathered infromation on failed e-builds to 
  create bugs in Bug Tracker automatically and automatically set priorities according to the 
  severity. 

  The severity could be assigned automatically from package popularity and failure rate stats.

  The users with the problems could receive e-mail automatically to follow up the quick arounds
  and solutions. 

  No need to change bug-tracker version, you just need a robot who would maintain bug-tracker 
  analyazing PortageQOS databases. 


I'm sure there could be more applications of the system. 



Back to the premise of bringing the distro to a whole new level. Some of the items above I think have merits on their own, but they still don't guide me to your ultimate goal. You outlined some of what I presume to be defects in Gentoo today.


It's hard to see more goals at this stage. It's like when you're building a hammer to set a nail and 
then when you have it you discover that you may use it to pull a nail too. It's for the future developers 
to decide. 

Knowledge: So I think in general I agree, insofar as more information can be helpful in making decisions. I think you should take note that there are at least 4-5 'gentoo stats' projects that have been tried and my understanding is that none of them are in operation today.

True, one of the reasons - the whole system planning shouldn't be apart from Gentoo developers. 

I could write the whole system on paper, with tools, soft used and the system design not narrowing to the table structures.
Then everyone could contribute his experience to it or see a problem at design time. 

I know how to design the system to make handling significant load. It has a good chance 
to work fast and to be scalable. 


With the design approved by others, programming is easy. 

It's not going to be very complex.
I estimate the whole system as 10 000 - 15 000 lines max in different languages. The most difficult part would 
be to make it scalable and fast as we don't know how many ebuilds are done world wide per second and we 
have to be prepared to hot-plug hardware at any time not redesigning the system. And the system shouldn't get 
overloaded/loose functionality. 



Package blocks that portage does not resolve automatically
Slot conflicts that portage does not resolve automatically
Compile failures
...What else?

So part of the Portage QoS system is that users will submit their failures and Portage QoS will serve as a knowledge base of known issues. To me, that is still a pretty bad User Experience. Can't we just get portage to handle these issues transparently, as per http://forums.gentoo.org/viewtopic-t-977936.html




If we ask users about questions they don't know answers to it will result in users loss. But it's totally 
out of my expertise if the portage interaction is GOOD 

Users are not doing anything usual than they do right now. The reports are sent transparently by 
portage to the PorageQOS load balancer and the suggestions are received.



Just a brief question - does anyone know how many ebuilds are assembled world 
wide each second?

I bet you more apks are installed per second ;)

-A
 
-- 
Best regards,
 LTHR                          mailto:lanthruster@×××××.com




-- 
Best regards,
 Igor                            mailto:lanthruster@×××××.com