Re: [gentoo-dev] the graveyard overlay

[james]

On 07/08/2016 04:50 PM, Alec Warner wrote:
>
>
> On Fri, Jul 8, 2016 at 1:21 PM, Philip Webb <purslow@××××××××.net
> <mailto:purslow@××××××××.net>> wrote:
>
>     160708 William Hubbs wrote:
>     > On Fri, Jul 08, 2016 at 05:56:04PM +0300, Andrew Savchenko wrote:
>      >> IMO the criteria should be whether they work or not,
>      >> not whether upstream is more or less active.
>     >> If they're blockers on other work, by all means cull them.
>     >> However, if the biggest problem with them is
>      >> that they're using a few inodes in the repo, they should
>     probably stay.
>      > There is an overlay for packages that are removed from the
>     official tree
>      > -- https://github.com/gentoo/graveyard --
>      > and that is where old software should go,
>     > if it doesn't have an active maintainer.
>
>     A lot of this lengthy discussion is missing some basic points,
>     though a few people have mentioned them in passing.
>     As someone who has used Gentoo exclusively since 2003
>     & who raised the objections to removal of Xcdroast + Nethack,
>     let me try to get you all to focus on the real-life issues.
>
>     (1) The fact that a pkg has little or no upstream support
>     or that it doesn't have an active Gentoo maintainer
>     is not a reason for removing it from the regular tree.

+1

>
> So basically what you are advocating for is:
> "Having completely unmaintained packages in the tree is OK."
> And honestly, I do not buy that premise.

I think what Phillip has outlined is that different use cases have 
different prospective on use, stability and security.  Take the ancient 
serial (rs232-C) code based net-dialup/minicom for example. It has long 
outlived it's usefulness for most users. Serial ports that have TCP/IP 
laid on top of them use to form the backbone of phone line modem access.

"Upstream:    None specified"

Yet it is still extraordinarily useful to embedded systems work, and 
industrial uses, such as modbusTCP/ppp/rs232. To a "go hacker" it needs 
to be cleaned out, but to an older or embedded hacker it is fundamental 
to raw hardware access. Granted the "Maintainer:  embedded@g.o 
(Embedded Gentoo)" has stepped up. So, my point is there are (probably)
lots of other example codes in the 1500 unmaintained packages that have 
not yet been scrutinized by the larger gentoo user community. I've got 
over a dozen deprecated packages in my /usr/local/portage/ tree, as I 
try to keep up with the tree cleaners on old useful codes.

"no maintainer" should not be part of the criteria for removal, at least 
not for a year or so. WE need to document the methods and procedures for 
proxy-maint, to a robust level, before such actions are warranted. Then 
a campaign to recruit proxy maintainers for a while.
Right now the devManual, quizzes, repo structures(github, sunrise etc), 
eapi, and many other aspects of gentoo are in a phase of rapid change. 
It's going to take a while to get things stabilized and documented and 
then a period of recruitment. I know, I for one have been on this path
and things are time consuming. Just look at the dev-wrangling over many 
of these issues. That is the reason we have 1500 orphaned packages.

For now, I suggest we just label these packages as deficient, 
no-maintainer, security, inactive-upstream and any other relevant 
statuses. While some parse the proxy-maint correspondences to first 
create FAQs and then a guide, with some basic examples. Then aggressive 
removal would be viable and community supported.

This effort would also bode well for corporations to train and maintain
their linux sources with lots of folks participating in a full-spectrum
structure, that these discussions illuminate, should we want to attract 
new and younger hackers to gentoo. A well defined proxy-maint program 
would do more for gentoo recruiting (of both devs and users) than most
any thing else, imho. Teach a user how to create and maintain their own 
code, and you'll have a user for life.


>     One basic reason some software is no longer being actively developed
>     is simply that they work perfectly well as they now are,
>     eg the file manager Krusader & the desktop manager Fluxbox :
>     both of these are very useful & have no drop-in replacements,
>     but very little development has occurred for several years.
>     The same is true of Xcdroast & Nethack, which have been threatened,
>     but which have been rescued after some small patches have been applied.
>     This is likely to be true of more + more pkgs, as time passes :
>     even changes in the kernel these days rarely affect desktop users.

!1.

>
> No one is trying to remove flubox (which had a release in 2015 and had
> activity in its git repo as recently as last week.)
>
> Xcdroast for example, hasn't had a release in 8 years and I can't even
> find its source tracker in sourceforge. These are the sorts of packages
> that I think are not great to have in the tree and for Xcdroast, if I
> were treecleaner lead i would probably advocate for working around the
> security bug (dropped SUID) instead of removal. I do not necessarily
> want to remove software that people are using.
>
> That being said, I do not want unmaintained software in the tree either.

Funny, I used xcdroast to burn a couple of systemrescue images, just 
yesterday, and I never used it before yesterday.


>     (2) There are  3  basic categories of Gentoo user :
>     (a) server-farm managers, (b) multi-user sysadmins, (c) single-users.
>     Each of these have different security concerns :
>     (a) need to be alert to the many threats from all over the Internet ;
>     (b) need (among other things) to prevent privilege escalation ;
>     (c) are largely immune to those types of threat,
>     though a few of the Internet variety can affect them.

great perspective. I'd add a forth category of user. 
Embedded/close-network users of gentoo. For products, you do not need to 
use the internet, until the very end of product development or product 
testing.
Old codes really shine in this forth category, and gentoo is definitely 
one of the champions in this stand-alone space. We are the only major 
distro to support both systemd and legacy init. It's not clear how many 
embedded projects will even use systemd. (think new arm boards that we 
all are anticipating).


> I appreciate the argument you are trying to make; but i do not think it
> really drives Gentoo Security Policy (nor should it.)

Basically the current gentoo security policy is a "greedy algorithm" 
that too aggressively makes too many mistakes (on what to remove) by
a limited scope of how linux is used in the wider world. Not every linux 
system is a multi-user internet connected risk. Granted Rf interfaces 
may change that, but most responsible designers have a way to 'turn 
down' RF interfaces. Surely a hacker can identify the single trace that 
is the antennae connection and ground it out, should the manufacture not 
provide a software switch. That is a vendor choice of Rf turndown is a 
problem in the new IoT family of products, but you can blame the NSA for 
that sort of crap. Sot if a system is not connect to the outside world, 
99% of the security risks disappear.

Please stop assuming the worst case environment that a code is used
for. If you want that, then create a tree within a tree, such as what 
the good folks of the 'hardened' project do. Work to assist them to add
codes to the musl platform, rather that working the fringes of the 
existing (rich) portage tree.

If you did that, then we could all (easily) routinely secure our 
connections to the internet, and keep other projects alive that do not
need a 'security onion' [1] level of security. Perhaps the security team 
needs to mimic the security onion project, but build everything on 
hardened gentoo? That effort would do vastly more to secure gentoo 
users, than fuzzing codes, one at time. We have Pentoo for penetration, 
but where is the equivalent for defense?

[1] https://securityonion.net/


> As my security manager used to say "security is not a race to the bottom."

Anecdotal limericks are mostly useless in real security. Real security, 
starts at the overall schema. Where was the security project on the 
recent malaise of check sums not matching for minimal iso? It appears to 
many that the security team, needs to be "refocused" and more amenable 
to valid suggestions and a  new set of priorities, imho. ymmv.

>     The security objections raised against Xcdroast + Nethack
>     were both problems which would arise only on multi-user systems,
>     yet single-users were also to be deprived of access to them.
>     Perhaps part of the problem is that many Gentoo developers
>     also earn their livings as sysadmins with many users or many servers :
>     the simpler happier world of single-users escapes their attention.

Excellent point!

>     (3) Users generally don't want to be developers : they're too busy
>     or too old.
>     Asking them "Are you willing to maintain it yourself ?" is a silly
>     excuse ;
>     offering them the chance to dig around in a graveyard is even worse ;
>     even maintaining an overlay is a nuisance : I tried it with KDE Sunset.
>     Neither Xcdroast nor Nethack belong in a graveyard of any kind :
>     once the obscure security problems have been fixed,
>     they belong in the regular tree marked 'stable',
>     like many other pkgs whose development has been completed.

Any effort to separate packages from the tree, should *first* be meet 
with a robust way to maintain and retrieve those codes. Destruction of a 
rich legacy of codes, many simple and tightly focus, robs gentoo or
a very rich part of it's heritage, needlessly.


>     Users all do -- or should -- appreciate the unpaid work of the
>     developers,
>     but developers also need to realise that without non-developer users
>     Gentoo would very quickly die & their justified pride + satisfaction
>     die too.
>
>
> I'm a bit confused by this argument.
>
> 1) It appears that no Gentoo developers want to maintain a software package.

(um, you missed the keyword that is missing, "currently")

> 2) The software package has no active upstream.

One fork fixes the no active upstream.

> 3) The software has no bugs.
> 4) We mask the software because it has bugs and no active maintianer,
> for years.

The word "bugs" means many things to different folks. It's a 
full-spectrum word that you use as an inaccurate 'monolithic' metric.

> 5) No one volunteers to proxy-maintain the software.

This is because of numerous, aforementioned changes, including but not 
limited to the roll out of github, and the lack of sufficient mechanisms 
for package retrieval, compared to what we had with the attic. If there 
is a robust method via git(hub), then just document those methods, 
because git is not an intuitively obvious collect of methodologies to 
replace what we had. That and the aggressive tree cleansing is unfair to 
the wider user communities, many of which only drop in occasionally.

>
> But you advocate we keep such software in the tree, because users are
> "too busy" or "too old" to maintain it themselves?

Not a consensus viewpoint but anecdotal, see above comments.


>     (4) I have  3  simple recommendations to fix the everyday problems.
>
>     (a) the justification for tree-cleaning should be explicitly
>     that a pkg either (i) won't compile, (ii) crashes when run
>     or (iii) has a serious security hole which affects all  3  types of
>     user.
>
>
>     (b) there needs to be a developer role 'General Maintainer',
>     who should be available to look at pkgs which have no regular
>     maintainer,
>     but which compile, run properly & are generally secure :
>     their job would be to step in, like Mr Savchenko -- thanks again -- ,
>     to fix small problems which would otherwise be neglected ;
>     less formally, all developers might see it as part of their role
>     to help out occasionally with such small problems.

Good points (+1). It takes time to grow up a proxy team. It takes time 
to develop documents that are github eapi 6/7 centric and a 
proxy-dev-guide. It takes time, so stop the aggressive tree pruning, in 
the absence of a fully functional, well documented system like the attic 
that is user friendly to learn and use.

> In an ideal world, the tree would be full of properly maintained packages.

Yes we agree on this, but, the security and tree-cleaners are being too 
aggressive as the proxy workflow is new and still not fully functional 
atm. Once this is fixed, it going to take a while (6 months to a year).
How long did the github migration planning and execution take?


> There are over 1500 packages in the tree in the 'maintainer-needed'
> state[1].

So what? Dont use them (gentoo is about choice right?). Modify a 
existing, common portage tool (eix?) to label your issues with these codes.

> Even if we allocated 100 packages per developer, this "General
> Maintainer" team would be 15 developers strong and one of the largest
> projects in Gentoo. To compare the Treecleaner project is 7 people; the
> Security project is 10 people.

How about a refocus on a security onion, gentoo centric, for the 
security team. Many of us would appreciate that sort of security much 
more than removal of old packages.


> This is in fact part of the rationale of the Treecleaner project itself.
> Ebuilds require maintenance (eclass updates, new EAPIs, etc) and someone
> has to do this work (or we end up with 6 supports EAPIs in the tree.)
> This is one reason why packages that are unmaintained are removed; we do
> not have 15 spare humans to clean up the unmaintained packages, so we
> remove them when it is feasible to do so.

Correct. What you are not fairly considering is the rapid changes and 
your time allowances for the user community to adapt to all of these 
changes. Are we nixos or gentoo? At least nixos has documentation [2].

[2] https://nixos.org/wiki/Development_Environments

On Gentoo's proxy project, there is scant documentation on a myriad of 
core issues. The proxy-maint ML is still not functioning for user 
questions and FAQ parsing.


>     (c) Gentoo's rules + policies need explicitly to reflect the fact
>     that there are  3  types of user, as described :
>     eg some pkgs might be marked as 'not safe for multi-user systems' ;
>     that would recognise real distinctions which are now being ignored.
>
>     HTH & thanks as always to all of you for making Gentoo work since 2003.

Likewise. I appreciated the devs and the gentoo distro, immensely. But, 
we need a well documented pathway from start to finish on the 
proxy-maint (regardless of what the details are) before such radical 
pruning, and that includes robust archiving of what is necessary to
personally restore old packages. Use the old attic as your basis of 
reasonable functionality.


James

> [1] https://qa-reports.gentoo.org/output/maintainer-needed.html
>
>
>     --
>     ========================,,============================================
>     SUPPORT     ___________//___,   Philip Webb
>     ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
>     TRANSIT    `-O----------O---'   purslowatchassdotutorontodotca
>
>
>