Re: [gentoo-dev] Considering dropping the hardened toolchain (A Quantitive Approach)

[Ned Ludd]

Alex know I love and respect your opinion but with you being mostly gone
and my own business starting is suffering due to the large amount of
time the project takes I'm not seeing a whole lot of alternatives unless
we get some help from some hard hitters.

Yes we could leave the patches in and _hope_ that things continue to
work. But who is going to watch the bugs and verify that something is
not a flaw in our own design? how do we deal with fundamental design
flaws with upstream security patches when they don't care? (I know you
know this one all to well) example
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132149

Where is our core documentation? Where is the comparative analysis of
security measures? Who is going to maintain our appropriate profiles?
Keeping up with virtuals is nearly a task in and of itself. Rolling
stages and working towards things like livecd's. These are fundamental
things that have to be done on a cycles that follow gentoo mainline. how
about aiming for getting some of these solutions to go gentoo mainline.
All suids and daemons for example should really be compiled the way we
do. But that's an uphill battle that I'm pretty sure I don't want to try
to tackle alone.

I see things like users that have completely uninstalled the
hardened-toolchain to work around small bugs when they don't have to. I
can't comment on every bug to say here is another way you can get the
same end goal without them having to dismiss the solution completely, be
that their own ignorance or ours it's a disservice to our community do
nothing.

Being on this team means more than just watching the "hardened" bugs
that get routed to us and replying to emails once every few weeks, one
has to watch all toolchain bugs and look for how things correlate to
each other and determine the right thing to do. 

Other distro's are wanting to adopt similar to the same solution but
those guys need to be helped so they don't make fatal mistakes. Yes it's
a very good sign when others want to adopt it. But that's also time
consuming. Ask yourself should we keep our dear precious all to
ourselves or dedicate some time and effort to other projects of the
world. I think we should as most of these designs are being developed in
house and we by far have the most experience.
See http://d-sbd.alioth.debian.org/ for more details.

How are we going to handle the other arches.
ppc/ppc64 These two arches could be added to mix relatively easy. But
**** we can't even get one of those teams to test simple kernel patches
that were developed especially for them after multiple requests.

Peter has many questions that he needs definitive answers to. With those
questions going unanswered and him being a key player these days in your
absence we could be potentially introducing more bugs than need be.
He is/was happy with his own homegrown build system and does not really
need to support us or our efforts.

You want to help pappy? Spend your time being proactive vs defensive.
Help me train some people so the workload is reduced on us and our time
can be better spent focusing on making the next steps with more ease.

After planting the initial seeds one day and seeing things being
maintained properly long term I'd like to be just a user ya know.

On Sat, 2004-09-18 at 11:54, Alexander Gabert wrote:
> Ned Ludd wrote:
>
> >  I really never wanted to send a mail like this but I don't know what
> >  else to do. ;/
> >
> >  Due to low positive feedback and user input I'm considering dropping
> >  the hardened toolchain
>
> Sat Sep 18 16:14:04 CEST 2004
>
> Subject: A Quantitive Approach
>
> Dear Solar,
>
> it does not surprise me that you are proposing a "change for the better"
> with
> dismissing the hardened toolchain.
>
> But you should think about the impact.
>
> Does the solution itself offer such poor quality that it should not/cannot
> survive?
>
> I think you are forgetting about one more important fact than unfixed bugs.
> I know that these bugs are mainly existing due to low manpower,
> which is also majorily my fault because i promised to commit to
> the solution in the past and did not deliver a considerable amount of
> time and
> work to fulfill my responsibilities at the Gentoo Hardened team.
>
> But before you are starting to put solutions to an end of life, i would like
> you to consider the benefits and the misunderstandings from a more
> "objective"
> point of view:
>
> When a beginner starts using the hardened toolchain (which is not what
> it was
> designed for), he or she will immediately suffer from deep impacts of our
> "evil" compiler modifications.
>
> We create documentation as easy as possible to make the "entry point" to the
> solution low.  But that does not mean an ape with a joystick can fly a
> rocket.
>
> You have to follow me on the thought that if someone with a professional
> attitude
> will see bugs and make up his/her own mind about those bugs with a
> certain degree
> of understanding what is going on, he or she is going to find the cause and
> report it back to us without stupid comments like "your shit does not work"
> (which is not a quote from a bug but what comes into my mind when i read
> many
> bugs, and yes, i read them)
>
> We give the tools- the people use it.  Thats the game, that is how it
> always has been.
> We make things easy.  But we cannot make things so foolproof that you cannot
> cut your throat with it.
>
> With compilers and toolchains its different than with media players or name
> servers.  Either a media player works or it plays videos too fast or does
> nothing- like my mplayer does sometimes.  People open bugs about mplayer.
>
> What people expect from mplayer is that it plays video.
>
> What security people expect from a hardened toolchain is that it compiles
> software in a defined and security-oriented fashion.
>
> Peter Mazinger is working very hard and he is successfully delivering
> the needed compiler modifications.
> The PaX patch, provided by the PaX team, is the dark eminence (i.e. kernel)
> behind the scenes that makes the things work as expected (and defined).
>
> What are the reasons that you want to postpone those two cornerstones of
> basic
> security improvements over default userland and default linux kernel?
>
> Do you call it a "failed experiment"?  No.
>
> You are just telling people to get up and support it or you will cancel it.
>
> As a project manager for the toolchain you can order your developers to
> get up
> to date with open patches and take care of the user base.  You are right.
>
> But what user base and feedback do you expect?
>
> The people using it do not file bugs, because they understand that
>
> a) using bleeding edge software may/can break shit
>
> b) using hardened toolchain to emerge (from a security perspective)
> unneeded stuff can break more shit
>
> Who said that the hardened toolchain can be blamed for people using it,
> compiling
> their desktop machines and after that joining the #channel and/or
> opening bugs
> without a slight degree (or as a matter of fact: interest) in
> understanding or even
> WORKING with the solution.
>
> Normally, people open bugs to get something fixed.
> But face it: mplayer will not be fixed.
>
> Do you think i am proud of the bugs?  You know that i am not and i know that
> it hurts you to see our stuff getting so "dissed".
>
> Trust me, they all want "security out of the box" and maybe our
> documentation is misleading
> and trying to "promise" that in a way.  But it is not wrong, the
> documentation.
>
> Lets go somewhere else because i was discussing "low entry level" and
> USE flags
> for ease of customer experience already above.
>
> We started this project as pioneers.  Now we are experienced users of
> our own
> solution.
>
> When people were asking me for "full scope" coverage of the hardened
> solution
> for the full Gentoo userland, it took me a very long time to think about
> possible ways for making "transparent" defuse switches.
>
> Then i said: it can be done, but only with my tricks.  My tricks have
> not been
> accepted by the developers (see discussions on public lists) and i
> agreed that
> it was not possible without losing important factors like "visibility" of
> changes and "reproducability" of toolchain and compilers involved.
>
> But those problems arising now were created by a simple mathematical
> function:
>
> Ignorance * People * Time / Commitment to the Solution = number of open
> bugs.
>
> If the commitment to the solution by the people opening bugs were high
> enough
> to "keep it going", we would not have to bother about too much "ignorance
> bugs".  I know this sounds harsh.
>
> And it does not mean to "insult" our customers and users.  But face it.
>
> If ignorance were near zero, this equation would hold our bug level nice
> and low.
> But it does not.
>
> If the number of people were low, it would also be nice and low.
>
> If the time had not shown the major deficiencies (which the solution clearly
> has), it would also not have become such an imminent point of "turning
> around"
> now.
>
> You are asking what to do.  I think that is just fair because it shows your
> community spirit, your true efforts and your heart for the solution and the
> Gentoo team.
>
> You don't want anybody to blame us for letting people wrestle in the mud
> of a
> "broken" hardened toolchain "solution".
>
> But as is said before: this is a team of volunteers- and volunteers means:
> people "sacrifice" their free time for the project and the ongoing
> quality of
> the project.
>
> If quality is low, it does not mean that the developers are stubborn fools.
>
> To give another mathematical example:
>
> free time * commitment = quality.
>
> If free time is low (which has been the case in my personal situation), the
> quality cannot be high.
>
> And i also know that you never suspected the commitment of team members
> to be
> under a needed level that the solution can "go on".
>
> When you say: "dropping the hardened toolchain", to me it sounds like "stop
> riding a dead horse".
>
> But, in my eyes, you are underestimating the negative impact of that
> decision on people
> successfully using the solution.
>
> Do you need success stories for letting it continue?
> Do you need mails of people that tell you: good job, things broke left and
> right of me, but i am a proud owner of a hardened gcc.
>
> You and me know that you will never get such mails.
>
> People do not bother.  They only bitch and whine when it breaks and the
> house
> is burning down because they have been playing with the dynamite and at the
> same time forgot the lit cigarette in their mouth.
>
> This is what you have been doing in the last weeks: collecting examples of
> "bad feedback".
>
> And you can declare the "low positive" feedback as that any "high positive"
> feedback does not exist.  Of course it does not- reason is above.
>
>
> Does the "large" userbase really "ad hoc" enable the hardened USE flag and
> then loads of workstations go down the drain and bugs.gentoo.org suffers
> major
> headache from people opening hundreds of duplicate bug requests?
>
> If you are comparing the negative impact of hardened gcc to the "mistakes"
> made by some glibc and gcc updates in the past, i cannot see a reason why a
> "large user base" that is reluctant to work with us and find solutions
> should
> be a reason for us to "retreat" from our positions of zero tolerance towards
> high security systems- this includes kernel, compilers and other tools like
> full runtime bounds checking (remember that the libmudflap source again
> did not
> make it to mainstream gcc).
>
> To be honest, your approach is too simple.
>
> You say: it does not work, lets drop it
>
> I say: WORKSFORME
>
> Peter Mazinger is putting big efforts in creating patches.
> The PaX kernel can go to its full strength on a Gentoo Hardened system.
>
>
> It takes people who think to implement security.
>
> We give the tools.
>
> People use tools.  Did you know that you can cut your throat with a
> screwdriver?  I do not remember seeing warnings on screwdriver not to
> cut your
> throat while operating it.
>
> We cannot be personally held responsible (or liable) for the ignorance of
> people who want "security out of the box".
>
> See you at the bitter end,
>
> Alex
>
> (And yes, this all could have been said with less than one quarter of the
> character count, but i was in the mood of defensive arguments ;-)
-- 
Ned Ludd <solar@g.o>
Gentoo (hardened,security,infrastructure,embedded,toolchain) Developer