Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 14 Jun 2017 14:24:48 +0200
Michał Górny <mgorny@g.o> wrote:

> On śro, 2017-06-14 at 11:06 +0200, Alexis Ballier wrote:
> > On Wed, 14 Jun 2017 00:13:42 +0200
> > Michał Górny <mgorny@g.o> wrote:
> >
> > > On wto, 2017-06-13 at 12:27 +0200, Alexis Ballier wrote:
> > > > On Mon, 12 Jun 2017 21:17:16 +0200
> > > > Michał Górny <mgorny@g.o> wrote:
> > > >
> > > > > I've actually started typing the initial specification
> > > > > yesterday [1]. As you can see, banning the extra constraints
> > > > > has made the algorithms much simpler. In particular:
> > > > >
> > > > > 1. You do not have to define 'falsify' for anything other than
> > > > > pure flags -- which makes it easy to inline it.
> > > > >
> > > > > 2. ||, ??, ^^ groups are only flat lists of flags -- which
> > > > > makes reordering and processing them trivial.
> > > > >
> > > > > 3. The algorithm is recursive only on USE-conditional groups.
> > > > > This makes it trivial to make it iterative. Optimizations
> > > > > become trivially possible.
> > > >
> > > >
> > > > While you're right in one sense, you're mixing two different
> > > > things here. What you wrote *is* recursive. It does not recurse
> > > > just because you're assuming a restricted syntax. You're only
> > > > saving two things: you don't need to define how to enforce to
> > > > false (that is 3 lines not 3 pages :=) ) and you're avoiding
> > > > the nested use conditionals that are already ill defined per
> > > > the current spec (foo? bar is equivalent to && ( foo bar ) when
> > > > nested) which I believe is already another problem.
> > > >
> > > > Then, remember how I wanted to be much more drastic than you in
> > > > the beginning by killing all ||,&&,^^ etc. and keep only use
> > > > conditionals in REQUIRED_USE ?  Well, that's where the
> > > > complexity comes. The whole deal then is to define rewriting
> > > > rules for the AST so that the algorithm you describe executes
> > > > the exact same instructions but the new AST only has use
> > > > conditionals. This is more like writing a compiler for the
> > > > spec, so this does not belong to the spec and there is no issue
> > > > here.
> > >
> > > I'm looking for a compromise here. Killing those groups
> > > completely is going to make things harder for users. Keeping them
> > > with functionality limited to what's used in ~99.9% ebuilds
> > > (based on your numbers) is IMO a better choice.
> >
> > I already said I see the limited syntax as a good thing because it
> > forces devs to write constraints that have a natural interpretation
> > in how it is solved. However, you can't limit the syntax without a
> > new EAPI, and more importantly, properly solving does not even
> > require limiting the syntax.
>
> Actually, you can, via a Gentoo policy. Since solving is not required
> by the PMS, there is no rule saying it has to work for every
> constraint allowed by the PMS.

Indeed. But you're trying to make rules that it has *not* to work for
some of them for reasons that look more like laziness in trying to
understand the problem than anything else.

> Much like, you can't force a particular ordering or forbid circular
> constraints without a new EAPI. Yet you do it because it gives
> a practical improvement.

I don't see this being enforced by a new EAPI. It's more a matter of QA
tools like repoman. The main difference is that there are real reasons
for forbidding circular constraints: For true circular ones, no solver
will ever be able to solve it...

[...]
> >
> > > > [BTW: checking the rewrite rules behave properly is what I
> > > > meant by rebasing solve() on top of it and being happy with
> > > > it]
> > >
> > > Could you reiterate the current solving rules (trueify/falsify)?
> > > Are they equal to the ones you listed last, or does the current
> > > implementation change anything?
> >
> > Let's recap a bit. nsolve() is poorly named and does not solve
> > anything. It translates to implications and checks whether the
> > implications solver will always provide a valid result in one pass.
> > So, if you only care about solving rules, read your spec man. For
> > the more general case it should behave like those trueify/falsify
> > with the change that nested implications are interpreted as && (so
> > no more !(a -> b) crap to worry about).
>
> How are && ( a b... ) falsified now? Leftmost only?

$ python3 to_impl.py '?? ( a ( b c ) )'
Normalized: [|| [!b, !c, !a]]
List of implications:
  [[c, a]? => !b]

Sounds like it. This is the only "stable" way anyway.


> > If you take solve() as an implementation of your spec, you have:
> > solve(x) <=> solve(to_impl.convert_to_implications(x)) when solve(x)
> > is defined; with the added benefit that
> > 'solve(to_impl.convert_to_implications(x))' is defined and should
> > provide proper results on the whole REQUIRED_USE syntax as currently
> > defined (granted that nsolve(x) does not report anything wrong).
>
> The point is, solve() is supposed to work without any additional
> transformations.

Then I already explained how to make it work.

Transformations are out of scope of the spec and more for improving
code sharing: Those are required for checking that the solver will
provide a proper solution; if there is some flaw or undefined behavior
in the spec, then solve() might change, if both the solver and the
checker work on the same input data then at least there won't be some
desynchronization between them.

[...]
> > > > > [1]:https://wiki.gentoo.org/wiki/User:MGorny/GLEP:ReqUse
> > > >
> > > > I really don't like the reordering thing. Even the restricted
> > > > syntax does not fix the issue with '^^ ( a b ) b? ( a )' already
> > > > mentioned here. It'd be much better and simpler for the spec
> > > > just to assign a fixed value and use the solving rules with
> > > > those.
> > >
> > > You're not going to convince me by providing examples that are
> > > utterly broken by design and meaningless ;-).
> >
> > Well... if it's so obvious that the example is broken by design that
> > you don't even bother to explain why, I assume you have an
> > algorithm for that. Where is the code ? What are the numbers ? How
> > many ebuilds might fail after reordering ? How can this be
> > improved ?
>
> Are you arguing for the sake of arguing here? I just presumed that
> this example is so obviously broken there is no point wasting any
> more time on it. The code of nsolve clearly detects that, so I don't
> really understand what you're trying to prove here.

Those are real questions. You should take breath, think a bit about
it, and try to run the 2 possible orderings of the ^^ through nsolve or
even solve.py. They both are very happy (and are right to be) with
the above ordering. You might want to think a bit more about what is the
relation between this broken 10 chars example and the 10 lines python
targets one below.

You should also realize that all the above questions have already been
answered in length if you do as I suggest.

[...] 
> > As for a real world example, I'll let you find some more interesting
> > ones, but this one will probably be interesting to you and is a
> > good start:
> >
> > app-text/wklej-0.2.1-r1 ^^ ( python_single_target_pypy
> > python_single_target_pypy3 python_single_target_python2_7
> > python_single_target_python3_4 python_single_target_python3_5
> > python_single_target_python3_6 ) python_single_target_pypy?
> > ( python_targets_pypy ) python_single_target_pypy3?
> > ( python_targets_pypy3 ) python_single_target_python2_7?
> > ( python_targets_python2_7 ) python_single_target_python3_4?
> > ( python_targets_python3_4 ) python_single_target_python3_5?
> > ( python_targets_python3_5 ) python_single_target_python3_6?
> > ( python_targets_python3_6 ) vim? ( ^^
> > ( python_single_target_python2_7 ) )
> >
> >
> > Hint: It loops as written here. Reordering the ^^ in a proper way
> > makes it solvable. Putting the 'vim? ( ... )' part first makes it
> > solvable in one pass.
>
> And that's the exact case where I was considering the problem of
> ebuild- eclass ordering. And yes, putting the 'vim?' first is an
> acceptable solution here. What's the problem here?

Forget about putting vim?() first, it just makes it require an extra
pass in the best case and is not the real issue here.

> Yes, without reordering you get more 'reliable' results. However,
> AFAICS nsolve detects a problem in all of the cases: if py2.7 is
> sorted first, it detects ordering issue; in all other cases, it
> detects circular dep. Am I missing something?

Again, where is the code to check those 'all other cases' ?
This is a real question as you don't seem to get the combinatorial
explosion that happens here. I fear that unless you experiment it first
hands you'll still be doubting it.

Also, it seems very wrong that PM will now happily toggle flags from USE
but will completely fail if I configure it with "please prefer python3
over python2".

Alexis.