Re: [gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Alexis Ballier]

On Wed, 31 May 2017 15:04:52 +0200
Michał Górny <mgorny@g.o> wrote:

> On śro, 2017-05-31 at 10:38 +0200, Alexis Ballier wrote:
> > > > What if I specifically set USE=-bar in make.conf ? Do we really
> > > > want PM to override that without telling me ?
> > >
> > > Yes. Unless you specifically and explicitly disable that
> > > (globally or for USE=bar), in which case the PM would just reject
> > > to proceed.
> >
> >
> > Then could you please explain how to get the list of useflags the
> > solver is allowed to toggle ? Preferably in a PMS-friendly way (aka
> > no USE=foo emerge). It's not clear to me what this would be and is
> > mandatory for determinism.
> >
> > Note that most definitions are acceptable, but there must be one.
>
> If we *really* want to set this for the users, it would simply be
> a variable defined in make.profile, e.g.:
>
>   REQUIRED_USE_STRICT="foo bar"
>
> which would mean the solver is globally forbidden from touching those
> flags, i.e. if the solution would involve touching them, PM must fail
> and request user to manually resolve it.
>
> However, as far as I'm concerned we'd be good at keeping this purely
> as user configuration, alike FEATURES=i-do-not-want-automatic-solving.

Ok, why not.

> > > > I believe that, from the ebuild POV, the ternary useflag model
> > > > is more appropriate: You have a whole bunch of ways to specify
> > > > useflags with portage (make.conf, package.use, profiles, command
> > > > line, ...). From the ebuild those are collapsed into 'user
> > > > input'. You only have IUSE (with its defaults) and that's what
> > > > the auto-solver should play with: those are the flags that can
> > > > be toggled.
> > >
> > > I see your point. However, it's merely a preference problem and we
> > > really don't want the constraints to become ternary and/or PM try
> > > to force the reverse solutions. That's an easy way to lose
> > > predictability.
> >
> > They're not ternary anymore after processing ebuild: IUSE="foo +bar"
> > means instantiate foo as -foo if not specified, and bar as +bar.
> > The way I see it, ternary model is useful in the sense that the 3rd
> > undefined state is what the solver can toggle, the others are fixed
> > (either by user input or e.g. use.mask).
> >
> > Basically, I see automatic solving of REQUIRED_USE as dynamic IUSE
> > defaults. But see above, this might not be the best way.
>
> I'm lost on what you're trying to achieve here. Maybe give a full run-
> out based on Portage behavior -- i.e. involving all the places USE
> flags can be altered in Portage, and how they're going to affect the
> result. Don't forget about USE_ORDER.

What I'm suggesting is: Flags that can be toggled are the same that
would be affected by IUSE defaults. Others are fixed and the
REQUIRED_USE formula is instantiated & reduced with those values. If
there is a contradiction already, fail hard. If not, apply the
algorithm to determine a set of IUSE defaults that would make it
work. Process the ebuild as if it had those IUSE default.

You seem to be going in another direction which is unclear to me.

[...]
> > > > > Now, this also means that every constraint that can't be
> > > > > solved in this easy fashion is invalid. We want to detect
> > > > > that, and warn the developer. Some of those could be tricky.
> > > > > Simple example:
> > > > >
> > > > >   foo? ( baz ) bar? ( !baz )
> > > > >
> > > > > This one is invalid because USE='foo bar' requires both 'baz'
> > > > > and '!baz' as a solution. Remember that we don't want to do
> > > > > any changes besides what's explicitly written there, no
> > > > > guessing.
> > > >
> > > > Besides that, what makes it invalid ?
> > >
> > > What makes it invalid is that you can't solve it in a predictable
> > > way.
> >
> > You can fail in a predictable way and ebuild writer can adjust it to
> > avoid that.
>
> If the point is to process constraints *automatically*, then failing
> is not the desired result. Yes, we can consider that a minor
> issue/warning level but it is still an issue. I named it 'invalid'
> because it prevents the automatic solving from working which is the
> purpose of this whole effort.

Ok. I was assuming we do not want to change anything user-specified.
When I set USE=foo, I want foo, not maybe foo. But why not: As you
noted this can be a PM feature and there's not much to be checked in
that case.

As for how to check that, it is still completely unclear to me if
there'd be anything better than enumerating all the possible inputs.


> >  Again, you *need* to process the constraints in order. '!a?
> > ( b ) !b? ( a )' is not deterministic when none of a and b are
> > enabled otherwise.
>
> You can't rely on any particular order of constraints, especially when
> eclass stacking comes into play. You could try defining some
> constraint- sorting algorithm but it's going to be complex and it's
> usefulness will be limited anyway due to various kinds of grouping.


Better have some order: If half of the above constraint comes from
ebuild and the other half comes from eclass, then PM might toggle 'a' or
'b' depending on the phase of the moon which is specifically what we're
trying to avoid.

eclass stacking is not a problem: specify if it's append or prepend and
be done.

Note that if you want to remove the need for an order, you'll need to
ensure that all orderings of the constraints give the same result. It's
not sufficient to "only" check all possible inputs.

Also, what happens if we applied all the constraints and obtained some
useflags setting that still fails REQUIRED_USE check ?


> > You'll get a message like:
> > "
> >  The constraint bar? ( !baz )' is violated.
> >     bar is enabled because $reason (say, make.conf)
> >     baz is enabled because of the constraint 'foo? ( baz )'
> > 			      foo is enabled because $reason
> > "
>
> You = user or ebuild author? Because per my proposal, this construct
> should result in QA error/warning, telling the ebuild writer to use
> predictable constraints.
>
> Of course, we could still accept the ebuild and just fail hard on user
> side (alike REQUIRED_USE). But that's really getting out of scope.

you = user because he asked for an invalid combination. Note that you
have not yet given a proper definition of predictable constraint.


[...]
> >
> > > which gives a single predictable solution:
> >
> > Then ebuild writer should not write 'foo? ( baz ) bar? ( !baz )' but
> > rather: 'foo? ( !bar baz ) bar? ( !baz )', which should cover more
> > cases. With USE="foo bar", the message would then be:
> > "
> >   The constraint 'foo? ( !bar )' is violated.
> >       foo is enabled because $reason
> >       bar is enabled because $reason
> > "
> > which is similar to the above except there's one less step for
> > explaining the reasons. It's not dramatic but it is, indeed,
> > desirable to have simple & clear reasons. I'd say that's more to
> > the argument for specifying completely how to solve that and leave
> > those small improvements to ebuild writers and/or QA.
>
> Is this message meant to be disabled for the purpose of explaining
> automatic decisions, or as an error?

As an error because user specified invalid combination, but now I see
your point: You want to override user settings too.

> Also, your example (unlike the one I gave in the original mail)
> prefers foo over bar. That's the only difference.

Yes, I realize that now :)

> >
> > [...]
> > > > > However, the
> > > > > following should be valid:
> > > > >
> > > > >   foo? ( baz ) bar? ( !foo !baz )
> > > > >
> > > > > Because now we clearly indicate that USE=bar disables USE=foo,
> > > > > and therefore makes the first constraint inapplicable. It
> > > > > clearly indicates course of action for all combinations:
> > > >
> > > > Ok, I now think you're aiming for giving full power to the
> > > > solver, overriding user inputs if necessary. Before going
> > > > further, I think we should first agree on what are the useflags
> > > > such a solver can toggle. I'm not sure 'USE=foo emerge blah'
> > > > should disable foo instead of failing for example.
> > > >
> > >
> > > As I said, it's a matter of configuration to decide which flags
> > > should be touched, and which not. Of course if we find that
> > > necessary, we may go into defining a specific set in the profiles
> > > or metadata.
> > >
> > > However, I would rather focus on getting a PoC solver and checker
> > > first, and play with existing constraints to see how it all
> > > works.
> >
> > The solver seems on good tracks, at least from the algorithmic POV.
> > The checker, however, is not clear at all to me. The main reason is
> > that to determine if the solver will be able to solve it, it needs
> > to know what the solver can toggle and what not.
> >
>
> The point would be: by definition, the solver should be able to touch
> *any* USE flag. If it can't, it mostly implies that you can't use
> the automatic solver, and so the case is irrelevant for checking.
> Attempting to go beyond that is going to give a lot of complexity
> and most likely kill the whole idea.
>
> One thing we need to worry about are masks. We need to think about
> them.

It makes zero difference for any solver if you replace variables
(useflags) by 'true' or 'false' constants (masked/forced/user-forced
useflags). It's even simpler actually. Whether the formula is not
constantly 'true' (ie REQUIRED_USE is useless) or constantly
'false' (ie there's no way to solve it) is equivalent to solving SAT.
We likely don't want that for repoman running on php.

Alexis.