[gentoo-dev] [RFC] Forced/automatic USE flag constraints (codename: ENFORCED_USE)

[Michał Górny]

Hello,

For a long time we seem to be missing appropriate tools to handle USE
flag constraints efficiently. EAPI 4 brought REQUIRED_USE but all things
considered, it has proven to be far from an optimal solution. I would
therefore like to discuss adding a better tool to amend or replace it,
to allow for automated handling of USE flag constraints.


1. Motivation
=============

[[This section is purely informational. It is not up for discussion. If
you really can't help yourself and need to discuss superiority of
example A over example B, then please have mercy and wait for a few days
to give developers a chance to stay focused on the proposal at hand.]]

EAPI 4 provides REQUIRED_USE as a simple tool to enforce USE flag
constraints for packages. It can be used to express dependencies between
flags (e.g. A requires B). However, it poses a few problems which
results in developers being reluctant to use them and which limit their
use cases. Most notably:

1. They require an explicit action from the user. A large number of USE
flag constraints can be frustrating, especially when most of them are
trivial to solve.

2. They can cause a significant clutter in package.use. As a result,
changing past decisions becomes hard as the user may need to find
past USE flag changes and revert them.

3. The constraint output is sometimes considered hard to comprehend.

Those issues have resulted in developers sometimes avoiding REQUIRED_USE
in favor of other solutions such as implicit behavior (ignoring USE
flags), additional control flags (think PYTHON_SINGLE_TARGET) or
pkg_pretend() (which is horribly slow).

I think that the problems are most visible when dealing with 'provider
flags', that is USE flags that are used to choose between different
libraries providing the same feature. There are three basic approaches
to the problem:

A. Binary provider flags (+ feature flags), e.g.:

  USE='[ssl] libressl'  -> use LibreSSL
  USE='[ssl] -libressl' -> use OpenSSL
  USE='-ssl'            -> disable SSL/TLS (USE=libressl ignored)

B. Unary provider flags with REQUIRED_USE:

  USE='libressl -openssl'  -> use LibreSSL
  USE='-libressl openssl'  -> use OpenSSL
  USE='libressl openssl'   -> invalid
  USE='-libressl -openssl' -> disable SSL/TLS or invalid

C. Unary provider flags without REQUIRED_USE:

  USE='libressl -openssl'  -> use LibreSSL
  USE='-libressl openssl'  -> use OpenSSL
  USE='libressl openssl'   -> use LibreSSL or OpenSSL, depending on pkg
  USE='-libressl -openssl' -> disable SSL/TLS or as above

Sometimes solutions B and C are amended with additional feature flags to
reduce the confusion.

Each of these solutions has its disadvantages. What's worse, they are
sometimes not even used consistently within a single ecosystem -- going
as far as using the same USE flag with slightly different meaning.
The Gentoo developer attitude 'my package, my way' does not help here.

Solving all of the listed problems is not possible with the existing
tools as each of the possible solutions has its own issues, and it is
impossible to get developers to agree on one of them.


2. Specification
================

2.1. Forced USE flag constraints
================================

This proposal introduces USE flag constraints that are meant to be
solved automatically by the package manager. This is somewhat similar to
REQUIRED_USE, except that it does not require the user to take any
explicit action. In that regard, it is closer to conditional
package.use.force/mask.

In the basic form, it can be used to conditionally force a specific flag
to be enabled or disabled. For example:

  foo? ( bar )

would mean that the bar flag is implicitly enabled (forced) if foo is
enabled as well. Appropriately:

  foo? ( !bar )

would mean that the bar flag is implicitly disabled (masked) in that
case.

It can also be used with multi-flag ??, ^^ and || constraints, i.e.:

- ?? means that at most one of the flags can be enabled. If user
configuration causes more than one of the flags to be enabled,
additional flags are implicitly disabled (masked) to satisfy
the constraint.

- || means that at least one of the flags must be enabled. If user
configuration causes none of the flags to be enabled, one of them is
enabled implicitly (forced).

- ^^ means that exactly one of the flags must be enabled. The behavior
is a combination of both above constraints.

The automated solving of USE constraints would require the developers to
consider the implicit effect of the constraints they are writing.


2.2. UI implications
====================

The automated flag handling would require an appropriate UI changes, to
clearly indicate whenever the flags are being altered automatically.
Otherwise, users might be confused of their requested USE flags not
being applied correctly.

In the most basic form, a solution similar to use.force/use.mask can be
used, e.g.:

  USE="[bar] [-baz] foo"

with square brackets indicating that the flags have been forced/masked
by other USE flags on the list. It would also be necessary for
the package manager to provide more verbose information on USE flag
constraints being applied, either implicitly or on request.

For simple constraints, the visual presentation can be even included
in place, e.g.:

  USE="foo -> [bar -baz]"

indicating that both constraints were forced by the flag 'foo' (i.e.
that disabling 'foo' would loosen them).

For multi-flag constraints (??, ^^, ||), the package manager could even
go as far to group the flags together with explicit constraint notation,
e.g.:

  USE="^^ [ foo -bar -baz ]"

The exact UI design is left to the package manager developers. Those are
merely proof-of-concept ideas.


2.3. Configuration bits
=======================

In its most basic form, this proposal does not require explicit
configuration -- all flags are applied automatically. However, a few
bits might be useful.

Firstly, the package manager might provide a way to explicitly forbid
switching specific flags or automatically applying the constraints
completely. If the user uses that, it will just fail in the same way as
REQUIRED_USE fails now.

Secondly, it might be reasonable to provide configurable priorities for
solving multi-flag constraints. For example, we could use rightmost-
preferred logic for package.use, e.g.:

  */*          PROVIDER_SSL: openssl gnutls
  dev-util/foo PROVIDER_SSL: polarssl

which would mean that for all packages, gnutls is preferred over openssl
(i.e. if ?? or ^^ applies, openssl will be disabled and gnutls will be
used), and polarssl is additionally preferred over everything else for
dev-util/foo.


2.4. Backwards compatibility
============================

Before choosing the exact way of implementing this, we need to answer
how far we want to provide backwards compatibility. In particular:

a. Do we need the REQUIRED_USE requiring explicit user selection? Or can
we rely completely on other solutions (automatic solving,
pkg_pretend())?

b. Would changing REQUIRED_USE behavior in place cause unintended side
effects? Are we going to accept those effects until the packages are
fixed? (e.g. selecting less preferred solution of constraint)

If we can agree on doing the change in place (retroactively), I think we
can start solving a few major issues. For example, I would be really
happy to kill PYTHON_SINGLE_TARGET flags in favor of constraints on
PYTHON_TARGETS being solved automatically. We could also look into
unifying provider flags into PROVIDER_SSL, PROVIDER_AV etc.

I should point out that strictly speaking, PMS does not define any
specific handling of REQUIRED_USE. It only defines that the ebuild must
not be used if the constraints are not met:

| If the package manager encounters a package version where REQUIRED_USE
| assertions are not met, it must treat this package version as if
it      
| was masked. No phase functions must be
called.                                                                 
                                 

In other words, it does not define whether the package manager should
request the user to explicitly solve the problem or whether it should be
solved automatically. However, from past discussions I recall that
the original intent of REQUIRED_USE was for machine processing, and so I
would consider it reasonable to start using it.


2.5. Ebuild interface
=====================

This section is most flexible as I don't really care how it's done,
as long as it's done. Depending on our preferences, and the answers to
questions in the preceding section, the options include:

1. changing behavior of REQUIRED_USE retroactively -- i.e. making
Portage start solving USE constraints automatically, and being able to
rely on that a few months from now,

2. changing behavior of REQUIRED_USE in a future EAPI,

3. adding ENFORCED_USE with the new behavior in a future EAPI
(and either keeping or removing REQUIRED_USE),

4. adding special syntax to REQUIRED_USE to indicate which constraints
can be solved automatically,

5. adding special syntax to IUSE to indicate which flags can be enabled
or disabled automatically via REQUIRED_USE (a little different from
the exact proposal),

6. adding a dedicated variable to indicate USE_EXPAND sets that can be
solved automatically via REQUIRED_USE (partial solution).


3. Rationale
============

I believe this solution to be quite optimal. It solves most of
the issues with the REQUIRED_USE, making it a feasible solution to many
of the existing problems, including those that were not even worth
considering with the current REQUIRED_USE semantics.

Reusing the existing syntax has the advantage of lowering the learning
curve and simplifying the specification. We can even enable the new
semantics without changing the PMS (provided that variant 1. is chosen),
or with as little duplication as possible. However, the developers will
be required to start considering the implicit effects of their
constraints (as a matter of policy).

Automatically solving USE constraints solve all three fore-mentioned
issues with REQUIRED_USE. By default, no user intervention is required
to solve USE constraints and package.use needs to be modified only to
enforce a non-standard solutons. While the 'readability of output' is
not really changed per se, most of the time users wouldn't have to read
it.

As pointed out throughout the text, automatically solved USE constraints
can be used to replace existing hacks for different kinds of problems:

A. Packages supporting selecting a subset of possible targets, e.g.
Python implementations. Currently we have packages supporting 'any
number of implementations', 'one implementation only' and 'one python2*
+ one python3*'. For user convenience, we hacked the second into
separate PYTHON_SINGLE_TARGET flags which encumbered the implementation
seriously, and we discouraged the third case completely. With
automatically solved USE constraints, we'd just use PYTHON_TARGETS for
everything, and let the constraints handle invalid combinations, e.g.:

  PYTHON_TARGETS="python2_7 python3_5 {-python3_4} {-python3_6} ..."

B. Provider flags. With automatically solved constraints, we can easily
move them all into unary USE_EXPAND sets with USE constraints, and let
the PM handle selecting the one most preferred automatically, e.g.:

  USE="[ssl]" PROVIDER_SSL="gnutls {-openssl} {-libressl} ..."

What's important, the automatic flags have the advantage of being both
clearer in meaning than implicit flags, and more convenient than
REQUIRED_USE. In other words, the user clearly sees which implementation
 (provider) is used, and can change that in a relatively easy way.

Furthermore, the explicit visibility of selection reduces the number of
USE flag combinations in binary packages. That is, we can easily reduce
the number of different apparent sets of USE flags that build the same
package, making it easier for binary packages to match.


4. Comments
===========

What do you think?


-- 
Best regards,
Michał Górny