Re: [gentoo-dev] glep 42 (news) round six

[Brian Harring]

On Sun, Dec 18, 2005 at 04:15:45AM +0000, Ciaran McCreesh wrote:
> Here we go again... Changes:
>
> * We're now supporting overlays, multiple repositories and magic flying
> chickens. To do this, we're shoving a whole load of new requirements
> onto Portage. Said requirements are documented under `Required Portage
> Enhancements`_. I now expect to get heavily flamed by a different set
> of Portage developers.

You forgot the magic mushroom badgers and snake.


> * Change to yyyy-mm-dd for GLEP 45 compatibility. Rereremove the
> timestamp on the Posted: header.
>
> * Tweak Display-If-{Profile,Installed} to work with multiple
> repositories.
>
> * Use ${repoid} rather than magic-chicken for news-*.* files.

Drop the magic-chicken crap (especially in light of your comments 
about 'professional' news inline in the glep).

Killjoy maybe, but it detracts from the point of the glep.

> * Be specific on how news-repoid.skip can work.

Still haven't gotten into specifics, merely a different bit of hand 
waving.


> You are encouraged to reply to this thread saying "I agree with ciaranm
> that repository IDs should not be allowed to contain spaces".
>
> TODO: ferringb wants spaces added to the first item on the list. I don't,
> because it makes repo id -> filename mappings nasty.
>
> * Every repository (including overlays) will require a unique identifier. It is
>   assumed that an identifier will be a string consisting of characters from
>   ``a`` to ``z``, ``A`` to ``Z``, ``0`` to ``9``, ``+`` (plus), ``-`` (hyphen),
>   ``:`` (colon) and ``_`` (underscore).

Not really.  Just requires your code to be space safe.

You write good code, right?  Especially since you need to keep our 
path handling safe for osx (/volumes) and for users who do strange 
things?

The news handling crap should be able to take spaces- remember the 
comments about user aliasing of repostory ID's down the line.  I don't 
care if the actual metadata/repo_id lacks spaces, but the handling 
code must be able to take spaces, as such the requirement that spaces 
not be used is arbitrary and uneeded.


> * Portage must provide a way for external programs to obtain a list of all
>   repository identifiers for a given system. It is assumed that this will be in
>   the form of a ``portageq`` command (e.g. ``portageq get_repo_ids``).
>
> * Portage must provide a way for external programs to obtain the base path for
>   a repository with a given ID. It is assumed that this will be in the form of
>   a ``portageq`` command (e.g. ``portageq get_repo_root gentoo-x86``).
>
> * Portage must extend ``portageq has_version`` to support restrictions to a
>   given repository ID.
>
> * Portage must extend ``portageq`` to implement a command which returns whether
>   or not the profile used for a given repository ID matches a certain base path
>   (e.g. ``portageq profile_used default-linux/sparc/sparc64/2004.3 gentoo-x86``).

profile_in_use, maybe.


> These extensions are assumed during the following specification.
>
> News Item Identities
> --------------------
>
> Each news item will have a unique identifier. This identifier will be in the
> form ``yyyy-mm-dd-short-name``, where ``yyyy`` is the year (e.g. ``2005``),
> ``mm`` is the month (``01`` through ``12``) and dd is the day of the month
> (``01`` through ``31``). The ``short-name`` is a very short name describing the
> news item (e.g. ``yoursql-updates``), consisting only of the characters ``a-z``,
> ``0-9``, ``+`` (plus), ``:`` (colon), ``-`` (hyphen) and ``_`` (underscore).
>
> News Item Directories
> ---------------------
>
> Each news item will be represented by a directory whose name is the same as the
> news item's identifier.
>
> The directory will contain a file named ``yyyy-mm-dd-short-name.en.txt``, which
> contains the text of the news item, in English, in the format described below.
>
> If a news item is translated, other files named ``yyyy-mm-dd-short-name.xx.txt``
> (where ``xx`` is the ISO 639 [#iso-639]_ two letter country code) will also be
> provided. However, only the English version of a news item is authoritative.
> This anglocentricity is justified by precedent [#glep-34]_.
>
> News Item Files
> ---------------
>
> A news item file is a text file, encoded using UTF-8 [#rfc-3629]_ for
> compatibility with and for the same reasons as existing Gentoo documentation
> [#docs-policy]_ and the tree [#glep-31]_.
>
> News items should be signed with a detached GPG signature: ::

why should vs must?
Should leaves open the possibility for a tree compromise and a news 
item with Very Bad(tm) instructions stuck into it.

Moving towards getting the whole tree signed, introducing new 
components that aren't required signed works against that.

> News Item Headers
> '''''''''''''''''
>
> The following headers describe the purpose and format of the news item:
>
> ``Title:``
>     A short (maximum 44 characters) descriptive title. Mandatory.
>
> ``Author:``
>     Author's name and email address, in the form ``Real Name <email@address>``.
>     Mandatory; multiple author headers may be specified if appropriate.
>
> ``Translator:``
>     For translated news items, the translator's name and email address. Multiple
>     translator headers may be specified if appropriate.
>
> ``Content-Type:``
>     Must be ``text/plain``. Mandatory.
>
> ``Posted:``
>     Date of posting, in ``yyyy-mm-dd`` format (e.g. 2005-12-18) for
>     compatibility with GLEP 45 [#glep-45]_. Mandatory.
>
> ``Revision:``
>     Initially 1. Incremented every time a non-trivial change is made.  Changes
>     which require a re-read of the news item should instead use a new news item
>     file. Mandatory.
non-trivial changes that don't require a re-read sounds like a 
contradiction.  Clarify, especially since portage will mark this as 
read _once_ and only once.

> The following headers are used for filtering:
>
> ``Display-If-Installed:``
>     A dependency atom or simple package name (for example,

Drop the 'simple package name'; it still is an atom.

>     ``<dev-lang/php-5_alpha`` or ``net-www/apache``). If the user has the
>     package specified installed from the repository from which the news item was
>     obtained, the news item should be displayed.

This isn't incredibly useful if ranged versions are ever introduced.  
Ammending the glep for that seems stupid, looser language might be 
wise.

> ``Display-If-Keyword:``
>     A keyword [#glep-22]_ name, for example ``mips`` or ``x86-fbsd``. If the
>     user is on the keyword in question, the news item should be displayed.
>
> ``Display-If-Profile:``
>     A profile path, for example ``default-linux/sparc/sparc64/server``. If the
>     user is using the exact profile in question, or a subprofile of this
>     profile, the news item should be displayed. This header may be used to
>     replace ``deprecated`` files in the future.
>
> .. Note:: When performing package moves, developers must also update any

>    relevant ``Display-If-Installed`` headers in news files.
Grounds for someone to get off their ass and do some work, but this 
should be automated in some fashion- specifically detection of it 
(auto-updating won't work with signing).

> The algorithm used to determine whether a news item is 'relevant' is as
> follows:
>
> * For each ``Display-If-`` header type which occurs at least once:
>
>   + The news item is not relevant if none of the headers of this type are
>     successfully matched.
>
> * Otherwise the news item is relevant.
>
> In particular, if no ``Display-If-`` header is specified, a news item will be
> relevant for all users.

implementation issue= be aware this must be cached.  No caching == 
slowing down portage pissing off users.

All news items *will* need to be cached in some format here- just 
because the keyword isn't what the user is running at the time of 
sync, doesn't mean that the keyword won't be used for ROOT!="/" cross 
compilation.

That's an implementation note, but I'm bringing it up since the time 
to do the filter/cache instantiation is during portage initialization 
(yes it sucks, but your stuck with stable)... so start thinking about 
ways to make it fast, since python -c'import portage' is the slowest 
part of portage queries


> News Item Quality Control
> -------------------------
>
> There have been complaints regarding the comprehensibility of some upgrade
> notices and news items in the past. This is understandable ??? not every Gentoo

'???' ?

> .. Note:: A previous draft of this GLEP allowed news items to be sent to
>    ``gentoo-core`` instead of ``gentoo-dev``. It is possible that a situation
>    may arise where this will be necessary (for example, a security update which
>    must break backwards compatibility and which cannot be revealed to the public
>    before a given date).

Drop that and just state that -core doesn't fly sans security 
consideration; referencing one of the previous 5 versions isn't needed 
(yes it's minor, but it's uneeded info).

> Server Side
> '''''''''''
>
> News items are to be made available via the standard rsync tree. This removes
> any need for polling of a remote source.
>
> A new repository will be created for news items. The type (CVS or Subversion),
> location and access controls on this repository are beyond the scope of this
> GLEP.

We generate a tree every 30 minutes.  You aiming for every update, or 
for less frequent (once an hour fex)?

Matters due to the fact rsync tree generation has a window it must not 
overrun- minor but continuing the "lets be explicit" goal of yours.


> This repository will contain directories named ``yyyy/mm/``, where ``yyyy`` is
> the current year and ``mm`` is the current month number (01 for January through
> 12 for December). This separation will help keep news items more manageable.
>
> The contents of this repository will automatically be merged with the main rsync
> tree, placing the items in a ``metadata/news/`` directory. The method used for
> merging these items is beyond the scope of this GLEP ??? a similar setup is
> already used for merging GLSAs into the rsync tree.
>
> The main rsync tree will **not** use the ``yyyy/mm/`` subdirectory layout.

What shall it use?  Again, be explicit.


> Client Side
> '''''''''''
>
> Whenever relevant unread news items are found, the package manager will create a
> file named ``/var/lib/gentoo/news/news-repoid.unread`` (if it does not
-file named ``/var/lib/gentoo/news/news-repoid.unread`` (if it does not
+file named ``/var/lib/gentoo/news/news-${repoid}.unread`` (if it does not

Clearer at a first read though imo.

> already exist) and append the news item identifier (eg
> ``2005-11-01-yoursql-updates``) on a new line.

Implementation details, but this handling will need to be integrated 
into portage (eg, no external scripts).  External scripts will slow 
things down (shell overhead/fork/exec), plus portage is going to have 
to query this cruft already for the notifications on emerge targets 
(no we're not execing everytime for that either ;)

> Checks for new news messages should be displayed:
>
> * After an ``emerge sync``
> * After an ``emerge --pretend``
> * Before an ``emerge <target>`` (which may also include a red warning message)
> * Before an ``emerge --ask <target>`` sequence

ask is superfluous, nuke it.

You haven't stated how the 'package manager' will trigger the user's 
reader of choice for these targets.  Should also extend this to allow 
a way to disable any news notices, lest someone's cronjob get hung 
displaying news (feature or not, it's needed).


> The package manager must keep track of news items that have already been added
> to the unread list to avoid repeatedly marking a deleted news item. This could
> be handled via a ``news-repoid.skip`` file containing the IDs of news items that
> have already been added to a ``news-repoid.unread`` file, but this method is not
> required by this GLEP.

Specifics.


> Once a news item is marked for reading, third party tools (or traditional core
> Unix tools) can be used to display and view the news files.
>
> When a news item is read, its name should be removed from the
> ``news-repoid.unread`` file. If a news client acts as an interactive reader
> rather than a gateway, it should then add the name to a ``news-repoid.read``
> file in the same directory with the same file format.

implementation issue; you need locking on this.  If portage has to 
farm out to the users reader of choice, then it needs to lock the file 
to keep readers/writers from screwing with each other.

Flock preferably, since it's faster then resorting to hardlink 
trickery (yes this may be harder for shell crap, but so it goes since 
hardlink locking sucks).


> News Item Removal
> -----------------
>
> News items can be removed (by removing the news file from the main tree) when
> they are no longer relevant, if they are made obsolete by a future news item or
> after a long period of time. This is the same as the method used for ``updates``
> entries.

implementation issue; updating unread/skip crap so it doesn't bloat is 
required, but time frame also matters (crap sync resulting in news 
getting removed, yet it still being valid, just missing from the local 
tree).

> Integration with Existing Systems
> =================================
>
> It would be simple to convert these news items into the format used for news
> items on the Gentoo website or posts for the ``gentoo-announce`` mailing list.
>
> There is an existing automated tool [#forums-glsa]_ for posting GLSAs to the
> forums. A similar tool can be used for these news items.

Pawned it off on someone, or something you'll be doing?

~harring