[gentoo-amd64] Re: wanted: script to build ebuilds

[Duncan]

[GRR!!  Insane quote ordering for proper contextual replies rearranged to
conventional.]

Dice R. Random posted
<d9b9989b0604280243h2f97255dt285379a59da555ef@××××××××××.com>, excerpted
below,  on Fri, 28 Apr 2006 02:43:44 -0700:
>
> On 4/28/06, Steve Herber <herber@×××××.com> wrote:
>> Portage is great but... it depends upon a person to create a new ebuild
>> when a new version is released.
>>
>> There are two times when I would like a script to create an ebuild
>> automatically.  I wonder if anyone has such a script or if someone is
>> working on such a script.
>>
>> Often a new software release is available from the home site but
>> portage has not been updated.  I would like a script that would:
>>
>>         check the source repository for a package
>>         find the latest version
>>         if the repository has a newer version
>>                 create an overlay with an ebuild for the new version
>>                 try to build the new package

This would be relatively easy to automate, due to the standard bump
procedure below.  It wouldn't always work, and the risk for even trying it
would be entirely the user's, because in some cases such a procedure WOULD
break the system, due to changes between versions that weren't
accounted for by the automated process, but it would be relatively easy to
write a script that would work for most minor bumps anyway, a reasonable
percentage of the time, risky tho it would indeed be to just blindly run
it.

>> Other times I will hear about a new software package and I want to try
>> it out.  I would like a script that would take a URL for the home site
>> as an argument and then would operate like the script above.

This simply couldn't be automated in a sane manner.  There's too much
variability in the way various things build, and too much risk in simply
assuming the normal otherwise automatable procedure will work, and will
not do something strange enough that it could be harmful to a working
system.  An initial build for an application isn't a simple matter.  One
must decide where things should go and supply the appropriate parameters
to have it install there.  One must check that the licensing is suitable
for Gentoo, and further, whether Gentoo can mirror the sources or whether
they have to be fetched directly from upstream due to licencing, possibly
manually, if the licensing is strict enough.  One must verify the
setup/unpack/configure/compile/install/qmerge steps and create
non-defaults for anything that doesn't work normally.  One must ensure
that the sequence doesn't violate the non-interactive rules of portage. 
One must ensure the merged application is sufficiently configured to "just
work" out of the box, with no additional configuration, if at all possible
(obviously it's not going to be possible with things like cups, where the
user must configure the printers they use as appropriate).  One must make
some attempt to verify sane compile settings and that the package will use
those supplied by portage over its default settings.  One must further
verify that "normal" CFLAGS/CXXFLAGS actually work.  One must evaluate
dependencies both optional and required, setting the correct build and
runtime dependencies, and setting up USE flags as appropriate for the
optional ones.  One must... One must... One must... Finally, appropriate
patches must be devised if the package doesn't by default pass any of
these steps, such that it does.

Really, the devs do more for you by supplying Gentoo than picking their
nose all day.  Don't you think if it were that easy (or even /possible/)
to automate creating an ebuild for a any random product out there, that
they would have such a system by now?  I'm not a dev, but if I was, I'd be
insulted by just how lazy or stupid you obviously think they are, lazy if
you think they have such an approach and are simply hiding it as some big
secret, so they can do the easy stuff and make it look hard, stupid if you
think it's possible to automate the thing, and they haven't done it.

Really, if there's no ebuild available at all, the user is best advised to
read the documentation to learn of any gotchas, then follow thru on the
steps manually, such that he can watch what's going on and see where any
breakage occurs, and attempt to fix it at that point.

>> If such scripts were available, it would be much easier to check out
>> new software releases.  I could imagine a system that would scan
>> listings from freshmeat or other non-Gentoo site, build the ebuild and
>> try the package.  In the simplest case you would get a go/no-go answer.
>> For the packages that download and build, you could try it out.  If it
>> failed you could ignore it or try to figure out the problem.

If it failed, you could indeed try to figure out the problem --
potentially try to figure out why nothing you try to run works any more,
and your system won't boot, because the automated script created an ebuild
that blindly deleted /usr/bin/*, or something equally dangerous to a
system, because someone thought it was a bright idea to trust the build to
an automated script, sight entirely unseen, README and INSTALL entirely
unread!

There's simply no sane nor safe way to automate that stuff.  For something
with an existing ebuild that might just need bumped, sure.  For something
without an existing ebuild, I must conclude you really hadn't thought the
implications of what you were asking for thru AT ALL!!

> I don't know about an automated process, there are a lot of variables
> that can factor into a version bump.  However, for many simple
> packages, i.e. things that are not heavily patched or that do not have
> changing software requirements, it is usually sufficient to simply
> copy the old ebuild over to the new version name.
>
> Example, suppose net-www/fooapp has recently been bumped from version
> 1.001 to 1.002.  You could do the following:
>
> mkdir -p /usr/local/portage/net-www
> cp -r /usr/portage/net-www/fooapp /usr/local/portage/net-www
> cd /usr/local/portage/net-www/fooapp
> cp fooapp-1.001.ebuild fooapp-1.002.ebuild
> ebuild fooapp-1.002.ebuild digest
> emerge -auv fooapp
>
> The reason for copying the entire /usr/portage/net-www/fooapp
> directory tree over is that there may be some patches or configuration
> files or such under the files/ subdirectory which are required by the
> ebuild.  As I said, this procedure works a good 8 or 9 times out of 10
> with simple ebuilds, sometimes some slight modifications are necessary
> and other times the modifications may be quite extensive.

A solid idea, the steps of which are done by many, but a couple
modifications make it a bit better.

Instead of copying over the entire dir, create the dir in the overlay (if
you don't already have it from a previous version), and copy over the
ebuild you intend to bump.  You don't have to worry about previous
versions, or the changelog, metadata, or manifest files, since you the
former two are simply informational and the latter will be recreated. 
Instead of copying over the files dir, make a symlink to the existing one
in the regular portage tree.  (If you have any of your own patches or
modify any existing ones, create a different dir for those, and edit the
ebuild accordingly.)

Since the files dir is a symlink, it will automatically change as the
portage tree changes.  Thus, if patches are changed or added in the Gentoo
tree, the copy in the overlay will stay current, and any remerge or the
next bump will use the new versions.  In some cases that's critical, as it
could mean you get security patches added since the static creation of the
overlay.

Doing it this way, only copying the specific ebuild you'll be working on,
bumping or whatever, and using a symlink for the files dir, avoids the
risk of overwriting any work you've done on the same package previously,
that you may wish to save.  It's also more space efficient.

...

In any case, with or without my changes, that the steps can be laid out as
a series of coommands like that indicates they can be made into a script. 
It's an exercise left to the reader, then, to create a script that takes
the CPV (category/package-version) of the existing ebuild, and the version
of the the target ebuild, as command line parameters, and executes the
sequence of coommands necessary to end up with the bumped ebuild ready to
go in the overlay.  Depending on how fancy you want to make the script,
with or without a --help output, with or without validity checking and
handling any possible errors, with or without the overlay location
hard-coded into the script vs. configurable using a config file or a
command line option, and depending on how good you are with bash scripting
or scripting in your language of choice (perl, python, whatever), reading
this thread could have taken you longer than creating the script will.

> BTW, if you do discover a package that has a new version and you get a
> good working ebuild for the new version make sure that you post it to
> bugzilla so that a developer can get it into the tree.

Always a good recommendation! =8^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


-- 
gentoo-amd64@g.o mailing list