Re: [gentoo-amd64] Re: wanted: script to build ebuilds

[Steve Herber]

People have responded in thoughtful ways but it seems to me that I may
not have been clear in explaining my intent.  I will respond in-line
below to this particular post because it included so much material.

I want to thank everyone who has replied to my request.  The comments
have helped me visualize some simple scripts that I can write to start
testing.

I think that I can safely conclude that the sort of script I envision
does not exist yet.

Cheers!

Steve Herber	herber@×××××.com		work: 206-221-7262
Security Engineer, UW Medicine, IT Services	home: 425-454-2399

On Fri, 28 Apr 2006, Duncan wrote:

> [GRR!!  Insane quote ordering for proper contextual replies rearranged to
> conventional.]
>
> Dice R. Random posted
> <d9b9989b0604280243h2f97255dt285379a59da555ef@××××××××××.com>, excerpted
> below,  on Fri, 28 Apr 2006 02:43:44 -0700:
>>
>> On 4/28/06, Steve Herber <herber@×××××.com> wrote:
>>> Portage is great but... it depends upon a person to create a new ebuild
>>> when a new version is released.
>>>
>>> There are two times when I would like a script to create an ebuild
>>> automatically.  I wonder if anyone has such a script or if someone is
>>> working on such a script.
>>>
>>> Often a new software release is available from the home site but
>>> portage has not been updated.  I would like a script that would:
>>>
>>>         check the source repository for a package
>>>         find the latest version
>>>         if the repository has a newer version
>>>                 create an overlay with an ebuild for the new version
>>>                 try to build the new package
>
> This would be relatively easy to automate, due to the standard bump
> procedure below.  It wouldn't always work, and the risk for even trying it
> would be entirely the user's, because in some cases such a procedure WOULD
> break the system, due to changes between versions that weren't
> accounted for by the automated process, but it would be relatively easy to
> write a script that would work for most minor bumps anyway, a reasonable
> percentage of the time, risky tho it would indeed be to just blindly run
> it.

 	I assume that installing a package based on an automatically
 	created ebuild may break a system.  But how else can you
 	determine that a new software package release is good on not?
 	You need to try it.  I just want a script that lowers the bar
 	for the person that is willing to try.

>>> Other times I will hear about a new software package and I want to try
>>> it out.  I would like a script that would take a URL for the home site
>>> as an argument and then would operate like the script above.
>
> This simply couldn't be automated in a sane manner.  There's too much

 	Sane?  A crude script that can create an ebuild with just
 	the minimum fields may not work very often, but if it works
 	for the package YOU want, then you have just saved YOUR time.
 	If it partially works, then you are that much closer to testing
 	the package.  That extra time could then be used to improve the
 	ebuild and maybe get it working.  That seems pretty sane to me.

> variability in the way various things build, and too much risk in simply
> assuming the normal otherwise automatable procedure will work, and will
> not do something strange enough that it could be harmful to a working
> system.  An initial build for an application isn't a simple matter.  One

 	I disagree.  Many applications need very simple ebuilds.
 	The feature I love about portage is that if you do end up
 	putting files in less than optimum places, you can just remove
 	the package and update the ebuild to use better locations.

 	I wish /usr/portage were split into two parts.  One for the core
 	packages that you need when Gentoo is used as your operating
 	system and another set for applications.  I expect this issue
 	has been discussed by the portage on Mac OS X people but I have
 	not looked at their solution yet.  The reason I mention this is
 	that I would generally be interested in automatically creating
 	ebuilds for the application set and would tend to leave the core
 	OS side alone to avoid the sort of catastrophic events that you
 	point out could happen.

> must decide where things should go and supply the appropriate parameters
> to have it install there.  One must check that the licensing is suitable
> for Gentoo, and further, whether Gentoo can mirror the sources or whether

 	I am not suggesting Gentoo ever use an automated systems
 	to put new software into the official portage tree.  I would
 	always want a person to be involved for the many reasons you
 	mention here.  That does not stop us from having a system that
 	jump starts the ebuild creation process.

> they have to be fetched directly from upstream due to licencing, possibly
> manually, if the licensing is strict enough.  One must verify the
> setup/unpack/configure/compile/install/qmerge steps and create

 	The script I am looking for will in fact have to create the
 	overlay area and run through all the steps you mention.  The
 	nice thing is that emerge will do each step for you and it keeps
 	multiple logs, so we can see if the build works or not.  It is
 	nice to not have to rewrite emerge.

> non-defaults for anything that doesn't work normally.  One must ensure
> that the sequence doesn't violate the non-interactive rules of portage.
> One must ensure the merged application is sufficiently configured to "just
> work" out of the box, with no additional configuration, if at all possible

 	The script I want may well create an ebuild that will fully
 	succeed with emerge but then NOT just work.  Ebuilds in the
 	official Portage ebuilds need to "just work", but experimental
 	ebuilds don't need to fully work.

> (obviously it's not going to be possible with things like cups, where the
> user must configure the printers they use as appropriate).  One must make
> some attempt to verify sane compile settings and that the package will use
> those supplied by portage over its default settings.  One must further
> verify that "normal" CFLAGS/CXXFLAGS actually work.  One must evaluate
> dependencies both optional and required, setting the correct build and
> runtime dependencies, and setting up USE flags as appropriate for the
> optional ones.  One must... One must... One must... Finally, appropriate
> patches must be devised if the package doesn't by default pass any of
> these steps, such that it does.
>
> Really, the devs do more for you by supplying Gentoo than picking their
> nose all day.  Don't you think if it were that easy (or even /possible/)
> to automate creating an ebuild for a any random product out there, that
> they would have such a system by now?  I'm not a dev, but if I was, I'd be
> insulted by just how lazy or stupid you obviously think they are, lazy if
> you think they have such an approach and are simply hiding it as some big
> secret, so they can do the easy stuff and make it look hard, stupid if you
> think it's possible to automate the thing, and they haven't done it.

 	This is a bit harsh.  The devs do a great job.
 	I don't know why there is not a an automated system to create
 	ebuilds for random software, that is why I asked the list.
 	Maybe the developers are busy improving the core system.  That
 	is what it looks like to me.

 	But it is a closed system in the sense that there is not a
 	public web site where a person like you or I can go, point to an
 	interesting project, and have a first cut at an overlay created
 	within the limits of the script.  The web site could even try to
 	compile and install in a virtual machine and give you a report
 	before you ever tried it on your system.  I would also want the
 	web to have a way for people to register their success and
 	failure and have a place where you could upload a fixed ebuild.

 	Allowing non portage developers a chance to try their hand on a
 	new ebuild and get feedback on their efforts seems to be taking
 	advantage of the Open Source methods.
>
> Really, if there's no ebuild available at all, the user is best advised to
> read the documentation to learn of any gotchas, then follow thru on the
> steps manually, such that he can watch what's going on and see where any
> breakage occurs, and attempt to fix it at that point.

 	Or use a script that generates a skeleton that you can start
 	with.
>
>>> If such scripts were available, it would be much easier to check out
>>> new software releases.  I could imagine a system that would scan
>>> listings from freshmeat or other non-Gentoo site, build the ebuild and
>>> try the package.  In the simplest case you would get a go/no-go answer.
>>> For the packages that download and build, you could try it out.  If it
>>> failed you could ignore it or try to figure out the problem.
>
> If it failed, you could indeed try to figure out the problem --
> potentially try to figure out why nothing you try to run works any more,
> and your system won't boot, because the automated script created an ebuild
> that blindly deleted /usr/bin/*, or something equally dangerous to a
> system, because someone thought it was a bright idea to trust the build to
> an automated script, sight entirely unseen, README and INSTALL entirely
> unread!

 	This sounds like the sort of problem you would get if you tried
 	to update one of the core OS packages.  I am thinking more along
 	the lines of Asterisk.  The portage tree is generally out of
 	date.  I think there is an alternate overlay site that is much
 	more current but just bumping the version has a good chance of
 	working.

 	Or, I just saw a new package, http://www.getdemocracy.com/, and
 	I was interested in trying it.  It is a video player.  I don't
 	expect it to totally trash my system.  But if it did and I had a
 	web site where I could report on it, someone else might know how
 	to fix the ebuild.
>
> There's simply no sane nor safe way to automate that stuff.  For something
> with an existing ebuild that might just need bumped, sure.  For something
> without an existing ebuild, I must conclude you really hadn't thought the
> implications of what you were asking for thru AT ALL!!

 	This just seems harsh.

 	We can decode the human genome with a Perl script but we can't
 	figure out how to use a few heuristics and a sandbox to get
 	close to a working ebuild.  I think we can and I am looking for
 	existing scripts and ideas from people that will help us get to
 	the point that most open source packages have an ebuild.

 	I have thought about this a little bit.

>
>> I don't know about an automated process, there are a lot of variables
>> that can factor into a version bump.  However, for many simple
>> packages, i.e. things that are not heavily patched or that do not have
>> changing software requirements, it is usually sufficient to simply
>> copy the old ebuild over to the new version name.
>>
>> Example, suppose net-www/fooapp has recently been bumped from version
>> 1.001 to 1.002.  You could do the following:
>>
>> mkdir -p /usr/local/portage/net-www
>> cp -r /usr/portage/net-www/fooapp /usr/local/portage/net-www
>> cd /usr/local/portage/net-www/fooapp
>> cp fooapp-1.001.ebuild fooapp-1.002.ebuild
>> ebuild fooapp-1.002.ebuild digest
>> emerge -auv fooapp

 	Thanks for this suggestion.  I will create a shell script to
 	automate this part of the process and test it out.  Thanks.
>>
>> The reason for copying the entire /usr/portage/net-www/fooapp
>> directory tree over is that there may be some patches or configuration
>> files or such under the files/ subdirectory which are required by the
>> ebuild.  As I said, this procedure works a good 8 or 9 times out of 10
>> with simple ebuilds, sometimes some slight modifications are necessary
>> and other times the modifications may be quite extensive.
>
> A solid idea, the steps of which are done by many, but a couple
> modifications make it a bit better.
>
> Instead of copying over the entire dir, create the dir in the overlay (if
> you don't already have it from a previous version), and copy over the
> ebuild you intend to bump.  You don't have to worry about previous
> versions, or the changelog, metadata, or manifest files, since you the
> former two are simply informational and the latter will be recreated.
> Instead of copying over the files dir, make a symlink to the existing one
> in the regular portage tree.  (If you have any of your own patches or
> modify any existing ones, create a different dir for those, and edit the
> ebuild accordingly.)
>
> Since the files dir is a symlink, it will automatically change as the
> portage tree changes.  Thus, if patches are changed or added in the Gentoo
> tree, the copy in the overlay will stay current, and any remerge or the
> next bump will use the new versions.  In some cases that's critical, as it
> could mean you get security patches added since the static creation of the
> overlay.
>
> Doing it this way, only copying the specific ebuild you'll be working on,
> bumping or whatever, and using a symlink for the files dir, avoids the
> risk of overwriting any work you've done on the same package previously,
> that you may wish to save.  It's also more space efficient.

 	Another good suggestion.  Thanks.
>
> ...
>
> In any case, with or without my changes, that the steps can be laid out as
> a series of coommands like that indicates they can be made into a script.
> It's an exercise left to the reader, then, to create a script that takes
> the CPV (category/package-version) of the existing ebuild, and the version
> of the the target ebuild, as command line parameters, and executes the
> sequence of coommands necessary to end up with the bumped ebuild ready to
> go in the overlay.  Depending on how fancy you want to make the script,
> with or without a --help output, with or without validity checking and
> handling any possible errors, with or without the overlay location
> hard-coded into the script vs. configurable using a config file or a
> command line option, and depending on how good you are with bash scripting
> or scripting in your language of choice (perl, python, whatever), reading
> this thread could have taken you longer than creating the script will.
>
>> BTW, if you do discover a package that has a new version and you get a
>> good working ebuild for the new version make sure that you post it to
>> bugzilla so that a developer can get it into the tree.

 	Yep!
>
> Always a good recommendation! =8^)
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman in
> http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html
> gentoo-amd64@g.o mailing list
-- 
gentoo-amd64@g.o mailing list