1 |
People have responded in thoughtful ways but it seems to me that I may |
2 |
not have been clear in explaining my intent. I will respond in-line |
3 |
below to this particular post because it included so much material. |
4 |
|
5 |
I want to thank everyone who has replied to my request. The comments |
6 |
have helped me visualize some simple scripts that I can write to start |
7 |
testing. |
8 |
|
9 |
I think that I can safely conclude that the sort of script I envision |
10 |
does not exist yet. |
11 |
|
12 |
Cheers! |
13 |
|
14 |
Steve Herber herber@×××××.com work: 206-221-7262 |
15 |
Security Engineer, UW Medicine, IT Services home: 425-454-2399 |
16 |
|
17 |
On Fri, 28 Apr 2006, Duncan wrote: |
18 |
|
19 |
> [GRR!! Insane quote ordering for proper contextual replies rearranged to |
20 |
> conventional.] |
21 |
> |
22 |
> Dice R. Random posted |
23 |
> <d9b9989b0604280243h2f97255dt285379a59da555ef@××××××××××.com>, excerpted |
24 |
> below, on Fri, 28 Apr 2006 02:43:44 -0700: |
25 |
>> |
26 |
>> On 4/28/06, Steve Herber <herber@×××××.com> wrote: |
27 |
>>> Portage is great but... it depends upon a person to create a new ebuild |
28 |
>>> when a new version is released. |
29 |
>>> |
30 |
>>> There are two times when I would like a script to create an ebuild |
31 |
>>> automatically. I wonder if anyone has such a script or if someone is |
32 |
>>> working on such a script. |
33 |
>>> |
34 |
>>> Often a new software release is available from the home site but |
35 |
>>> portage has not been updated. I would like a script that would: |
36 |
>>> |
37 |
>>> check the source repository for a package |
38 |
>>> find the latest version |
39 |
>>> if the repository has a newer version |
40 |
>>> create an overlay with an ebuild for the new version |
41 |
>>> try to build the new package |
42 |
> |
43 |
> This would be relatively easy to automate, due to the standard bump |
44 |
> procedure below. It wouldn't always work, and the risk for even trying it |
45 |
> would be entirely the user's, because in some cases such a procedure WOULD |
46 |
> break the system, due to changes between versions that weren't |
47 |
> accounted for by the automated process, but it would be relatively easy to |
48 |
> write a script that would work for most minor bumps anyway, a reasonable |
49 |
> percentage of the time, risky tho it would indeed be to just blindly run |
50 |
> it. |
51 |
|
52 |
I assume that installing a package based on an automatically |
53 |
created ebuild may break a system. But how else can you |
54 |
determine that a new software package release is good on not? |
55 |
You need to try it. I just want a script that lowers the bar |
56 |
for the person that is willing to try. |
57 |
|
58 |
>>> Other times I will hear about a new software package and I want to try |
59 |
>>> it out. I would like a script that would take a URL for the home site |
60 |
>>> as an argument and then would operate like the script above. |
61 |
> |
62 |
> This simply couldn't be automated in a sane manner. There's too much |
63 |
|
64 |
Sane? A crude script that can create an ebuild with just |
65 |
the minimum fields may not work very often, but if it works |
66 |
for the package YOU want, then you have just saved YOUR time. |
67 |
If it partially works, then you are that much closer to testing |
68 |
the package. That extra time could then be used to improve the |
69 |
ebuild and maybe get it working. That seems pretty sane to me. |
70 |
|
71 |
> variability in the way various things build, and too much risk in simply |
72 |
> assuming the normal otherwise automatable procedure will work, and will |
73 |
> not do something strange enough that it could be harmful to a working |
74 |
> system. An initial build for an application isn't a simple matter. One |
75 |
|
76 |
I disagree. Many applications need very simple ebuilds. |
77 |
The feature I love about portage is that if you do end up |
78 |
putting files in less than optimum places, you can just remove |
79 |
the package and update the ebuild to use better locations. |
80 |
|
81 |
I wish /usr/portage were split into two parts. One for the core |
82 |
packages that you need when Gentoo is used as your operating |
83 |
system and another set for applications. I expect this issue |
84 |
has been discussed by the portage on Mac OS X people but I have |
85 |
not looked at their solution yet. The reason I mention this is |
86 |
that I would generally be interested in automatically creating |
87 |
ebuilds for the application set and would tend to leave the core |
88 |
OS side alone to avoid the sort of catastrophic events that you |
89 |
point out could happen. |
90 |
|
91 |
> must decide where things should go and supply the appropriate parameters |
92 |
> to have it install there. One must check that the licensing is suitable |
93 |
> for Gentoo, and further, whether Gentoo can mirror the sources or whether |
94 |
|
95 |
I am not suggesting Gentoo ever use an automated systems |
96 |
to put new software into the official portage tree. I would |
97 |
always want a person to be involved for the many reasons you |
98 |
mention here. That does not stop us from having a system that |
99 |
jump starts the ebuild creation process. |
100 |
|
101 |
> they have to be fetched directly from upstream due to licencing, possibly |
102 |
> manually, if the licensing is strict enough. One must verify the |
103 |
> setup/unpack/configure/compile/install/qmerge steps and create |
104 |
|
105 |
The script I am looking for will in fact have to create the |
106 |
overlay area and run through all the steps you mention. The |
107 |
nice thing is that emerge will do each step for you and it keeps |
108 |
multiple logs, so we can see if the build works or not. It is |
109 |
nice to not have to rewrite emerge. |
110 |
|
111 |
> non-defaults for anything that doesn't work normally. One must ensure |
112 |
> that the sequence doesn't violate the non-interactive rules of portage. |
113 |
> One must ensure the merged application is sufficiently configured to "just |
114 |
> work" out of the box, with no additional configuration, if at all possible |
115 |
|
116 |
The script I want may well create an ebuild that will fully |
117 |
succeed with emerge but then NOT just work. Ebuilds in the |
118 |
official Portage ebuilds need to "just work", but experimental |
119 |
ebuilds don't need to fully work. |
120 |
|
121 |
> (obviously it's not going to be possible with things like cups, where the |
122 |
> user must configure the printers they use as appropriate). One must make |
123 |
> some attempt to verify sane compile settings and that the package will use |
124 |
> those supplied by portage over its default settings. One must further |
125 |
> verify that "normal" CFLAGS/CXXFLAGS actually work. One must evaluate |
126 |
> dependencies both optional and required, setting the correct build and |
127 |
> runtime dependencies, and setting up USE flags as appropriate for the |
128 |
> optional ones. One must... One must... One must... Finally, appropriate |
129 |
> patches must be devised if the package doesn't by default pass any of |
130 |
> these steps, such that it does. |
131 |
> |
132 |
> Really, the devs do more for you by supplying Gentoo than picking their |
133 |
> nose all day. Don't you think if it were that easy (or even /possible/) |
134 |
> to automate creating an ebuild for a any random product out there, that |
135 |
> they would have such a system by now? I'm not a dev, but if I was, I'd be |
136 |
> insulted by just how lazy or stupid you obviously think they are, lazy if |
137 |
> you think they have such an approach and are simply hiding it as some big |
138 |
> secret, so they can do the easy stuff and make it look hard, stupid if you |
139 |
> think it's possible to automate the thing, and they haven't done it. |
140 |
|
141 |
This is a bit harsh. The devs do a great job. |
142 |
I don't know why there is not a an automated system to create |
143 |
ebuilds for random software, that is why I asked the list. |
144 |
Maybe the developers are busy improving the core system. That |
145 |
is what it looks like to me. |
146 |
|
147 |
But it is a closed system in the sense that there is not a |
148 |
public web site where a person like you or I can go, point to an |
149 |
interesting project, and have a first cut at an overlay created |
150 |
within the limits of the script. The web site could even try to |
151 |
compile and install in a virtual machine and give you a report |
152 |
before you ever tried it on your system. I would also want the |
153 |
web to have a way for people to register their success and |
154 |
failure and have a place where you could upload a fixed ebuild. |
155 |
|
156 |
Allowing non portage developers a chance to try their hand on a |
157 |
new ebuild and get feedback on their efforts seems to be taking |
158 |
advantage of the Open Source methods. |
159 |
> |
160 |
> Really, if there's no ebuild available at all, the user is best advised to |
161 |
> read the documentation to learn of any gotchas, then follow thru on the |
162 |
> steps manually, such that he can watch what's going on and see where any |
163 |
> breakage occurs, and attempt to fix it at that point. |
164 |
|
165 |
Or use a script that generates a skeleton that you can start |
166 |
with. |
167 |
> |
168 |
>>> If such scripts were available, it would be much easier to check out |
169 |
>>> new software releases. I could imagine a system that would scan |
170 |
>>> listings from freshmeat or other non-Gentoo site, build the ebuild and |
171 |
>>> try the package. In the simplest case you would get a go/no-go answer. |
172 |
>>> For the packages that download and build, you could try it out. If it |
173 |
>>> failed you could ignore it or try to figure out the problem. |
174 |
> |
175 |
> If it failed, you could indeed try to figure out the problem -- |
176 |
> potentially try to figure out why nothing you try to run works any more, |
177 |
> and your system won't boot, because the automated script created an ebuild |
178 |
> that blindly deleted /usr/bin/*, or something equally dangerous to a |
179 |
> system, because someone thought it was a bright idea to trust the build to |
180 |
> an automated script, sight entirely unseen, README and INSTALL entirely |
181 |
> unread! |
182 |
|
183 |
This sounds like the sort of problem you would get if you tried |
184 |
to update one of the core OS packages. I am thinking more along |
185 |
the lines of Asterisk. The portage tree is generally out of |
186 |
date. I think there is an alternate overlay site that is much |
187 |
more current but just bumping the version has a good chance of |
188 |
working. |
189 |
|
190 |
Or, I just saw a new package, http://www.getdemocracy.com/, and |
191 |
I was interested in trying it. It is a video player. I don't |
192 |
expect it to totally trash my system. But if it did and I had a |
193 |
web site where I could report on it, someone else might know how |
194 |
to fix the ebuild. |
195 |
> |
196 |
> There's simply no sane nor safe way to automate that stuff. For something |
197 |
> with an existing ebuild that might just need bumped, sure. For something |
198 |
> without an existing ebuild, I must conclude you really hadn't thought the |
199 |
> implications of what you were asking for thru AT ALL!! |
200 |
|
201 |
This just seems harsh. |
202 |
|
203 |
We can decode the human genome with a Perl script but we can't |
204 |
figure out how to use a few heuristics and a sandbox to get |
205 |
close to a working ebuild. I think we can and I am looking for |
206 |
existing scripts and ideas from people that will help us get to |
207 |
the point that most open source packages have an ebuild. |
208 |
|
209 |
I have thought about this a little bit. |
210 |
|
211 |
> |
212 |
>> I don't know about an automated process, there are a lot of variables |
213 |
>> that can factor into a version bump. However, for many simple |
214 |
>> packages, i.e. things that are not heavily patched or that do not have |
215 |
>> changing software requirements, it is usually sufficient to simply |
216 |
>> copy the old ebuild over to the new version name. |
217 |
>> |
218 |
>> Example, suppose net-www/fooapp has recently been bumped from version |
219 |
>> 1.001 to 1.002. You could do the following: |
220 |
>> |
221 |
>> mkdir -p /usr/local/portage/net-www |
222 |
>> cp -r /usr/portage/net-www/fooapp /usr/local/portage/net-www |
223 |
>> cd /usr/local/portage/net-www/fooapp |
224 |
>> cp fooapp-1.001.ebuild fooapp-1.002.ebuild |
225 |
>> ebuild fooapp-1.002.ebuild digest |
226 |
>> emerge -auv fooapp |
227 |
|
228 |
Thanks for this suggestion. I will create a shell script to |
229 |
automate this part of the process and test it out. Thanks. |
230 |
>> |
231 |
>> The reason for copying the entire /usr/portage/net-www/fooapp |
232 |
>> directory tree over is that there may be some patches or configuration |
233 |
>> files or such under the files/ subdirectory which are required by the |
234 |
>> ebuild. As I said, this procedure works a good 8 or 9 times out of 10 |
235 |
>> with simple ebuilds, sometimes some slight modifications are necessary |
236 |
>> and other times the modifications may be quite extensive. |
237 |
> |
238 |
> A solid idea, the steps of which are done by many, but a couple |
239 |
> modifications make it a bit better. |
240 |
> |
241 |
> Instead of copying over the entire dir, create the dir in the overlay (if |
242 |
> you don't already have it from a previous version), and copy over the |
243 |
> ebuild you intend to bump. You don't have to worry about previous |
244 |
> versions, or the changelog, metadata, or manifest files, since you the |
245 |
> former two are simply informational and the latter will be recreated. |
246 |
> Instead of copying over the files dir, make a symlink to the existing one |
247 |
> in the regular portage tree. (If you have any of your own patches or |
248 |
> modify any existing ones, create a different dir for those, and edit the |
249 |
> ebuild accordingly.) |
250 |
> |
251 |
> Since the files dir is a symlink, it will automatically change as the |
252 |
> portage tree changes. Thus, if patches are changed or added in the Gentoo |
253 |
> tree, the copy in the overlay will stay current, and any remerge or the |
254 |
> next bump will use the new versions. In some cases that's critical, as it |
255 |
> could mean you get security patches added since the static creation of the |
256 |
> overlay. |
257 |
> |
258 |
> Doing it this way, only copying the specific ebuild you'll be working on, |
259 |
> bumping or whatever, and using a symlink for the files dir, avoids the |
260 |
> risk of overwriting any work you've done on the same package previously, |
261 |
> that you may wish to save. It's also more space efficient. |
262 |
|
263 |
Another good suggestion. Thanks. |
264 |
> |
265 |
> ... |
266 |
> |
267 |
> In any case, with or without my changes, that the steps can be laid out as |
268 |
> a series of coommands like that indicates they can be made into a script. |
269 |
> It's an exercise left to the reader, then, to create a script that takes |
270 |
> the CPV (category/package-version) of the existing ebuild, and the version |
271 |
> of the the target ebuild, as command line parameters, and executes the |
272 |
> sequence of coommands necessary to end up with the bumped ebuild ready to |
273 |
> go in the overlay. Depending on how fancy you want to make the script, |
274 |
> with or without a --help output, with or without validity checking and |
275 |
> handling any possible errors, with or without the overlay location |
276 |
> hard-coded into the script vs. configurable using a config file or a |
277 |
> command line option, and depending on how good you are with bash scripting |
278 |
> or scripting in your language of choice (perl, python, whatever), reading |
279 |
> this thread could have taken you longer than creating the script will. |
280 |
> |
281 |
>> BTW, if you do discover a package that has a new version and you get a |
282 |
>> good working ebuild for the new version make sure that you post it to |
283 |
>> bugzilla so that a developer can get it into the tree. |
284 |
|
285 |
Yep! |
286 |
> |
287 |
> Always a good recommendation! =8^) |
288 |
> |
289 |
> -- |
290 |
> Duncan - List replies preferred. No HTML msgs. |
291 |
> "Every nonfree program has a lord, a master -- |
292 |
> and if you use the program, he is your master." Richard Stallman in |
293 |
> http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html |
294 |
> gentoo-amd64@g.o mailing list |
295 |
-- |
296 |
gentoo-amd64@g.o mailing list |