1 |
Hi Keshav, |
2 |
|
3 |
thanks for pointing out this thread. "log messages" is an interesting |
4 |
title to discuss the future directions for Sage. :) I added |
5 |
gentoo-science to the CC list, since that is the most likely location |
6 |
for sage-on-gentoo discussion and the gentoo-developers might be |
7 |
interested in the contents of this thread. |
8 |
|
9 |
On Mon, 20 Feb 2012 22:45:24 +0800 |
10 |
Keshav Kini <keshav.kini@×××××.com> wrote: |
11 |
|
12 |
> Robert Bradshaw <robertwb@×××××××××××××××.edu> writes: |
13 |
> > On Fri, Feb 17, 2012 at 1:09 AM, Keshav Kini |
14 |
> > <keshav.kini@×××××.com> wrote: |
15 |
> >> Robert Bradshaw <robertwb@×××××××××××××××.edu> writes: |
16 |
> >> In the category of "glue code" I meant to include everything in |
17 |
> >> $SAGE_LOCAL/bin/sage-*. I see much of that stuff as more related to |
18 |
> >> maintenance of the entire distribution of software we ship than to |
19 |
> >> the actual Python/Cython code in the Sage library. Of course, if |
20 |
> >> we do switch to Prefix or something like it, most of it will become |
21 |
> >> unnecessary, or can actually be fit into the portage data tree as |
22 |
> >> "Tools", so I see your point. |
23 |
> > |
24 |
> > Yeah, tools describes it well (or at least what remains after the |
25 |
> > package management stuff is gone, like testing and coverage and |
26 |
> > stuff). |
27 |
> |
28 |
> Well, I meant that there is a "Tools" directory in sage-on-gentoo. I'm |
29 |
> not sure if this is a standard thing in the directory structure of a |
30 |
> Gentoo Prefix tree, or, if it is, what sort of stuff is supposed to go |
31 |
> there, but I see some utility scripts, presumably written by François |
32 |
> or Christopher, in there and was suggesting we might be putting our |
33 |
> sage-* scripts there too. But I don't know enough about Prefix to be |
34 |
> sure whether that makes sense. |
35 |
|
36 |
AFAIK, Tools contains the scripts that are used to maintain the |
37 |
overlay. sage-on-gentoo contains these packages for Sage: |
38 |
|
39 |
sage-baselayout |
40 |
sage-clib |
41 |
sage-data-conway_polynomials |
42 |
sage-data-elliptic_curves |
43 |
sage-data-graphs |
44 |
sage-data-polytopes_db |
45 |
sage-doc |
46 |
sage-extcode |
47 |
sage-flasknotebook |
48 |
sage-notebook |
49 |
sage |
50 |
|
51 |
baselayout is what gentoo calls basic scripts. That is the package |
52 |
containing the relevant part of sage-scripts. While I like the |
53 |
separation of the data packages (I assume these don't depend on code |
54 |
from the sage library), I am not sure about the clib, doc and the |
55 |
library separation. I'd like to keep those together to match the Sage |
56 |
development pattern as close as possible. |
57 |
|
58 |
|
59 |
> >>> I think Sage will be monolithic and Windows be VM for the near |
60 |
> >>> future at least, with a larger percentage of people using a Sage |
61 |
> >>> install "in the cloud" on a university or otherwise hosted server |
62 |
> >>> for the near-mid term future. But as you said who knows... |
63 |
> >> |
64 |
> >> Yes, I agree. That seems like the most likely future, at the |
65 |
> >> moment. However, William has asked me to write a Sage Enhancement |
66 |
> >> Proposal for switching to git, and I think we concluded on IRC |
67 |
> >> that it might make sense to make a long term timeline for other |
68 |
> >> big changes as well, such as what we're talking about now - or at |
69 |
> >> least a proposal for such a timeline :) |
70 |
> > |
71 |
> > +1, which is why we should be having this discussion now. Switching |
72 |
> > to git could probably happen a lot faster than revamping the build |
73 |
> > system, but at least we could consolidate some of the repositories |
74 |
> > and re-seat it at a higher level in the directory structure so |
75 |
> > moving to a new build system is "just a patch" :). |
76 |
> |
77 |
> Absolutely. Switching to git is much easier than revamping the build |
78 |
> system. The only reason I'm mentally grouping them together is that |
79 |
> consolidating SPKGs, which would be part of remaking the build system |
80 |
> as I've described, is also related to version control. |
81 |
> |
82 |
> But that actually doesn't make a lot of sense now that I think about |
83 |
> it. We would basically be throwing away the old SPKGs' files, so I |
84 |
> don't know if it's even really worth preserving their history in the |
85 |
> portage tree of Prefix, assuming we move to Prefix - we could easily |
86 |
> preserve their history somewhere else instead. |
87 |
> |
88 |
> Of course, even if it's not really related to the switch to git, I'd |
89 |
> definitely like to keep a build system revamp on the horizon! |
90 |
> |
91 |
> > Thanks for filling me in. I was aware of the sage-on-gentoo effort, |
92 |
> > but never looked into it technically. The more I see of it the |
93 |
> > better it sounds. This seems to have precisely the flexibility we |
94 |
> > need. |
95 |
> |
96 |
> I'm very glad to hear you think so too! Spread the word! :P |
97 |
> |
98 |
> Just to be clear, sage-on-gentoo itself has no goal of being a way to |
99 |
> distribute Sage in general. It is simply a port of Sage and its SPKGs |
100 |
> to the Gentoo package management system, using the "overlay" |
101 |
> architecture the Gentoo distro has (similar to Personal Package |
102 |
> Archives on Ubuntu, if you're familiar with those). Basically you |
103 |
> just put something in a config file on your Gentoo system which says |
104 |
> "augment the package repository with ebuilds from sage-on-gentoo", |
105 |
> and you can now install Sage system-wide. |
106 |
> |
107 |
> It is Burcin's lmonade project which combines sage-on-gentoo with |
108 |
> Gentoo Prefix, which is a portable version of the Gentoo package |
109 |
> management system which can be run in a folder on any system, and |
110 |
> more importantly, can be customized to supply any set of packages, |
111 |
> not necessarily the entire set of officially sanctioned packages that |
112 |
> comprise the Gentoo Linux distribution. As the name suggests, it's |
113 |
> similar to how you might use `./configure --prefix=/some/dir` in |
114 |
> order to install a program to /some/dir/bin rather than /usr/bin, say. |
115 |
|
116 |
Thanks Keshav, that's a great summary of the relation between |
117 |
gentoo-prefix, sage-on-gentoo and lmonade [1]. For the record, I'd be |
118 |
very happy to see Sage use lmonade for the distribution aspects. |
119 |
|
120 |
[1] http://www.lmona.de/ |
121 |
|
122 |
lmonade was born because I thought that a customized Sage distribution |
123 |
like William's psage [2] was not maintainable in the long term. It |
124 |
still needs some work to cover all functionality of the Sage |
125 |
distribution, such as relocation and support for development of Sage. |
126 |
Given time, these are not hard problems. Unfortunately, I haven't had |
127 |
much time to work on lmonade lately. |
128 |
|
129 |
[2] http://purple.sagemath.org/ |
130 |
|
131 |
> Obviously, this is pretty necessary for Sage proper, which likes to |
132 |
> sit in its own folder and have control over its own private copy of |
133 |
> various things. To do otherwise would be to abandon doing our own |
134 |
> packaging altogether, which I think is basically impossible. So if we |
135 |
> were to adapt anything, it would be lmonade, not pure sage-on-gentoo. |
136 |
> |
137 |
> > One thing seems a bit unclear: are the sage-specific changes pushed |
138 |
> > into some global repository for, e.g. gap, or are they local to the |
139 |
> > sage install? If they are, how is it proposed that versioning be |
140 |
> > handled (will there be a sage-5.0 and sage.5.1 tag, or sage-r1 and |
141 |
> > sage-r2, ?) Or is this repository of Sage use metadata something |
142 |
> > that we maintain? |
143 |
> |
144 |
> I'm not really sure what you mean by a "global repository" or "local |
145 |
> to the sage install". Versioning will be basically the same as how we |
146 |
> do SPKGs right now, namely that we'll take the upstream version of the |
147 |
> package and add on a "Sage patch version", which in portage must be |
148 |
> represented as a suffix of the form -r1, -r2, -r3, etc. So for example |
149 |
> what is now sympow-1.018.1.p11.spkg would become, in the new system, a |
150 |
> possible set of patches, plus sympow-1.018.1-r11.ebuild, containing |
151 |
> (among other things) a URL from which to download the sympow 1.018.1 |
152 |
> source tarball. |
153 |
> |
154 |
> Here is a highly abbreviated directory tree of lmonade, as it appears |
155 |
> in the installation I made the other day, for reference: |
156 |
> |
157 |
> lmonade/ |
158 |
> | dist package management / build system |
159 |
> | | distfiles/ downloaded upstream source tarballs |
160 |
> | | log/ build logs |
161 |
> | ` portage/ ebuilds, patches, etc. |
162 |
> | ` sci-mathematics/ |
163 |
> | ` gap/ |
164 |
> | | files/ |
165 |
> | | | gap-4.4.12-sage-and-steve-lintons-itanium.patch |
166 |
> | | ` gap-4.4.12-sage-strict_aliasing.patch |
167 |
> | | gap-4.4.12-r2.ebuild |
168 |
> | ` metadata.xml |
169 |
> ` local/ software is installed here |
170 |
> | bin/, etc/, include/, lib/, libexec/, sbin/ |
171 |
> ` share/ |
172 |
> ` sage/ |
173 |
> ` devel/ corresponds to $SAGE_ROOT/devel today |
174 |
> |
175 |
> So the "package repository" would be dist/portage, and the "sage |
176 |
> library repository" would be kept at local/share/sage/devel/sage-main |
177 |
> (or hopefully just ...devel/sage, once we get rid of `sage -clone`). |
178 |
> I don't actually know my way around this directory structure very |
179 |
> well, but those are the main takeaways, I think. |
180 |
|
181 |
Your description is accurate. The development repositories will be in |
182 |
devel/. pynac and pypolymake live ebuilds already put their files in |
183 |
this location. The separation of the sage ebuilds stopped me from using |
184 |
the same scheme for sage up to now. The goal is to put the development |
185 |
repository for a package foo in devel/foo once the user runs |
186 |
|
187 |
lmnd devel foo |
188 |
|
189 |
> You can see the contents of gap-4.4.12-r2.ebuild here: |
190 |
> https://github.com/cschwan/sage-on-gentoo/blob/83deaae5/sci-mathematics/gap/gap-4.4.12-r2.ebuild |
191 |
> |
192 |
> There are some seeming oddities there, like the USE flag "emacs" which |
193 |
> pulls in Emacs as a dependency and then installs the Emacs mode for |
194 |
> GAP into it - clearly not something we want to do, and indeed it's not |
195 |
> enabled by default in lmonade. It's there because, as I said, |
196 |
> sage-on-gentoo is meant to allow you to install Sage into a full |
197 |
> Gentoo system, where it *would* make sense to do that. |
198 |
> |
199 |
> I guess to avoid extra maintenance load we'll just have to leave that |
200 |
> extra stuff in, both in sage-on-gentoo and on our own packaging repo. |
201 |
> (And who knows, maybe someone *does* want a local copy of Emacs or |
202 |
> whateverinside their Sage installation...) I doubt we'll be able to |
203 |
> actually use sage-on-gentoo directly as our packaging repo, but at |
204 |
> least we can keep them similar (?). |
205 |
> |
206 |
> By the way, it looks like only three packages in sage-on-gentoo |
207 |
> actually have a "sage" USE flag defined (PolyBoRi, Python, and |
208 |
> LinBox). It seems that the others, including GAP, just install Sage's |
209 |
> patches whether you want them or not. I'm not sure why this is - |
210 |
> maybe it's just because it's painful to specially create a "sage" USE |
211 |
> flag for every package, or maybe it's because sage-on-gentoo wants to |
212 |
> get Sage's patches included in Gentoo proper as default patches, or |
213 |
> maybe there's some other reason. |
214 |
|
215 |
AFAIR, the only patch contained in the GAP spkg was a fix for itanium |
216 |
from Steve Linton. This is useful in general. No need to hide it behind |
217 |
a use flag. |
218 |
|
219 |
> I'm getting way out of my depth here, as you can tell, so I'm CCing |
220 |
> Burcin and François, in case they haven't been reading this exchange |
221 |
> and have something to add or remove from what I've been saying! I'm |
222 |
> sure Burcin in particular has thought a lot more about where all this |
223 |
> could go than I have. Of course, feel free not to jump in if you're |
224 |
> too busy. |
225 |
|
226 |
Clearly you've also thought about this a lot. We seem to agree on almost |
227 |
everything. I am also happy to see that even with the current lack of |
228 |
documentation, main design criteria come through. I'd really like to |
229 |
see some of your explanations on the lmonade wiki actually. :) |
230 |
|
231 |
> Well, actually I'm going to send this message through Gmane, so maybe |
232 |
> they won't be CC'd after all. We'll see :) |
233 |
|
234 |
Gmane CC works. |
235 |
|
236 |
> >> I think we can get a pretty good alternative to this with the split |
237 |
> >> system I'm describing (nimble package management repo for everyone |
238 |
> >> and carefully reviewed sage repo for creating releases from), if I |
239 |
> >> understand correctly. We would include the ebuild for Sage itself |
240 |
> >> inside the Sage repository, and export it to our package |
241 |
> >> repository when we make a release of Sage. |
242 |
> > |
243 |
> > By "our package repository" are you thinking of something maintained |
244 |
> > by us, or by gentoo? |
245 |
> |
246 |
> Gentoo's involvement with sage-on-gentoo and lmonade ends at providing |
247 |
> the package manager software, and in the case of sage-on-gentoo, |
248 |
> putting sage-on-gentoo on the official list of overlays, so that you |
249 |
> can attach sage-on-gentoo to your Gentoo system with a simple |
250 |
> command, `layman -a sage-on-gentoo`. |
251 |
> |
252 |
> So our package repository would be maintained by us. Of course, we |
253 |
> would only need to actively maintain the ebuilds and patches for |
254 |
> programs which we actually patch. No more would it be necessary to |
255 |
> create an SPKG for a program which needs no Sage-specific patches, as |
256 |
> long as that package is already available on Gentoo - we can just |
257 |
> steal the ebuild file from Gentoo's portage tree. |
258 |
|
259 |
We could also share the ebuilds with Gentoo of course. Most Gentoo users |
260 |
don't mind using bleeding edge versions of software anyway. The whole |
261 |
point of using portage instead of the spkg system is to eliminate the |
262 |
need to duplicate this packaging work. |
263 |
|
264 |
> > But yes, this sounds like a great idea! So the |
265 |
> > collection of .ebuild files (and their supporting patches, etc.) is |
266 |
> > in the Sage repository, but periodically exported somewhere central |
267 |
> > (owned by us? Gentoo?) that people pull from, right? |
268 |
> |
269 |
> Er, nope. The ebuild corresponding to the Sage library itself would be |
270 |
> stored in the Sage repository, and periodically (upon the release of a |
271 |
> new Sage version) be exported into our package repository. All other |
272 |
> ebuilds and patches would be put directly into that package |
273 |
> repository, which would be hosted live somewhere, say on github. Our |
274 |
> package repository would look very similar to the current |
275 |
> sage-on-gentoo repository, i.e. be a portage tree providing ebuilds. |
276 |
> The other repository would be the Sage repository. |
277 |
> |
278 |
> In other words, we would consider the Sage library as another |
279 |
> "upstream package", except that it's aware of its part to play in the |
280 |
> Sage package repository, and thus provides its own ebuild as part of |
281 |
> the distributed source code each time it distributes a new version |
282 |
> (i.e when we release a new version), whereas for other packages we |
283 |
> would have to write our own ebuilds for them (or borrow them from |
284 |
> Gentoo), not constantly, but whenever the upstream developers |
285 |
> released a new version. |
286 |
|
287 |
I don't see the need to keep the Sage ebuild in the repository |
288 |
containing the Sage library. It should also be in the package |
289 |
repository. Just like: |
290 |
|
291 |
https://bitbucket.org/burcin/lmnd-prefix/src/tip/sci-mathematics/sage/ |
292 |
|
293 |
(I really need to update. Actually, I was just working on using git to |
294 |
automate synchronizing with gentoo-x86, gentoo-prefix, gentoo-science |
295 |
and sage-on-gentoo.) |
296 |
|
297 |
> > All I could ask for more is that the package management/build system |
298 |
> > be functional (hermetic) and reversible. |
299 |
> |
300 |
> Well, I don't know how hermetic portage is, but it's a sight more so |
301 |
> than SPKGs, anyway :) |
302 |
> |
303 |
> I remember someone was discussing using Nix_ for Sage on sage-devel |
304 |
> several months ago. But I don't think it's really mature enough to do |
305 |
> so, not to mention that we lose the advantage of being able to steal |
306 |
> packages from a mature Linux distribution like Gentoo. |
307 |
> |
308 |
> .. _Nix: http://nixos.org/nix/ |
309 |
|
310 |
Dag mentioned it a while ago: |
311 |
|
312 |
https://github.com/dagss/scidist/blob/master/ideas.rst |
313 |
|
314 |
Being able to reuse the work of gentoo-prefix, gentoo-science and |
315 |
sage-on-gentoo folk was exactly my reason for going with gentoo-prefix |
316 |
and portage. Many thanks to the Gentoo developers, Francois and |
317 |
Christopher for making this possible. |
318 |
|
319 |
|
320 |
|
321 |
Cheers, |
322 |
Burcin |