Re: [gentoo-dev] [GLEP] Web Application Installation

[Stuart Herbert]

On Monday 04 August 2003 6:11 pm, Max Kalika wrote:
> Good morning!

Evening, Max.

> See, I don't think that running two (or more) copies of an application is
> supporting virtual hosts.  Upgrading all the files is a pain.  Why not have
> one installation of the core files and multiple installations of the config
> files?  This is, of course, very application specific -- most of them will
> still need database upgrades, but that has to be done by the sysadmin.

Because that won't quite work, unless the app is aware of virtual hosting - 
and many (if not most) aren't.  Think about it.

Here's an example.  Imagine hosting (oh, I don't know) www.iammax.com and 
www.maxisgreat.com on the same physical box.  Document roots for each are 
/var/www/<host>/public_html/ for arguments sake.

Now imagine running phpBB on both domains, to provide separate forums.  As far 
as phpBB is concerned, when it accesses, say, login.php on www.iammax.com, 
the URL is http://www.iammax.com/phpbb/login.php, which translates through as 
/var/www/www.iammax.com/public_html/phpbb/login.php.  Similarly, login.php on 
www.maxisgreat.com translates through as 
/var/www/www.maxisgreat.com/public_html/phpbb/login.php

Given this scenario, how do you make these two sites share the same phpBB code 
files?  Here's a few possible ways, and the problems that they cause.  Chip 
in with others - because this is the core problem.

a) A bit of mod_alias magic, and you make the /phpbb/ directories aliases for 
/usr/share/webapps/phpBB-<version>/files/.  If you do this, though, how do 
you get each copy of phpBB to use a separate set of configuration files?  
What happens when the app needs write-access to the directories?  The 
directories are on /usr, and we're all agreed that /usr should be mountable 
read-only.  And does every webserver have something like mod_alias in the 
first place?

If I've understood your eclass correctly, this is what it tries to do, yes?

b) A bit of .htaccess magic, and you have the directory structure of the 
webapp on /var, but directives telling PHP where to find the .php files by 
setting the include_path.  The config file problem is solved, because you can 
drop in local config files, and there are real directories that can be made 
writable.  Only problem with this approach is that not every type of web 
server has the equivalent of .htaccess files - and the PHP SAPI for each type 
of web server doesn't necessarily support configuration directives in config 
files either.  And that's before we think about Perl, Python and other ways 
that webapps can be implemented.

I believe that this is Robin's basic idea, if I've understood correctly.  It's 
a neat solution, but perhaps not a universal one.

c) A bit of find and cp -l, and you've got the directory structure of the 
webapp on /var, plus links back to the original on /usr.  Again, the config 
file problem is solved, because local copies can be parachuted in, and again 
there are real directories that can be made writable.  No webserver-specific 
tricks are required - as far as the web server is concerned, each domain has 
its own installation of phpBB.

d) ?? There must be other solutions to the problem that haven't been discussed 
in this thread.  Please - contribute!

> How do they handle db updates?  How do they handle config file updates?

I've never asked, and I don't want to know ;-)  Maybe Debian just doesn't 
change that much from year to year <grin>.

> > 1) Your eclass doesn't correctly (as I understand correct usage to be!)
> > support the apache2 USE flag.  Easy enough to do - see my webapp-apache
> > eclass for an example.
>
> Why base it on the flag?  If the webserver is installed and is supported,
> configure for it.

Because that's what USE flags are there for.  If the user puts '-apache2' in 
their USE flags, it's the job of the ebuild to respect that.  Otherwise, the 
ebuild is broken - and probably in breach of policy too.

> > 2) Your eclass doesn't specify the permissions that the source files
> > should be  installed under.  Again, easy enough to fix.
>
> This is very straightforward to add.

Agreed.

> > 3) Your eclass doesn't provide support for running multiple copies of the
> > same  app on the same machine.  This is a showstopper.
>
> This whole thing needs a separate discussion.  There's no portage support
> for dealing with more than one installation of the same version of the same
> package.

Why a separate discussion?  If this GLEP isn't here to address this 
fundamental issue, then I'd say that the GLEP has the wrong scope.

You're right - Portage currently can't do all of this on its own.  Perhaps it 
never will be able to.  As I understand it, that's why Robin's volunteered to 
write and maintain additional tools to bridge the gap.

> > 4) Your eclass requires the admin to stop and start Apache as part of the
> > install.  This is a showstopper.  Not every site will want to stop and
> > start  their web server just because they've installed a new app.
> > Imagine a site  hosting hundreds of domains, and having to take them
> > *all* offline at the  same time just because phpBB's been upgraded (for
> > example!).  Robin's idea of  creating .htaccess files under the document
> > root deals with this much more  managably - although I think we're gonna
> > end up using symlinks, as that'll  make it easy to support multiple web
> > servers.
>
> ..htaccess files only work in already web-accessable directories (in your
> case, DocumentRoot).  If we're putting applications in /usr/share/webapps,
> the webserver has no idea to look there. Which is what the Alias directive
> does in all the .conf files that are generated.  However, Alias is not
> allowed in .htaccess AFAIK.

See about for why I think .htaccess isn't the way to go anyway ;-)

> http://httpd.apache.org/docs-2.0/mod/mod_alias.html#alias
>
> Besides that, the server doesn't have to be restarted -- just HUPed.

Fair enough ;-)

> > 6) I *like* the idea that check_php() is in this
> > eclass, because that check is  specific to mod_php under Apache.  I'm
> > gonna steal that and add it to my  webapp-apache eclass ;-)
>
> Steal away!

Done ;-)

> Keep in mind it is not completely accurate if the per-package
> USE flags goes into portage (http://bugs.gentoo.org/show_bug.cgi?id=13616).
> Things in PUSE will also have to be taken into account.  But this is a
> completely different matter and will be easy to integrate.

It's a nice start though.  Ideally, we need a programatic interface to 
portage, an API we can use to handle these types of queries.  karltk is 
working on one ;-)

> > 7) Your variable names are not generic enough for my liking ;-)
> > AWEB_CFG, for  example, might be better off being WEBAPP_CFG.
>
> Hey!  I originally had them as WEBAPP_* but was told to change the eclass
> to apache-webapp to distinguish from other servers so I wanted the
> variables to reflect the name of the eclass.  I'm as flexible on this as
> playdoh. :-)

/me bites back his comment about the person who told you to do that ;-)

> > 8) Instead of trying to supply an all-encompasing
> > apache-webapp_src_install(),  relying as it does on defining global
> > variables, I'd have supplied a number  of individual functions to do each
> > bit.  Say, a webapp-install-appconfig,  webapp-install-serverconfig each
> > taking parameters (this is off the top of my  head here ;-)  This is a
> > personal preference thing.
>
> Thats just more to call from the ebuild.  My goal was to have very small
> ebuilds -- just declare a few variables and inherit the eclass.

Hmm ... there's a tradeoff here.  Smaller ebuilds and less flexible (and 
re-usable) eclasses.  Or larger ebuilds, but more re-usable eclasses.  I 
guess I prefer the latter.

> > 9) If I'm not mistaken, your eclass does nothing to ensure that the
> > webapp can  find the configuration files you've moved into
> > /etc/webapps/$PN/.   Personally, I'm coming to the conclusion that
> > /etc/webapps/$PN/ isn't a good  idea, because again it doesn't support
> > the idea of running multiple copies of  the same app on the same machine.
>
> I'd say we should be shooting for an easy upgrade path first.

Good point ;-)  But what's the point of installing and upgrading configuration 
files that the application is never going to look at? ;-) (Sorry, it's 22:30, 
and still ridiculously hot here)

> If config
> files aren't stored in /etc/webapps/${PN}, then we need to have a way of
> generating an env.d/${PN} which contains a CONFIG_PROTECT line, otherwise
> we're forcing sysadins to reconfigure the application at every upgrade.

Yeah, but if each instance of the installed app has its own config files, then 
what's the relevance of /etc/webapps at all?

> And as I mentioned before, there's no way for portage to handle
> multiple-copy installs, so I'm not sure the best way to go about achieving
> this goal.

This is the problem that I think we should be solving - how to support the 
installation, configuration, and upgrading of multiple-copy installs.

> > 10) I'm coming to the conclusion that 'emerge -u <webapp>' shouldn't
> > overwrite  the older version, but should always install alongside, in a
> > different slot,  so that sites can easily run different versions of apps
> > as required.  Perhaps  this should be configurable somehow?  Your eclass
> > doesn't make this possible.
>
> Why should this be handled any different then the rest of apps handled by
> portage?

Because most of the rest of the apps handled by portage don't run in a virtual 
domain environment.  Webapps really are a different beast.

> If a sysadmin doesn't want older versions removed, just add
> AUTOCLEAN=no to make.conf.  Granted, the current eclass doesn't use the
> full ${PF} in the target directory, but that is easily changed.

Here's an example.  Imagine you're running your own hosting firm, and you have 
a non-trivial number of customers using the same webapp (say, phpBB for 
arguments sake).  A new version of phpBB comes out.  Some customers will ask 
for the upgrade, and some explicitly will ask you not to upgrade.  So, in 
this situation, you need to have two copies of phpBB installed on the same 
box at the same time.

Now let's look at what happens when you run 'emerge -u phpBB', with the 
appropriate ACCEPT_KEYWORDS of course.  Portage goes and installs the new 
version of phpBB over the top of the old phpBB files.  The old version of 
phpBB gets overwritten, yes?  I don't see how AUTOCLEAN will prevent that 
from happening.

The whole point of SLOTing apps (as I understand it) is to allow you to have 
multiple versions installed alongside each other.  This is the mechanism that 
Portage offers us.  

> Ok.  Hopefully we can get this thing hashed out and out the door so things
> can start getting into shape soon.

Hell yes.

> > 1) Apache1/Apache2 conundrum
> >
> > My eclass uses the detection technique adopted for mod_php, and no-one
> > has  complained about that.  If this eclass is invalid, then so's the
> > ebuild for  mod_php.  And I don't think it is.
>
> Ok, this one is a keeper.

Kudos to Robin - it's his algorithm.

> > 2) Support for multiple DocumentRoot configurations, and also
> > 3) Binary packages installing on machines with different DocumentRoot
> > values
> >
> > Until the GLEP is firmed up and approved, we don't have an agreed
> > solution to  implement.
>
> Ah!  Ok, so before we have this issues ironed out, the current status-quo
> will just be maintained.  Fair enough.

Agreed.

> > Please excuse me, but I don't want to put support for existing ebuilds on
> > hold  while we debate the GLEP.  I believe that we *have* to continue
> > support until  we're ready and able to switch.  Stopping maintenance
> > activities is *not* an  option.
>
> Since I'm the laziest person I know, I didn't want to do the work twice.
> I'll just hold off on introducing more ebuilds which will have to be
> converted later. :-)

That's up to you.  If you're maintaining webapp ebuilds currently in portage, 
though, I'd urge you not to stop maintaining them just because you're waiting 
for a design solution via this GLEP.

> > 4) Which user/group to use
> >
> > My class uses Robin's suggestion, and assumes that Apache is running with
> > the  default settings of apache.apache.
>
> Right, but as we already agreed, not all apps need to have their files
> owned by apache:apache.  This should be configurable in the eclass.
> Correct?

Do we need a new user/group to own most of the files?  And then we just make 
the files that need write-access owned by the webserver?

> > 5) DocumentRoot pointing to a read-only mount
> >
> > As far as I'm concerned, that's like trying to do 'make bzlilo' with
> > /boot not  mounted, or run an 'emerge' with /usr mounted read-only.  It's
> > the sysadmin's  job to make sure that any necessary filesystems are
> > mounted read/write before  an installation is attempted.  This is not a
> > problem unique to web  applications.
>
> The issue is not whether DocumentRoot is a read-only mount during the
> install, but during the day-to-day operations.  Running 'emerge' with /usr
> mounted read-only is not the same as having apache running a webapp that
> needs to write to /usr -- one is done seldomly, the other is all the time.

In his email in that thread you pointed me at, Robin was explicitly talking 
about /var being a read-only NFS mount at install time.

> > 6) "it's weak"
> >
> > That's not an argument, it's an opinion ;-)  Anyway, I've taken 5-10
> > lines of  broken and incorrectly duplicated code from a number of
> > ebuilds, and moved  them all into one place where they can be re-used and
> > maintained for now.   Reduced defects is a strong argument, not a weak
> > one.
>
> I never made the "weak" argument

Again, this is in response to the comments in the archived thread - not this 
one ;-)  You didn't make the comment - someone else did.

> -- I don't believe in it myself.  Things
> need to be justified a bit better than that, so I'm completely with you on
> this one.

:)

> Ok, agreed.  Troy, is there any chance for another draft with some of these
> things incorporated?  (Thanks a bezillion, btw, for putting up with me) :-)

Who's "putting up" with you?  I don't feel like I am!  I'm just grateful that 
we're both interested in finding a solution to this problem.  Now, if just a 
few more people would chip in and help these discussions ... ;-)

> And a successful one at that from what I saw in my emerge --sync this
> morning. :-)  Thanks!

By the time I got Saturday evening, it was all but over.  Didn't see a single 
person in #gentoo-bugs who I could help :(

> If you say it is very easy to do, I'm on board.

If it's not easy to do, then we'll scrap it and come up with something better.

> I can't personally speak
> for other webservers, so I'll leave the decision of whether/how to support
> others to those with the experience.

Fair enough.

> So in any case, we have to pull the
> apache-specific things out into a separate framework.  Therefore things
> like DocumentRoot can't even be considered.  So a central location for
> webapps must be once again taken into account.

Yep.  How does this sound as the design of the central layout?  Let's agree a 
design, so that it can be added to the GLEP.

* /usr/webapps/<app-name> as the main directory.  
* /usr/webapps/<app-name>/public_html/ for files served by the web server
* /usr/webapps/<app-name>/cgi-bin/ for CGI-BIN files
* /etc/webapps/<app-name>/ to hold the box-default config files
* <app-name> is ${PN} for non-slotted packages
* <app-name> is ${P} for slotted packages

> ok, I didn't realize that it wasn't a permanent solution.  All is ok.

Neat.

> >> How is mod_php related to the way applications are installed?
> >
> > Erm, how about the whole 'do I use Apache 1 or Apache 2' conumdrum?  See
> > the  mod_php ebuild for details.
>
> Whichever is installed gets configured.  Either or both get touched,
> depending on what is detected.  It is up to the sysadmin to start one or
> the either automatically with rc-update.  I don't see a problem if both are
> configured (if detected), but only one is running.

Yeah, but as discussed earlier: 

a) you need a standard way of detecting which one is installed, and 
b) you need to honour the 'apache2' USE flag

Take care,
Stu
-- 
Stuart Herbert                                              stuart@g.o
Gentoo Developer                                       http://www.gentoo.org/
Beta packages for download            http://dev.gentoo.org/~stuart/packages/

GnuGP key id# F9AFC57C available from http://pgp.mit.edu
Key fingerprint = 31FB 50D4 1F88 E227 F319  C549 0C2F 80BA F9AF C57C
--