[gentoo-dev] The results of an evening of Apache and PAM debugging

[Anthony Gorecki]

This message is fairly long, however I've decided to pass it along in the hope 
that someone may find it useful as a reference source. It was originally 
intended to be a problem description and debugging request for the gnomes to 
ponder over.


I've recently encountered an interesting debugging paradox. Upon deploying 
Subversion with an Apache-based server, I discovered an interesting issue 
with PAM that seems to be intent on defying explanation:

I have a repository that's accessible through <https://svn.domain.com/>, with 
the repository root set at location / (using Subversion 1.2.0, which has a 
fix for the cache bug that prevented repositories from being located at the 
root of a web address). The server is setup to allow read-only access to the 
repository through unencrypted HTTP connections, while write operations must 
use HTTPS (TLSv1, AES256). 

When accessing the repository through HTTPS, Apache dispatches an 
authorization request to mod_auth_pam and PAM, which in turn passes the 
request off as follows:

The username and password authentication (auth) process is passed to pam_krb5 
and the accompanying KDC. This process works flawlessly on every service, 
including Apache's requests. The logs indicate successful logins when 
submitting a username and password via web request.

The account management (account) process is passed to pam_unix, which then 
does a bit of nsswitch magic to tie everything together. In this context, 
only network-based user accounts are permitted, therefore nsswitch uses the 
NSVS (MySQL backend) service to retrieve all of the information. For example, 
calls to getspnam are handled as follows:

getspnam \
        SELECT \
'%1$s','x',1,0,99999,7,0,DATEDIFF(expiration_date,'1970-01-01'),0 \
        FROM users \
        WHERE username='%1$s' \
        LIMIT 1

Herein lies the problem. During the authorization process, PAM will fail 
during the account stage with the error: "PAM: user 'agorecki'  - invalid 
account: Authentication service cannot retrieve authentication info."

For whatever reason, the debug and audit flags in the pam_unix module are not 
functional. As a result, I extracted the information I needed directly from 
the source code, and found the error location to be:

./Linux-PAM-0.78/modules/pam_unix/pam_unix_acct.c:
127         spent = _pammodutil_getspnam (pamh, uname);
128     else
129         return PAM_SUCCESS;
130
131     if (!spent)
132         if (on(UNIX_BROKEN_SHADOW,ctrl))
133             return PAM_SUCCESS;
134
135     if (!spent)
136         return PAM_AUTHINFO_UNAVAIL;    /* Couldn't get username from 
shadow */

The call on line 127 takes place, and the error originates on line 136.

For the purposes of debugging, I've removed all chroot restrictions from 
Apache and have given it complete access to the filesystem. The Apache 
Subversion interface works through the unencrypted connections, which do not 
require authentication.

Hours of Google searching turned up nothing of use; what I don't understand is 
how Apache can fail to retrieve the shadow information when it's accessible 
by any unprivileged--

While writing the above, it occurred to me to run a final check on the debug 
output of nsvsd, which produced: 

run_query: query: SELECT 'agorecki','x',user_id,group_id,CONCAT(first_name,' 
',last_name),CONCAT('/home/',username),'/bin/bash' FROM users WHERE 
username='agorecki' AND group_id IS NOT NULL LIMIT 1
send_response: sending 120-byte response
main_loop: pid 12314 unauthorized for type 8
main_loop: pid 12314 unauthorized for type 8

The obligatory check of the code found:

./nsvs/src/nsvsd/nsvsd.c:
449               /*
450                * Don't allow shadow queries to non-root users
451                * It is CRITICAL that this code occurs AFTER reading the 
key
452                * to avoid clientside sigpipe issues
453                */
454               if ((req.type == GETSPBYNAME || req.type == GETSP) && 
caller.uid != 0)
455                 {
456                   nsvs_log (LOG_DEBUG, "%s: pid %d unauthorized for type 
%d",
457                             __FUNCTION__, caller.pid, req.type);


Having non-root users access the shadow-emulated MySQL information is of no 
consequence to me as I don't use that system for password storage, however 
I'd prefer not to start patching software to work around the above 
restriction. 

Having wasted forty-five minutes typing the above only to find the cause of 
the problem that was illuding me, I'm left with the next problem: 

My network authentication scheme cannot solely rely on Kerberos' built-in 
account expiration and suspension system, because a user could still login 
through SSH using a certificate. As a result, I've tied the entire system's 
account expiration process into the emulated shadow suite so that I can 
expire all access simply by changing the expiration date of the account in 
the MySQL database. Consequently, I can't use the broken_shadow option of 
pam_unix to get around the problem that way, otherwise I wouldn't be able to 
keep someone from modifying the repository without deleting their account or 
changing the account's password. 

Therefore, the only viable option would seem to be to use mod_auth_external 
and an authentication binary to serve as an intermediary between PAM and 
Apache. Unfortunately, that process would be considerably slower than calling 
the PAM functions directly or through a library. I also don't see a SASL 
module for a Apache. 

Does anyone have a suggestion that would fit the requirements, other than 
using mod_auth_external, coupled with an authentication program? An interface 
to Cyrus-SASL would be ideal since I already have saslauthd running for 
Postfix. 

In addition, does anyone have an idea as to what authentication program could 
be used as the aforementioned intermediary?

Thanks.


-- 
Anthony Gorecki
Ectro-Linux Foundation