Config for httpd as an nfs-client while also handling Kerberos auth of its clients

[jflorian]

The docs do a good job of covering these two use cases independently but I'm struggling to figure out how to achieve both at once. I thought I had it once, but couldn't get back there and think it may have been a fluke of caching or not everything having been restarted. Whenever I get what looks like a good clean test, it seems gssproxy will either only use one of the configs or complain that they have duplicate match rules. I tried to merge the two configs into a monolithic thing I don't quite understand, but it didn't work either.

I don't have any of my configs or logs to show here because I've tried so many combinations. I figure this can't be terribly uncommon so thought I'd ask to see what others have gotten to work. Also, given this is a security mech, I don't really want to do something that works but is insecure. I would greatly appreciate any insight and/or guidance!

[simo5]

It would be necessary, at the very least to know what httpd and nfs configurations you are starting with.

We changed them some over time, so a pointer to the one you have installed on your system would be good.

Secondarily a description of exactly what you want to achieve with HTTPD as an NFS client is also needed. Does not need to delve into proprietary details, just what you expect to happen in general when Http is in the mix.

[jflorian]

TLDR: I may have this working. See last paragraph.

I'm building my web server atop Fedora 35 and have joined the host to my FreeIPA domain.

One goal is to serve HTML content from remote storage that has been mounted with sec=krb5. If I mounted with sec=sys, there was no issue. With sec=krb5 however, httpd logs indicated that the apache user lacked read access even though the file/dir modes would otherwise be fine (as they were with sec=sys).

gssproxy-0.8.4-4.fc35.x86_64 to the rescue! It provides /etc/gssproxy/99-nfs-client.conf:

[service/nfs-client]
  mechs = krb5
  cred_store = keytab:/etc/krb5.keytab
  cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U
  cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab
  cred_usage = initiate
  allow_any_uid = yes
  trusted = yes
  euid = 0

That matches exactly (nice!) what I find at https://github.com/gssapi/gssproxy/blob/main/docs/NFS.md#keytab-based-client-initiation and all makes reasonably good sense to me, so I left this alone. I restarted httpd, gssproxy and sssd thinking that should cover everything and found it to work nicely.

The other goal I want to achieve is Kerberos authentication of the user accessing the same httpd service, say for something private. I took https://github.com/gssapi/gssproxy/blob/main/docs/Apache.md to be perfect for that. So I now have the following config for httpd:

<Location /private>
    AuthType GSSAPI
    AuthName "GSSAPI Login"
    Require valid-user
</Location>

and I couple that with the other steps for adding the service principal, getting the keytab, starting httpd with GSS_USE_PROXY=1. My rpm doesn't include the mentioned 80-httpd.conf, live, example, or otherwise (bummer). So I created my own /etc/gssproxy/80-httpd.conf from the git repo here, so it looks like:

[service/HTTP]
  mechs = krb5
  cred_store = keytab:/etc/gssproxy/http.keytab
  cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
  euid = apache

I restart the same services again, and it all works perfectly. That was a surprise because that's not how it went the first time and I think my attempts at correcting it only made things worse or me more confused. I've rebooted the server and it still works as expected. Maybe much of my problems were only in how I tested. I eventually did gain a new distinct impression that when getting a new keytab from the KDC I must also do a new kinit to renew myself as a client before testing access to services secured with those keytabs. If that is indeed true, perhaps that's where I messed up.

[simo5]

I eventually did gain a new distinct impression that when getting a new keytab from the KDC I must also do a new kinit to renew myself as a client before testing access to services secured with those keytabs. If that is indeed true, perhaps that's where I messed up.

This is indeed required, when you change the principal key you invalidate any ticket that may be outstanding. So using a credential cache obtained before key rotation will lend you errors.

[jflorian]

This is fantastic to know as a certainty rather than a hunch. It actually makes the whole mechanism easier to understand conceptually.

[jflorian]

I would appreciate a review of my keytabs to ensure I didn't do something stupid & insecure. These have been sanitized of specifics. www is a CNAME for webby. I especially question the host principals for the 2nd and 3rd keytabs as I don't think I included those in my earliest (troubled) testing.

# klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 host/[email protected]
   1 host/[email protected]

# klist -k /etc/gssproxy/http.keytab
Keytab name: FILE:/etc/gssproxy/http.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   2 HTTP/[email protected]
   2 HTTP/[email protected]
   1 HTTP/[email protected]
   1 HTTP/[email protected]
   2 HTTP/[email protected]
   2 HTTP/[email protected]
   1 HTTP/[email protected]
   1 HTTP/[email protected]
   1 host/[email protected]
   1 host/[email protected]

# klist -k /var/lib/gssproxy/clients/48.keytab
Keytab name: FILE:/var/lib/gssproxy/clients/48.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 [email protected]
   1 [email protected]
   1 [email protected]
   1 [email protected]
   1 host/[email protected]
   1 host/[email protected]

[simo5]

So you elected to create a principal named apache@ for file access, that is an acceptable solution to allow the httpd process access to the NFS share.
However I do not fully understand why you also added the host keys to that keytab. From a security pov only the gss-proxy should be given access to those keys, so there is no immediate issue of leaking them to the apache process. However an intruder that may gain privileges as the apache user might be able (in certain configurations) to obtain tickets to itself (the "host" principal) as any user, which could then be used to ssh into the host as any user.

Similarly for the http.keytab.

the host/fqdn principal can be considred the "machine key", and it is used by the SSH daemons to authenticate incoming users. It should not be randomly distributed to other services. And access permissions to keytabs containing those keys should be strictly reseved to system components (ssh, gssproxy, root user).

HTH.

[jflorian]

Ah yes, the user principle for the file access was another piece I forgot to mention.
I'm quite certain my thinking for adding host keys to the keytab was because I was getting desperate as to why I struggled for an entire day with only one of the two goals achieved or errors. Sort of a nuke option. Since it's all working and given your gracious and excellently detailed security stance on it, I did the only reasonable thing and busted out kt_util to delete the host/ entries. Those are now gone from everywhere except /etc/krb5.tab and everything is still working.

I honestly can't say why I had so much trouble now, but things are always easier once you've actually seen the thing work. I am most appreciative of the security implications as that really helps my understanding of Kerberos in practice. Thank you so much!

[jflorian]

I'm afraid my reported success has stopped. gssproxy behaves well to authorize me as Kerberos user to httpd but I am no longer to access files served by httpd when they are mounted with sec=krb5. httpd logs:

==> /var/log/httpd/ssl_error_log <==
[Wed Jan 19 15:57:57.597523 2022] [core:error] [pid 1266:tid 1294] (13)Permission denied: [client 192.168.10.12:51968] AH00035: access to /repos/index.html denied (filesystem path '/srv/www/repos/index.html') because search permissions are missing on a component of the path
[Wed Jan 19 15:57:57.815278 2022] [core:error] [pid 1266:tid 1294] (13)Permission denied: [client 192.168.10.12:51968] AH00035: access to /repos/index.php denied (filesystem path '/srv/www/repos/index.php') because search permissions are missing on a component of the path

repos/ is just a directory I'm serving with Options Indexes MultiViews FollowSymLinks. I'm not sure why the latter is attempted, but I have MediaWiki on the same server and with it PHP, of course. Anyway, neither file exists but a normal directory listing would be served if this directory was mounted instead with sec=sys. In any event, here are the relevant permissions:

# namei -l /srv/www/repos/
f: /srv/www/repos/
dr-xr-xr-x root   root   /
drwxr-xr-x root   root   srv
drwxr-xr-x root   root   www
drwxr-xr-x apache apache repos

I think I caught the problem in the gssproxy logs, however:

Jan 19 16:17:24 webby.example.com gssproxy[1964]: [2022/01/19 21:17:24]: Client [2022/01/19 21:17:24]: (/usr/sbin/rpc.gssd) [2022/01/19 21:17:24]:  connected (fd = 12)[2022/01/19 21:17:24]:  (pid = 759) (uid = 48) (gid = 48)[2022/01/19 21:17:24]:  (context = system_u:system_r:gssd_t:s0)[2022/01/19 21:17:24]:
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022/01/19 21:17:24]: Connection matched service HTTP
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022/01/19 21:17:24]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48,socket: (null)
Jan 19 16:17:24 webby.example.com gssproxy[1964]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "HTTP/[email protected]" [ { "HTTP/[email protected]" { 1 2 840 113554 1 2 2 } ACCEPT 0 4294967295 } ] [ u..U..bm....MHR.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 )
Jan 19 16:17:24 webby.example.com gssproxy[1964]:     GSSX_RES_ACQUIRE_CRED( status: { 720896 { 1 2 840 113554 1 2 2 } 100001 "The referenced credential has expired" "Success" [  ] } output_cred_handle: { "" [ ] [  ] 0 } )
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022/01/19 21:17:24]: Connection matched service HTTP
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022/01/19 21:17:24]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48,socket: (null)
Jan 19 16:17:24 webby.example.com gssproxy[1964]:     GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [  ] } input_cred_handle: { "HTTP/[email protected]" [ { "HTTP/[email protected]" { 1 2 840 113554 1 2 2 } ACCEPT 0 4294967295 } ] [ u..U..bm....MHR.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 )
Jan 19 16:17:24 webby.example.com gssproxy[1964]:     GSSX_RES_ACQUIRE_CRED( status: { 720896 { 1 2 840 113554 1 2 2 } 100001 "The referenced credential has expired" "Success" [  ] } output_cred_handle: { "" [ ] [  ] 0 } )

The part that catches my eye is that it's matching on the service HTTP when I would expect nfs-client here instead. If I'm right, that means httpd would be using the keys for the service principles (HTTP/www.example.com, etc.) instead of the apache user principle key. On that hunch, I thought I might be able to better constrain the service by adding program = /usr/sbin/httpd to 80-httpd.conf. After I did that, I once again restarted gssproxy, httpd and sssd and that looks to have resolved the problem.

However.... I can take that line back out, restart the services and still access the repos/ directory when I would have expected it to fail once again. I believe I want that line, but I don't understand my process to confirm that I do. Short of experts here, that is.

[simo5]

That's because you are using the same ccache file in both configurations:
cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
So once yo obtain tickets ,GSSAPI will reuse valid tickets until they expire.

You should probably use disjointed caches like:
cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_nfs_%U
vs
cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_httpd_%U

[jflorian]

Ok, I can imagine how that might create issues. I've revised as you suggested and so far so good. Would it make sense to update the example documentation to steer others away from my mistake?

[simo5]

I would definitely accept help in that direction.
Either a pull request or proposed text.

[jflorian]

I'll work on a PR tomorrow. It's the least I can do after so much helpful guidance here.