Category Archives: SSL

mod_nss and NSS Context part 2

I’ve made quite a lot more progress on migrating mod_nss to using NSS contexts, but I’m not quite done yet.

The server starts and handles some requests, so that’s something. It just freaks out a bit if there are multiple nicknames for the “same” virtual host (bug in SNI code) and it is generally unfriendly when it comes to nicknames.

When multiple NSS databases are initialized at the same time the nicknames become quite odd. The “first” database initialized, something probably not predictable in mod_nss, is a traditional database where references to the nickname without a token are handled.

Other databases use the slot in the nickname, e.g. NSS Certificate DB or in the case of mod_nss, internal. I have a huge hack in place now to try both iterations and it works when there are just two NSS databases and uniquely named nicknames. I think that I’ll need to detect duplicate nicks and blow up early otherwise things just don’t work as they should.

I spent an afternoon hacking on gencert to make it more generic so I can more easily generate multiple NSS certificate databases for testing. I started testing with 3 and like I said, it sorta works.

I’ve run into two difficult problems at the moment:

  1. The SNI code is based only on the vhost id so if there are multiple nicknames for a vhost (on different ports for example) then the wrong nickname could be returned
  2. Even if I have the right nickname looking it up using PK11_FindCertFromNickname() is failing to find it. I’ve tried using Server-Cert and internal:Server-Cert and nothing is found. I’m a bit baffled by this and unfortunately will have to dive into gdb. I say unfortunately because IIRC the PKCS#11 handling is a nightmare of callbacks.

Still, the progress has promise.

I need to double check on this but I’m pretty sure that NSS keeps a single trust table internally so my idea of being able to separate CA trust by vhost seems to be a pipe-dream for now.

mod_nss and NSS Contexts

There has been a long standing need to convert mod_nss from using a single NSS database to support NSS Contexts which will allow for much greater flexibility.

I started hacking on it between other work a couple of days ago and progress has been slow to say the least. It’s amazing the number of assumptions baked into the code that there is a single database. So far I’ve had to:

  • Move the database config option from being treated only as a module config option to a proper Apache server context.
  • Move a lot of initialization code around that verifies the NSS database filesystem read permissions and make it iterable
  • Convert nss_pcache from validating the token password to just storing it. I now do a NSS_NoDB_Init() because there could be multiple sources and I don’t want to deal with that headache. This shifts the burden of identifying a bad password a bit later but it should be handled ok

I’ve run into another roadblock similar to the database config option being treated as global: the password dialog is also treated as a global. It really needs to be handled per-server context. And this is where I am.

Fortunately it has mostly been a matter of reorganizing some structures and fixing all uses of them, so it’s been more grunt work than anything else.

On the bright side I’ve got it passing the in-tree tests by specifying a single NSS database. Under the hood it’s using NSS contexts and the new nss_pcache backend.

Something I’ll need to look into is how to release this new code. I’m not 100% sure it isn’t going to blow up on people, depending on how they have things configured. This may also represent mod_nss 2.0, we’ll see. It won’t be much of a difference functionally, with the exception that different VS would be able to have different CA trust, but internally it’ll be quite different.

So stay tuned.

FreeIPA, mod_nss and OCSP

I was messing around today on my IPA master and decided to enable OCSP. I set NSSOCP on in nss.conf and restarted and Apache failed to start after quite a long timeout.

After an embarrassingly long pause myself I figured out what the problem was: the IPA master is itself the OCSP responder and since it is proxies requests to dogtag there is a chicken and egg problem.

I worked around it by using the default responder settings in mod_nss ala:

NSSOCSPDefaultResponder on
NSSOCSPDefaultName ocsp

I also had to add the OCSP signing cert to my Apache database:

# certutil -L -d /var/lib/pki-ca/alias/ -n 'ocspSigningCert cert-pki-ca' -a > /tmp/ocsp.pem
# certutil -A -d /etc/httpd/alias -t C,, -n ocsp -a -i /tmp/ocsp.pem

Now it starts ok. It starts because it is talking directly to the CA instead of trying to talk to itself before it is up.

There is still a rather major problem though: this would fail to start on boot because Apache is configured to start before dogtag. So yet another chicken and egg problem. I’ve no easy solution for this just yet.

As a funny side note, the first time I started httpd after configuring the default responder it still didn’t start because the server cert had been revoked!? I’ve had this toy master around for a while, who knows what I’ve done to it. I resubmitted the cert request using certmonger and got a new cert and then it started just fine.

This also assumes that the CA resides on the local box. It means I don’t need to punch any holes through firewalls to make this port available to my master.

This also wouldn’t survive a renewal of the OCSP signing cert. I’d have to manually re-add the updated cert to the Apache NSS database.


I’m pretty sure my original post was against an IPA 3.x server. I had a 4.2.x server lying around so I double-checked the instructions and it still works but the OCSP port needs to be changed from 9180 to 8080:


NSS and forking

I was reminded recently that if you are going to use NSS as your SSL library then you need to be sure to get all the forking out of the way before you call NSS_Init*. The NSS PKCS#11 loader, including the built-in softokn, has fork detection such that it refused to load if a fork is detected.

I had to bend over backwards to work around this in mod_nss a few years ago.

SubjectAltNames and NSS

NSS is very strict when it comes to validating a certificate hostname. Per RFC 2818 if there is a subjectAltName defined then ONLY the subjectAltName is used to validate the certificate. Based only a little bit of testing, it appears that OpenSSL is a bit more lenient because a certificate that returns an error in NSS worked with s_client.

The error you’d get out of NSS via curl is:

Unable to communicate securely with peer: requested domain name does not match the server's certificate.

So, if you have host and you want to create a certificate to serve that and you’ll need to include two subjectAltName DNSNames in the CSR, one for each host.

With certmonger it might look something like this:

# ipa-getcert request -d /etc/httpd/alias -n Server-Cert -p /etc/httpd/alias/pwdfile.txt -N "" -D "" -K HTTP/ -D ""

Note that this won’t actually work well with IPA by default. You’d also need to tweak ipa-rewrite.conf so the request doesn’t result in a 301 redirect.

Enabling SSL or tls-proxy in devstack

If you want to create an OpenStack environment using devstack with most endpoints protected by SSL there are two ways to do it: native SSL or a TLS proxy (aka an SSL terminator). Both are supported in devstack.

To enable native SSL, add this to your local.conf


To enable via TLS Proxy (stud in this case), add this to your local.conf


This will enable SSL endpoints for:

  • keystone
  • nova
  • cinder
  • glance
  • swift
  • neutron

devstack will generate its own CA certificate and add it to the global trust so all clients on the local machine should just work(tm).

Keystone and HAProxy

I’m trying to get the astapor puppet module (used in the Openstack Foreman Installer and Staypuft) to configure SSL via a proxy. I’m going to use haproxy since it may already be available on the system and it supports SSL termination.

I’m starting with Keystone, as usual, since it is the core of things. Here are some notes from my first crack at doing it manually.

I cheated a bit and used this blog entry to get the basic jist on configuring haproxy for SSL termination. I just copied the default haproxy.cfg to keystone.cfg, deleted the default listeners and added this block:

frontend main *:5000
bind ssl crt /etc/pki/tls/private/combined.pem
default_backend keystone-backend

frontend admin *:35357
bind ssl crt /etc/pki/tls/private/combined.pem
default_backend admin-backend

backend keystone-backend
redirect scheme https if !{ ssl_fc }
server keystone1 check

backend admin-backend
redirect scheme https if !{ ssl_fc }
server admin1 check

I started it with:

# haproxy -f /etc/haproxy/keystone.cfg

And of course it failed because keystone is already listening on those ports. So I left it dead for now. I switched gears and started following my previous blog post on configuring keystone for SSL. The difference is that I just need to create the new secure endpoint, then re-configure keystone.cfg to listen on ports 5001 and 35358 instead.

Note: HAproxy only takes a single option for SSL so you need to concatonate the public cert, private key and CA cert(s) into a single file and use that. When I generate these certs using certmonger I’ll probably end up using a post-save script to do this concatonation.

So I did that, deleted the original keystone endpoint, restart the openstack-keystone service and finally I was able to start up haproxy.

I then fixed my adminrc to use SSL and include OS_CACERT=/path/to/ca and then tried a keystone endpoint-list only to get an SSL failure.

The problem is in python-backports-ssl_match_hostname. The puppet manifests I’m using currently put IP addresses in for everything and I’ve no time or skill to track all that down so I figured I could cheat for a bit and use an IP Address SAN. The problem is that this is explicitly not allowed in match_hostname so the request fails. For now I added some matching code so it works:

if key == 'IP Address':
    if value == hostname:

So with that in place I can now run keystone endpoint-list successfully. I then moved onto the rest of my previous blog on manually converting to secure Keystone and was able to get nova, glance and cinder working. I’m just about ready to fire up a VM at this point.

CA verification and requests

I’ve seen several projects that use requests that try to pass in local CA information. This is fine and generally pretty functional for those that use self-signed certificates, but the fallback when no CA is provided tends to be None. This causes requests to check two environment variables: REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE. If neither is set then you get no CA validation at all which basically dooms the request to failure.

Instead, IMHO, verify should be set to requests.cert.where() if no CA is provided by the client. Really this should be the default in requests.

Adding CAs to the global store is easier than ever and generally a lot easier to handle that copying PEM files all over the place and referencing long paths in potentially multiple configuration files (in the case of OpenStack).

devstack, CA_BUNDLE, requests and pip

In the SSL patches I’m working on for OpenStack in devstack I’m trying to move away relying on client-specific CA file options. There has been pushback from upstream projects on adding new options for every server -> server connection (e.g. glance -> cinder, glance ->  swift, etc).

The system CA bundle was working nicely until I stood up a new dev box. Suddenly I was seeing a bunch of SSL verification errors.

The problem turned out to be requests. I was using the pip-installed requests which uses its own CA bundle by default, rather than the Fedora python-requests package which uses the system bundle in /etc/pki/certs/ca-bundle.crt. The module contains this comment:

If you are packaging Requests, e.g., for a Linux distribution or a managed
environment, you can change the definition of where() to return a separately packaged CA bundle.

We return “/etc/pki/tls/certs/ca-bundle.crt” provided by the ca-certificates package.

So if you are having problems with trust, try installing the distro-specific package. It worked for me.

SSL endpoints for nova, glance and cinder

Continuing on the theme of adding SSL endpoints in OpenStack, lets do a few more. Note that I’m using native SSL here. It is believed that this will suffer from rather bad performance in production . You’ve been warned.

You are going to need to obtain a bunch of SSL server certificates for this to work. It is possible to use the same certificate for each service but it’s bad practice. In my case I’ve used my local IPA server to obtain the certificates, YMMV. Feel free to skip over the IPA parts. In order for this to work with IPA You need to enroll your system(s) with ipa-client-install.

I’m demonstrating this with a packstack installation using nova networking.

Before doing anything, I’d strongly recommend booting an image and ensuring that OpenStack is properly functioning.

Start by securing the Keystone endpoint.

Next add the CA to the global trust.

Let’s start with Cinder.

Create a certificate for us to use. IPA associates a certificate with a service, so we’ll create a service in IPA to store the certificate:

# kinit admin
# ipa service-add cinder/
# ipa-getcert request -f /etc/pki/tls/certs/cinder.crt -k /etc/pki/tls/private/cinder.key -K cinder/

Either way you get the certificate, make sure the cinder use can read the certificate and keys:

# chown cinder /etc/pki/tls/certs/cinder.crt
# chown cinder /etc/pki/tls/private/cinder.key

Find the cinder service endpoints, there will be two. One for the v1 API and one for the v2 API:

# keystone endpoint-list|grep 8776

Delete the existing endpoints:

# keystone endpoint-delete <id>
# keystone endpoint-delete <id>

Now re-create the endpoints using the system FQDN and https:

# keystone endpoint-create --publicurl "" --adminurl "" --internalurl "" --service cinder_v2
# keystone endpoint-create --publicurl "" --adminurl "" --internalurl "" --service cinder

Edit /etc/cinder/cinder.conf to add the SSL options.

ssl_cert_file = /etc/pki/tls/certs/cinder.crt
ssl_key_file = /etc/pki/tls/private/cinder.key

Restart the Cinder API service:

# service openstack-cinder-api restart

Edit /etc/nova/nova.conf to tell it how to talk to Cinder:

cinder_endpoint_template =

Restart the Nova API:

# service openstack-nova-api restart

Test to be sure things still work:

# cinder list
# nova volume-list

Now we move onto the Glance service.

Get a certificate from IPA:

# ipa service-add glance/
# ipa-getcert request -f /etc/pki/tls/certs/glance.crt -k /etc/pki/tls/private/glance.key -K glance/

Fix the permissions on the certificate and key files:

# chown glance /etc/pki/tls/certs/glance.crt
# chown glance /etc/pki/tls/private/glance.key

Find and delete the glance endpoint:

# keystone endpoint-list |grep 9292
# keystone endpoint-delete <id>

And add back the endpoint using the FQDN and https:

# keystone endpoint-create --publicurl --internalurl --adminurl --service glance

Edit /etc/glance/glance-api.conf.

In [DEFAULT] add:

cert_file = /etc/pki/tls/certs/glance.crt
key_file = /etc/pki/tls/private/glance.key

Restart the Glance API service:

# service openstack-glance-api restart

And test that the Glance client works:

# glance image-list

Update Nova to tell it about the secure Glance API. Edit /etc/nova/nova.conf, in [DEFAULT]:


Restart the Nova API:

# service openstack-nova-api restart

And test that Nova can talk to Glance:

# nova image-list

Finally, secure the Nova service (just Nova for now, not EC2 or S3).

Get a certificate for Nova:

# ipa service-add nova/
# ipa-getcert request -f /etc/pki/tls/certs/nova.crt -k /etc/pki/tls/private/nova.key -K nova/

Fix the permissions:

# chown nova /etc/pki/tls/certs/nova.crt
# chown nova /etc/pki/tls/private/nova.key

Find and delete the nova endpoint:

# keystone endpoint-list|grep 8774
# keystone endpoint-delete <id>

Re-create the endpoint with the FQDN and https:

# keystone endpoint-create --publicurl "" --adminurl "" --internalurl "" --service nova

Edit nova.conf, in the [DEFAULT] section:


Restart the Nova API service:

# service openstack-nova-api restart

And finally, verify that Nova works:

# nova list

When I did this, just to be sure, I restarted the world:

# openstack-service restart

Up to this point we’ve only done some very basic validation of each service as we’ve secured them. Now for the real test, fire up a VM:

# nova boot --flavor <flavor> --image <image> ssltest

Make sure you got an address, the image came up, and you can ssh into it.

I’m working on adding this native SSL support, plus via a TLS Proxy, to devstack in bug