SSL endpoints for nova, glance and cinder

Continuing on the theme of adding SSL endpoints in OpenStack, lets do a few more. Note that I’m using native SSL here. It is believed that this will suffer from rather bad performance in production . You’ve been warned.

You are going to need to obtain a bunch of SSL server certificates for this to work. It is possible to use the same certificate for each service but it’s bad practice. In my case I’ve used my local IPA server to obtain the certificates, YMMV. Feel free to skip over the IPA parts. In order for this to work with IPA You need to enroll your system(s) with ipa-client-install.

I’m demonstrating this with a packstack installation using nova networking.

Before doing anything, I’d strongly recommend booting an image and ensuring that OpenStack is properly functioning.

Start by securing the Keystone endpoint.

Next add the CA to the global trust.

Let’s start with Cinder.

Create a certificate for us to use. IPA associates a certificate with a service, so we’ll create a service in IPA to store the certificate:

# kinit admin
# ipa service-add cinder/set3client2.example.com
# ipa-getcert request -f /etc/pki/tls/certs/cinder.crt -k /etc/pki/tls/private/cinder.key -K cinder/set3client2.example.com

Either way you get the certificate, make sure the cinder use can read the certificate and keys:

# chown cinder /etc/pki/tls/certs/cinder.crt
# chown cinder /etc/pki/tls/private/cinder.key

Find the cinder service endpoints, there will be two. One for the v1 API and one for the v2 API:

# keystone endpoint-list|grep 8776

Delete the existing endpoints:

# keystone endpoint-delete <id>
# keystone endpoint-delete <id>

Now re-create the endpoints using the system FQDN and https:

# keystone endpoint-create --publicurl "https://set3client2.example.com:8776/v2/%(tenant_id)s" --adminurl "https://set3client2.example.com:8776/v2/%(tenant_id)s" --internalurl "https://set3client2.example.com:8776/v2/%(tenant_id)s" --service cinder_v2
# keystone endpoint-create --publicurl "https://set3client2.example.com:8776/v1/%(tenant_id)s" --adminurl "https://set3client2.example.com:8776/v1/%(tenant_id)s" --internalurl "https://set3client2.example.com:8776/v1/%(tenant_id)s" --service cinder

Edit /etc/cinder/cinder.conf to add the SSL options.

[DEFAULT]
ssl_cert_file = /etc/pki/tls/certs/cinder.crt
ssl_key_file = /etc/pki/tls/private/cinder.key

Restart the Cinder API service:

# service openstack-cinder-api restart

Edit /etc/nova/nova.conf to tell it how to talk to Cinder:

[DEFAULT]
cinder_endpoint_template = https://set3client2.example.com:8776/v1/%(project_id)s
cinder_ca_certificates_file=/etc/ipa/ca.crt

Restart the Nova API:

# service openstack-nova-api restart

Test to be sure things still work:

# cinder list
# nova volume-list

Now we move onto the Glance service.

Get a certificate from IPA:

# ipa service-add glance/set3client2.example.com
# ipa-getcert request -f /etc/pki/tls/certs/glance.crt -k /etc/pki/tls/private/glance.key -K glance/set3client2.example.com

Fix the permissions on the certificate and key files:

# chown glance /etc/pki/tls/certs/glance.crt
# chown glance /etc/pki/tls/private/glance.key

Find and delete the glance endpoint:

# keystone endpoint-list |grep 9292
# keystone endpoint-delete <id>

And add back the endpoint using the FQDN and https:

# keystone endpoint-create --publicurl https://set3client2.example.com:9292 --internalurl https://set3client2.example.com:9292 --adminurl https://set3client2.example.com:9292 --service glance

Edit /etc/glance/glance-api.conf.

In [DEFAULT] add:

cert_file = /etc/pki/tls/certs/glance.crt
key_file = /etc/pki/tls/private/glance.key

Restart the Glance API service:

# service openstack-glance-api restart

And test that the Glance client works:

# glance image-list

Update Nova to tell it about the secure Glance API. Edit /etc/nova/nova.conf, in [DEFAULT]:

glance_api_servers=https://set3client2.example.com:9292

Restart the Nova API:

# service openstack-nova-api restart

And test that Nova can talk to Glance:

# nova image-list

Finally, secure the Nova service (just Nova for now, not EC2 or S3).

Get a certificate for Nova:

# ipa service-add nova/set3client2.example.com
# ipa-getcert request -f /etc/pki/tls/certs/nova.crt -k /etc/pki/tls/private/nova.key -K nova/set3client2.example.com

Fix the permissions:

# chown nova /etc/pki/tls/certs/nova.crt
# chown nova /etc/pki/tls/private/nova.key

Find and delete the nova endpoint:

# keystone endpoint-list|grep 8774
# keystone endpoint-delete <id>

Re-create the endpoint with the FQDN and https:

# keystone endpoint-create --publicurl "https://set3client2.example.com:8774/v2/%(tenant_id)s" --adminurl "https://set3client2.example.com:8774/v2/%(tenant_id)s" --internalurl "https://set3client2.example.com:8774/v2/%(tenant_id)s" --service nova

Edit nova.conf, in the [DEFAULT] section:

ssl_cert_file=/etc/pki/tls/certs/nova.crt
ssl_key_file=/etc/pki/tls/private/nova.key
...
enabled_ssl_apis=osapi_compute

Restart the Nova API service:

# service openstack-nova-api restart

And finally, verify that Nova works:

# nova list

When I did this, just to be sure, I restarted the world:

# openstack-service restart

Up to this point we’ve only done some very basic validation of each service as we’ve secured them. Now for the real test, fire up a VM:

# nova boot --flavor <flavor> --image <image> ssltest

Make sure you got an address, the image came up, and you can ssh into it.

I’m working on adding this native SSL support, plus via a TLS Proxy, to devstack in bug https://bugs.launchpad.net/devstack/+bug/1328226

SSL CA in devstack

I’veĀ  been trying to configure devstack to install SSL-enabled endpoints. This is generally straightforward but hampered by a several bugs related to server to server communication (e.g. nova to glance) where the are no options to specify the location of the issuing CA.

One workaround I’m looking at is rather than passing the CA around as a path everywhere is to add it to the system CA bundle and let the client libraries handle things. This is less-than-ideal but seems to work. I’m a Fedora guy so I use the new CA trust commands:

# cp /path/to/cacert.pem /etc/pki/ca-trust/source/anchors

OR

# cp /path/to/cacert.pem /usr/share/pki/ca-trust-source
# update-ca-trust extract
The difference between the directories is low or high priority to the CA trust tool. I’m honestly not entirely sure what that means, but the low priority version works for me.
Strangely I still need to specify the CA for some things. I haven’t yet figured that one out yet, but doing this I was able to get a vanilla OpenStack install via devstack with nova, keystone, glance and cinder secured with SSL.
The Fedora feature is documented at http://fedoraproject.org/wiki/Features/SharedSystemCertificates