Category Archives: OpenStack

Enabling SSL or tls-proxy in devstack

If you want to create an OpenStack environment using devstack with most endpoints protected by SSL there are two ways to do it: native SSL or a TLS proxy (aka an SSL terminator). Both are supported in devstack.

To enable native SSL, add this to your local.conf

USE_SSL=TRUE

To enable via TLS Proxy (stud in this case), add this to your local.conf

ENABLED_SERVICES+=,tls-proxy

This will enable SSL endpoints for:

  • keystone
  • nova
  • cinder
  • glance
  • swift
  • neutron

devstack will generate its own CA certificate and add it to the global trust so all clients on the local machine should just work(tm).

Keystone and HAProxy

I’m trying to get the astapor puppet module (used in the Openstack Foreman Installer and Staypuft) to configure SSL via a proxy. I’m going to use haproxy since it may already be available on the system and it supports SSL termination.

I’m starting with Keystone, as usual, since it is the core of things. Here are some notes from my first crack at doing it manually.

I cheated a bit and used this blog entry to get the basic jist on configuring haproxy for SSL termination. I just copied the default haproxy.cfg to keystone.cfg, deleted the default listeners and added this block:

frontend main *:5000
bind 192.168.0.4:5000 ssl crt /etc/pki/tls/private/combined.pem
default_backend keystone-backend

frontend admin *:35357
bind 192.168.0.4:35357 ssl crt /etc/pki/tls/private/combined.pem
default_backend admin-backend

backend keystone-backend
redirect scheme https if !{ ssl_fc }
server keystone1 192.168.0.4:5001 check

backend admin-backend
redirect scheme https if !{ ssl_fc }
server admin1 192.168.0.4:35358 check

I started it with:

# haproxy -f /etc/haproxy/keystone.cfg

And of course it failed because keystone is already listening on those ports. So I left it dead for now. I switched gears and started following my previous blog post on configuring keystone for SSL. The difference is that I just need to create the new secure endpoint, then re-configure keystone.cfg to listen on ports 5001 and 35358 instead.

Note: HAproxy only takes a single option for SSL so you need to concatonate the public cert, private key and CA cert(s) into a single file and use that. When I generate these certs using certmonger I’ll probably end up using a post-save script to do this concatonation.

So I did that, deleted the original keystone endpoint, restart the openstack-keystone service and finally I was able to start up haproxy.

I then fixed my adminrc to use SSL and include OS_CACERT=/path/to/ca and then tried a keystone endpoint-list only to get an SSL failure.

The problem is in python-backports-ssl_match_hostname. The puppet manifests I’m using currently put IP addresses in for everything and I’ve no time or skill to track all that down so I figured I could cheat for a bit and use an IP Address SAN. The problem is that this is explicitly not allowed in match_hostname so the request fails. For now I added some matching code so it works:

if key == 'IP Address':
    if value == hostname:
        return

So with that in place I can now run keystone endpoint-list successfully. I then moved onto the rest of my previous blog on manually converting to secure Keystone and was able to get nova, glance and cinder working. I’m just about ready to fire up a VM at this point.

devstack, CA_BUNDLE, requests and pip

In the SSL patches I’m working on for OpenStack in devstack I’m trying to move away relying on client-specific CA file options. There has been pushback from upstream projects on adding new options for every server -> server connection (e.g. glance -> cinder, glance ->  swift, etc).

The system CA bundle was working nicely until I stood up a new dev box. Suddenly I was seeing a bunch of SSL verification errors.

The problem turned out to be requests. I was using the pip-installed requests which uses its own CA bundle by default, rather than the Fedora python-requests package which uses the system bundle in /etc/pki/certs/ca-bundle.crt. The requests.certs.py module contains this comment:

If you are packaging Requests, e.g., for a Linux distribution or a managed
environment, you can change the definition of where() to return a separately packaged CA bundle.

We return “/etc/pki/tls/certs/ca-bundle.crt” provided by the ca-certificates package.

So if you are having problems with trust, try installing the distro-specific package. It worked for me.

SSL endpoints for nova, glance and cinder

Continuing on the theme of adding SSL endpoints in OpenStack, lets do a few more. Note that I’m using native SSL here. It is believed that this will suffer from rather bad performance in production . You’ve been warned.

You are going to need to obtain a bunch of SSL server certificates for this to work. It is possible to use the same certificate for each service but it’s bad practice. In my case I’ve used my local IPA server to obtain the certificates, YMMV. Feel free to skip over the IPA parts. In order for this to work with IPA You need to enroll your system(s) with ipa-client-install.

I’m demonstrating this with a packstack installation using nova networking.

Before doing anything, I’d strongly recommend booting an image and ensuring that OpenStack is properly functioning.

Start by securing the Keystone endpoint.

Next add the CA to the global trust.

Let’s start with Cinder.

Create a certificate for us to use. IPA associates a certificate with a service, so we’ll create a service in IPA to store the certificate:

# kinit admin
# ipa service-add cinder/set3client2.example.com
# ipa-getcert request -f /etc/pki/tls/certs/cinder.crt -k /etc/pki/tls/private/cinder.key -K cinder/set3client2.example.com

Either way you get the certificate, make sure the cinder use can read the certificate and keys:

# chown cinder /etc/pki/tls/certs/cinder.crt
# chown cinder /etc/pki/tls/private/cinder.key

Find the cinder service endpoints, there will be two. One for the v1 API and one for the v2 API:

# keystone endpoint-list|grep 8776

Delete the existing endpoints:

# keystone endpoint-delete <id>
# keystone endpoint-delete <id>

Now re-create the endpoints using the system FQDN and https:

# keystone endpoint-create --publicurl "https://set3client2.example.com:8776/v2/%(tenant_id)s" --adminurl "https://set3client2.example.com:8776/v2/%(tenant_id)s" --internalurl "https://set3client2.example.com:8776/v2/%(tenant_id)s" --service cinder_v2
# keystone endpoint-create --publicurl "https://set3client2.example.com:8776/v1/%(tenant_id)s" --adminurl "https://set3client2.example.com:8776/v1/%(tenant_id)s" --internalurl "https://set3client2.example.com:8776/v1/%(tenant_id)s" --service cinder

Edit /etc/cinder/cinder.conf to add the SSL options.

[DEFAULT]
ssl_cert_file = /etc/pki/tls/certs/cinder.crt
ssl_key_file = /etc/pki/tls/private/cinder.key

Restart the Cinder API service:

# service openstack-cinder-api restart

Edit /etc/nova/nova.conf to tell it how to talk to Cinder:

[DEFAULT]
cinder_endpoint_template = https://set3client2.example.com:8776/v1/%(project_id)s
cinder_ca_certificates_file=/etc/ipa/ca.crt

Restart the Nova API:

# service openstack-nova-api restart

Test to be sure things still work:

# cinder list
# nova volume-list

Now we move onto the Glance service.

Get a certificate from IPA:

# ipa service-add glance/set3client2.example.com
# ipa-getcert request -f /etc/pki/tls/certs/glance.crt -k /etc/pki/tls/private/glance.key -K glance/set3client2.example.com

Fix the permissions on the certificate and key files:

# chown glance /etc/pki/tls/certs/glance.crt
# chown glance /etc/pki/tls/private/glance.key

Find and delete the glance endpoint:

# keystone endpoint-list |grep 9292
# keystone endpoint-delete <id>

And add back the endpoint using the FQDN and https:

# keystone endpoint-create --publicurl https://set3client2.example.com:9292 --internalurl https://set3client2.example.com:9292 --adminurl https://set3client2.example.com:9292 --service glance

Edit /etc/glance/glance-api.conf.

In [DEFAULT] add:

cert_file = /etc/pki/tls/certs/glance.crt
key_file = /etc/pki/tls/private/glance.key

Restart the Glance API service:

# service openstack-glance-api restart

And test that the Glance client works:

# glance image-list

Update Nova to tell it about the secure Glance API. Edit /etc/nova/nova.conf, in [DEFAULT]:

glance_api_servers=https://set3client2.example.com:9292

Restart the Nova API:

# service openstack-nova-api restart

And test that Nova can talk to Glance:

# nova image-list

Finally, secure the Nova service (just Nova for now, not EC2 or S3).

Get a certificate for Nova:

# ipa service-add nova/set3client2.example.com
# ipa-getcert request -f /etc/pki/tls/certs/nova.crt -k /etc/pki/tls/private/nova.key -K nova/set3client2.example.com

Fix the permissions:

# chown nova /etc/pki/tls/certs/nova.crt
# chown nova /etc/pki/tls/private/nova.key

Find and delete the nova endpoint:

# keystone endpoint-list|grep 8774
# keystone endpoint-delete <id>

Re-create the endpoint with the FQDN and https:

# keystone endpoint-create --publicurl "https://set3client2.example.com:8774/v2/%(tenant_id)s" --adminurl "https://set3client2.example.com:8774/v2/%(tenant_id)s" --internalurl "https://set3client2.example.com:8774/v2/%(tenant_id)s" --service nova

Edit nova.conf, in the [DEFAULT] section:

ssl_cert_file=/etc/pki/tls/certs/nova.crt
ssl_key_file=/etc/pki/tls/private/nova.key
...
enabled_ssl_apis=osapi_compute

Restart the Nova API service:

# service openstack-nova-api restart

And finally, verify that Nova works:

# nova list

When I did this, just to be sure, I restarted the world:

# openstack-service restart

Up to this point we’ve only done some very basic validation of each service as we’ve secured them. Now for the real test, fire up a VM:

# nova boot --flavor <flavor> --image <image> ssltest

Make sure you got an address, the image came up, and you can ssh into it.

I’m working on adding this native SSL support, plus via a TLS Proxy, to devstack in bug https://bugs.launchpad.net/devstack/+bug/1328226

SSL CA in devstack

I’ve  been trying to configure devstack to install SSL-enabled endpoints. This is generally straightforward but hampered by a several bugs related to server to server communication (e.g. nova to glance) where the are no options to specify the location of the issuing CA.

One workaround I’m looking at is rather than passing the CA around as a path everywhere is to add it to the system CA bundle and let the client libraries handle things. This is less-than-ideal but seems to work. I’m a Fedora guy so I use the new CA trust commands:

# cp /path/to/cacert.pem /etc/pki/ca-trust/source/anchors

OR

# cp /path/to/cacert.pem /usr/share/pki/ca-trust-source
# update-ca-trust extract
The difference between the directories is low or high priority to the CA trust tool. I’m honestly not entirely sure what that means, but the low priority version works for me.
Strangely I still need to specify the CA for some things. I haven’t yet figured that one out yet, but doing this I was able to get a vanilla OpenStack install via devstack with nova, keystone, glance and cinder secured with SSL.
The Fedora feature is documented at http://fedoraproject.org/wiki/Features/SharedSystemCertificates

Configure Keystone to use SSL in OpenStack

It is possible to convert an existing OpenStack installation to use SSL for Keystone and other services. I’ll tackle those later.

BEWARE: No Keystone == No OpenStack, so tread carefully as you can easily hose your installation.

In my case I installed OpenStack Havana using RDO on RHEL 6.5 I used packstack. I suspect that the configuration is largely similar to other versions as well:

# packstack --gen-answers-file=/root/answers.txt

In my case I modified answers.txt and set CONFIG_NEUTRON_INSTALL to n. I keep things rather simple in my environment.

Next i installed OpenStack using packstack:

# packstack --answer-file=/root/answers.txt

Go grab a cup of coffee, it takes a while to download and configure all the necessary bits.

Assuming you are following along at home, at this point I’d take the time to confirm that things are indeed working by launching a nova instance.

You need to start by figuring out which interface you are going to talk over. One was picked for you by packstack, that is probably the right one. I’m configuring a private cloud so I’m going to use the same name everywhere. If you are going to contact this over multiple interfaces you’ll need to look into getting a certificate with a Subject Alt Name, or taking the lazy way out and getting a wildcard cert.

In my case I’m doing a simple private cloud so I’m going to use the system FQDN, in my case set2client1.example.com. I use vftool to simplify setting up hosts for testing.

So first you need to ensure that the system hostname is the FQDN:

# hostname set2client1.example.com

Now get a certificate for this hostname from your favorite source. I enrolled the machine into my IPA infrastructure and used ipa-getcert. In my case I put the result into /etc/pki/tls/certs/keystone.crt and /etc/pki/tls/private/keystone.key respectively. Be sure to set the ownership to keystone:

# chown keystone /etc/pki/tls/certs/keystone.crt
# chown keystone /etc/pki/tls/private/keystone.key

Now we’re ready to begin securing things. This is the tricky part. We need to create a new endpoint for SSL, delete the old one, then restart and things should work (see BEWARE above).

First we find the current keystone endpoint:

# keystone endpoint-list|grep 5000

Next we create a new one, using our FQDN:

# keystone endpoint-create --publicurl https://set2client1.example.com:5000/v2.0 --internalurl https://set2client1.example.com:5000/v2.0 --adminurl https://set2client1.example.com:35357/v2.0 --service keystone

Delete the original endpoint:

# keystone endpoint-delete <endpoint-id>

Edit /etc/keystone/keystone.conf to add the SSL options. It should look like this:

[ssl]
enable = True
certfile = /etc/pki/tls/certs/keystone.crt
keyfile = /etc/pki/tls/private/keystone.key

Now restart the keystone service

# service openstack-keystone restart

Fix up the environment to match our new configuration:

# export export OS_AUTH_URL=https://set2client1.example.com:35357/v2.0/
# export OS_CACERT=/path/to/ca.crt

And finally, test to be sure basic things work:

# keystone endpoint-list

Assuming that was successful we can move onto the other services. You can restart these individually afterwards but I just use openstack-service restart to restart the entire world after configuring all of the services.

Edit /etc/nova/nova.conf, it should look something like this:

[keystone_authtoken]
auth_protocol = https
auth_port = 35357
auth_host = set2client1.example.com
auth_uri = https://set2client1.example.com:5000/v2.0
cafile = /path/to/ca.crt

Edit /etc/glance/glance-api.conf to look like:

[keystone_authtoken]
auth_protocol = https
auth_port = 35357
auth_host = set2client1.example.com
auth_uri = https://set2client1.example.com:5000/v2.0
cafile = /path/to/ca.crt

Edit /etc/glance/glance-registry.conf to look like:

[keystone_authtoken]
auth_host=set2client1.example.com
auth_port=35357
auth_protocol=https
auth_uri=https://set2client1.example.com:5000/v2.0
cafile = /etc/ipa/ca.crt

Edit /etc/cinder/cinder.conf to look like:

[keystone_authtoken]
auth_host=set2client1.example.com
service_port=5000
auth_uri=https://set2client1.example.com:5000/v2.0
auth_port=35357
service_host=set2client1.example.com
service_protocol=https
auth_protocol=https
cafile=/etc/ipa/ca.crt

Edit /etc/cinder/api-paste.ini to look like:

[filter:authtoken]
auth_host=set2client1.example.com
service_port=5000
auth_uri=https://set2client1.example.com:5000/v2.0
auth_port=35357
service_host=set2client1.example.com
service_protocol=https
auth_protocol=https
cafile=/etc/ipa/ca.crt

Edit /etc/neutron/neutron.conf to look like this, but note that I installed without Neutron networking in my case so this is quite untested, but it follows the pattern:

[keystone_authtoken]
auth_host = set2client1.example.com
auth_port = 35357
auth_protocol = https
auth_uri=https://set2client1.example.com:5000/v2.0
cafile=/etc/ipa/ca.crt

Edit /etc/neutron/api-paste.ini

[filter:authtoken]
auth_port=35357
auth_protocol=https
auth_uri=https://set2client1.example.com:5000/v2.0
auth_host=set2client1.example.com
cafile=/etc/ipa/ca.crt

Edit /etc/neutron/metadata_agent.ini

[DEFAULT]
auth_url = https://set2client1.example.com:35357/v2.0

Similarly I didn’t have anything stored in swift, but this should be correct.  Edit /etc/swift/proxy-server.conf

[filter:authtoken]
auth_host = set2client1.example.com
auth_port = 35357
auth_protocol = https
auth_uri = https://set2client1.example.com:5000/v2.0
cafile = /etc/ipa/ca.crt

And finally edit /etc/ceilometer/ceilometer.conf:

[service_credentials]
auth_host=set2client1.example.com
auth_port = 35357
auth_protocol=https
auth_uri=https://set2client1.example.com:5000/v2.0
cafile=/etc/ipa/ca.crt

Now restart the world and test things out:

# openstack-service restart
# openstack-status

In case you are unfamiliar with it, openstack-status will show the status of the current running services and do a whole lot of connection testing for you.

That’s basically it. I fired off a new nova instance to afterward to triple check as that exercises all sorts of services.

Finally, reconfigure Horizon to use the secure Keystone endpoint

Edit /etc/openstack-dashboard/local_settings:

OPENSTACK_HOST = "set2client1.example.com"
OPENSTACK_KEYSTONE_URL = "https://%s:5000/v2.0" % OPENSTACK_HOST

Restart Apache

# service httpd restart