Peer certificate cannot be authenticated with given CA certificates

I’ve seen this twice now so I’m documenting the fix so I don’t have to go hunting again.

Basically it starts as a typical “oh crap my certs expired” question on #freeipa or freeipa-users. Sadly the usual things don’t seem to help (go back in time).

The last time this happened there was the added twist that the renewal master was gone so we had to first reconfigure a replica to do the renewal (you do have more than one CA right? RIGHT?)

Anyway, he would persistently get the Peer certificate cannot be authenticated message. We tried:

  • Confirming that ipaCert was the correct value in the IPA RA entry in LDAP
  • The CA is up and running: curl --cacert /etc/ipa/ca.crt -v https://`hostname`:8443/ca/ee/ca/getCertChain
  • Ensuring that certutil -L -d /etc/pki/pki-tomcat/alias-n 'caSigningCert cert-pki-ca' -a and cat /etc/ipa/ca.crt are the same cert

Then I remembered a fellow Red Hatter had reported a similar issue and discovered that the fix was to reset the NSS trust flags in the Apache NSS database (which certmonger uses).

# SSL_DIR=/etc/httpd/alias/ curl -v -o /dev/null --cacert /etc/ipa/ca.crt https://`hostname`:8443/ca/agent/ca/profileReview

You should get client certificate not found. If you don’t then try this:

# certutil -M -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' -t ,,
# certutil -M -d /etc/httpd/alias -n 'EXAMPLE.COM IPA CA' -t CT,C,C


Setting up AD for winsync testing

It had literally been years since I had to setup an AD test environment to do basic winsync testing. I found some scraggly notes and decided to transcribe them here for posterity. They were written for AD 2003 and things for 2008 are a bit different but I still found it fairly easy to figure out (in 2008 there is less need to go to the Start menu).

I don’t in fact remember what a lot of these notes do so don’t kill the messenger.

Start with an AD 2008 instance by following

Once that is booted:

  1. Change the hostname
  2. My Computer -> right click -> Properties -> Computer Name -> Change = win2003
  4. Manage your Server
    1. Add or remove a role -> Next [Preliminary Steps]
    2. Custom -> Domain Controller
    3. Domain controller for a new domain
    4. Domain in a new forest
    5. Fill DNS name for new domain:
  5. If conflict select Install and Configure DNS on this server
  7. Start -> Control Panel -> Add or Remove Programs
    1. Add/Remove Windows Components
    2. Certificate Services, yes to the question
    3. Next
    4. Enterprise root CA
    5. AD CA for the common name
    6. Accept other defaults
    7. Ok about IIS
  8. REBOOT (or wait a little while for certs to issue)
  9. Start -> Admin Tools -> Certificate Authority
    1. Certificate Authority -> AD CA -> Issued Certificates
    2. Select the cert, double click
    3. Certificate Path
    4. Select AD CA, view certificate
    5. Details
    6. Copy to file
    7. Base 64-encoded x509 (.cer)
  10. Install WinSCP
  11. Copy cert to IPA

Now on the IPA master the agreement can be created:

# ipa-replica-manage connect –winsync –cacert=/home/rcrit/adca.cer -v –no-lookup –binddn ‘cn=administrator,cn=users,dc=example,dc=com’ –bindpw <AD pw> –passsync <something>

As I recall I tended to put the AD hostname into /etc/hosts (hence the –no-lookup).

IPA configuration files and context

There are times when you may want more information out of the IPA server logs. I’ve seen people suggest adding debug = True to /etc/ipa/default.conf. This is fine (and it works) but it enables debugging in both the client and the server which can be annoying for command-line users.

What I do instead is create /etc/ipa/server.conf containing:

debug = True

The context that is set during initialization drives what configuration files are loaded so only the server will load this so the client remains quiet by default.

When the context is set during api.initialize it sets api.env.context. The original idea is this could drive different code paths depending on the context but in reality it hasn’t been used all that often. Being able to load context-specific configuration files is pretty neat though.

oslo messaging and notifications

novajoin needs to monitor nova notifications to know when an instance is deleted so the host can be removed from IPA as well. I originally coded it to use the notifications topic but ceilometer also uses this topic so novajoin-notify was getting only a subset of the deletes.

The fix is very easy, add a new topic to the topics option in nova.conf, e.g.


This isn’t at all obvious from any documentation.

novajoin and the 2016 OpenStack Barcelona Summit

The nova v2 vendordata (dynamic vendordata plugins) came up at the 2016 OpenStack Summit in Barcelona. It was expected at the outset to be a fairly short meeting since the two proposals were seemingly straightfoward: add authentication and caching. It ended up using all 40 minutes but in the end wasn’t all that contentious.

It was decided that authentication would be sent in all cases to the configured vendordata plugins. There was some brief discussion about whether the user token would be included but as I took it, nova would use its own (or some pre-configured) credential. This will require some paste configuration changes in novajoin but authentication should otherwise be supported.

The metadata server will also cache responses. Exactly how long, what, etc is TBD. If I remember I’ll update this post with the gerrit review link once it comes out.

The idea is to commit to master than back port to Newton stable.

The contentious part was related to that user token I mentioned. Adam Young from Keystone wanted that token to be sent along, even if expired, so one could know the roles of the user that kicked things off. The problem of course is that the user is just a snapshot in time. Roles change. Users are deleted. Apparently some users completely hammer on the metadata service today, some as frequently as every few minutes. At some point things could break if that user went away.

I was ambivalent about it. Adam’s point was that it could be used for access control which is a good idea. I think that if the roles were cached instead of the user, that might make more sense. But even then people would complain that they revoked or added access and the user can/can’t do things. It’s a no-win I think. I kept my mouth shut in any case.

In the end this is good news for novajoin. I was quite uncomfortable having unauthenticated requests at all (e.g. metadata requests from an instance) so that’ll go away soon.

The caching will solve the problems I had bending over backwards with the IPA OTP. There could still be problems if the time to enroll the instance > nova metadata cache so I’ll probably leave in my “last update wins” code, but this does make things a bit more predictable and will certainly be faster.

mod_nss and NSS Context part 2

I’ve made quite a lot more progress on migrating mod_nss to using NSS contexts, but I’m not quite done yet.

The server starts and handles some requests, so that’s something. It just freaks out a bit if there are multiple nicknames for the “same” virtual host (bug in SNI code) and it is generally unfriendly when it comes to nicknames.

When multiple NSS databases are initialized at the same time the nicknames become quite odd. The “first” database initialized, something probably not predictable in mod_nss, is a traditional database where references to the nickname without a token are handled.

Other databases use the slot in the nickname, e.g. NSS Certificate DB or in the case of mod_nss, internal. I have a huge hack in place now to try both iterations and it works when there are just two NSS databases and uniquely named nicknames. I think that I’ll need to detect duplicate nicks and blow up early otherwise things just don’t work as they should.

I spent an afternoon hacking on gencert to make it more generic so I can more easily generate multiple NSS certificate databases for testing. I started testing with 3 and like I said, it sorta works.

I’ve run into two difficult problems at the moment:

  1. The SNI code is based only on the vhost id so if there are multiple nicknames for a vhost (on different ports for example) then the wrong nickname could be returned
  2. Even if I have the right nickname looking it up using PK11_FindCertFromNickname() is failing to find it. I’ve tried using Server-Cert and internal:Server-Cert and nothing is found. I’m a bit baffled by this and unfortunately will have to dive into gdb. I say unfortunately because IIRC the PKCS#11 handling is a nightmare of callbacks.

Still, the progress has promise.

I need to double check on this but I’m pretty sure that NSS keeps a single trust table internally so my idea of being able to separate CA trust by vhost seems to be a pipe-dream for now.

mod_nss and NSS Contexts

There has been a long standing need to convert mod_nss from using a single NSS database to support NSS Contexts which will allow for much greater flexibility.

I started hacking on it between other work a couple of days ago and progress has been slow to say the least. It’s amazing the number of assumptions baked into the code that there is a single database. So far I’ve had to:

  • Move the database config option from being treated only as a module config option to a proper Apache server context.
  • Move a lot of initialization code around that verifies the NSS database filesystem read permissions and make it iterable
  • Convert nss_pcache from validating the token password to just storing it. I now do a NSS_NoDB_Init() because there could be multiple sources and I don’t want to deal with that headache. This shifts the burden of identifying a bad password a bit later but it should be handled ok

I’ve run into another roadblock similar to the database config option being treated as global: the password dialog is also treated as a global. It really needs to be handled per-server context. And this is where I am.

Fortunately it has mostly been a matter of reorganizing some structures and fixing all uses of them, so it’s been more grunt work than anything else.

On the bright side I’ve got it passing the in-tree tests by specifying a single NSS database. Under the hood it’s using NSS contexts and the new nss_pcache backend.

Something I’ll need to look into is how to release this new code. I’m not 100% sure it isn’t going to blow up on people, depending on how they have things configured. This may also represent mod_nss 2.0, we’ll see. It won’t be much of a difference functionally, with the exception that different VS would be able to have different CA trust, but internally it’ll be quite different.

So stay tuned.

CentOS 7, cloud-init and DataSourceConfigDriveNet

Something I ran into when developing the novajoin service was that my cloud-init script was not executed if either force_config_drive was True in nova.conf or if config_drive was enabled for a particular instance. What I’d see is that no metadata would come across and cloud-init would do very little work at all beyond adding keypairs and configuring networking.

The image I was working on was CentOS-7-x86_64-GenericCloud.qcow2 (1511). If I used a similar RHEL 7 image things worked fine.

I determined the issue to be with the version of cloud-init. You need cloud-init-0.7.6 for config-drive to work. I got a copy of the cloud-init rpm that Red Hat uses and used virt-customize to update my CentOS image and things worked after that.

$ virt-customize -a CentOS-7-x86_64-GenericCloud.qcow2 --install

With this image the novajoin service can push a cloud-init script that will enroll the instance into IPA.

nova metadata REST API

The nova service includes a metadata server where information about an instance is made available to that instance (for use during cloud-init, for example). This includes common things like the hostname, root password, ssh keypairs, etc.

A relatively new feature in Newton adds dynamic providers. When a request is made for metadata nova will contact the configured providers using a REST API and include the returned values in the metadata.

To enable dynamic metadata, add “DynamicJSON” to the vendordata_providers configuration option. This can also include “StaticJSON”

The vendordata_dynamic_targets configuration option specifies the URLs to be retrieved when metadata is requested.tance.

The format for an entry in vendordata_dynamic_targets is: @

Name is a string to distinguish this dynamic metadata from other dynamic providers. This will used as the key to the metadata returned to the instance.

Where name is a short string not including the ‘@’ character, and where the
URL can include a port number if so required. An example would be::

For example: test@

This dynamic metadata is available in a new file, openstack/2016-10-06/vendor_data2.json

It can be retrieved as an URL from within an instance using:

$ curl

The output will look something like:

    "test": {
        "key1": "somedata",
        "key2": "something else",

The following is passed to the dynamic REST server when nova receives a metadata request:

project-idThe UUID of the project that owns this instance.
instance-idThe UUID of this instance.
image-idThe UUID of the image used to boot this instance.
user-dataAs specified by the user at boot time.
hostnameThe hostname of the instance.
metadataAs specified by the user at boot time (aka properties)

Openstack services obtaining tokens

I had a difficult time finding information on how a service might obtain a token to talk to other services. There is certainly a lot of code to pull from but the calls are quite distributed. I got most of this from the ceilometer project.

The basic idea is that you register the keystoneauth1 options into your configuration, then instantiate it, then you can make calls to services.

I’m using a global session in this code. You can just as easily drop that and generate a new session every time you need to talk to something.

This code will obtain a token using the information from the [service_credentials] section in /etc/test/test.conf and make a request to nova and print the results.

from keystoneauth1 import loading as ks_loading
from oslo_config import cfg
from novaclient import client as nova_client


CFG_GROUP = "service_credentials"

_AUTH = None

def get_session():
    global _SESSION
    global _AUTH

    if not _AUTH:
        auth = ks_loading.load_auth_from_conf_options(cfg.CONF, 'service_credentials')

    if not _SESSION:
        _SESSION = ks_loading.load_session_from_conf_options(
            CONF, CFG_GROUP, auth=auth)

    return _SESSION

def novaclient():
    session = get_session()
    return nova_client.Client('2.1', session=session)

def register_keystoneauth_opts(conf):
    ks_loading.register_auth_conf_options(conf, CFG_GROUP)
        conf, CFG_GROUP,
        deprecated_opts={'cacert': [
            cfg.DeprecatedOpt('os-cacert', group=CFG_GROUP),
            cfg.DeprecatedOpt('os-cacert', group="DEFAULT")]

CONF([], project='test')

x = novaclient()
print x.flavors.list()

Create a configuration file

# mkdir /etc/test

Add this to /etc/test/test.conf (note that I’m re-using the nova credentials for this demo):


This was done using a Newton pre-release installation I’m developing on.