Category Archives: IPA

IPA configuration files and context

There are times when you may want more information out of the IPA server logs. I’ve seen people suggest adding debug = True to /etc/ipa/default.conf. This is fine (and it works) but it enables debugging in both the client and the server which can be annoying for command-line users.

What I do instead is create /etc/ipa/server.conf containing:

debug = True

The context that is set during initialization drives what configuration files are loaded so only the server will load this so the client remains quiet by default.

When the context is set during api.initialize it sets api.env.context. The original idea is this could drive different code paths depending on the context but in reality it hasn’t been used all that often. Being able to load context-specific configuration files is pretty neat though.

novajoin microservice integration

novajoin is a project for Openstack and IPA integration. It is a service that will allow instances created in nova to be added to IPA and a host OTP generated automatically. This OTP will then be passed into the instance to be used for enrollment during the cloud-init stage.

The end result is that a new instance will seamlessly be enrolled as an IPA client upon first boot.

Additionally, a class can be associated with an instance using Glance metadata so that IPA automember rules will automatically assign this new host to the appropriate hostgroups. Once that is done you can setup HBAC and sudo rules to grant the appropriate permissons/capabilities for all hosts in that group.

In short it can simplify administration significantly.

In the current iteration, novajoin consists of two pieces: a REST microservice and an AMQP notification listener.

The REST microservice is used to return dynamically generated metadata that will be added to the information that describes a given nova instance. This metadata is available at first boot and this is how novajoin injects the OTP into the instance for use with ipa-client-install. The framework for this change is being implemented in nova in this review: .

The REST server just handles the  metadata, cloud-init does the rest. A cloud-init script is provided which glues the two together. It installs the needed packages, retrieves the metadata, then calls ipa-client-install with the requisite options.

The other server is an AMQP listener that will identify when an IPA-enrolled instance is deleted and remove host from IPA . It may eventually handle floating IP changes as well, automatically updating IPA DNS entries. The issue here is knowing what hostname to assign to the floating IP.

Glance images can have metadata as well which describes the image, such as OS distribution and version. If these have been set then novajoin will include this in the IPA entry it creates.

The basic flow looks something like this:

  1. Boot instance in nova. Add IPA metadata, specifying ipa_enroll True and optionally ipa_hostclass
  2. Instance boots. During cloud-init it will retrieve metadata
  3. During metadata retrieval ipa host-add is executed, adding the host to IPA and generating an OTP and any image metadata available.
  4. OTP and FQDN is returned in the metadata
  5. Our cloud-init script is called to install the IPA client packages and retrieve the OTP and FQDN
  6. Call ipa-client-install –hostname FQDN –password OTP

This leaves us with an IPA-enrolled client which can have permissions granted via HBAC and sudo rules (like who is allowed to log into this instance, what sudo commands are allowed, etc).

Nova join (take 2)

Rich Megginson started a project in the Openstack Nova service to enable automatic IPA enrollment when an instance is created. I extended this to add support for metadata and pushed it into github as novajoin,

This used a hooks function within nova to allow one to extend certain functions (like add, delete, networking) etc. Unfortunately this was not well documented, nor apparently well-used, and the nova team wasn’t too keen on allowing full access to all nova internals, so they killed it.

The successor is an extension of the metadata plugin system, vendordata:

The idea is to allow one to inject custom metadata dynamically over a REST call.

IPA will provide a vendordata REST service that will create a host on demand and return the OTP for that host in the metadata. Enrollment will continue to happen via a cloud-init script which fetches the metadata to get the OTP.

A separate service will listen on notifications to capture host delete events.

I’m still working on networking as there isn’t a clear line which IP should be associated with a given hostname, and when. In other words, there is still a lot of handwaving going on.

I haven’t yet pushed the new source yet but I’m going to use the same project after I tag the current bits. There is no point continuing development of the hooks-based approach since nova will kill it after the Newton release.

FreeIPA, mod_nss and OCSP

I was messing around today on my IPA master and decided to enable OCSP. I set NSSOCP on in nss.conf and restarted and Apache failed to start after quite a long timeout.

After an embarrassingly long pause myself I figured out what the problem was: the IPA master is itself the OCSP responder and since it is proxies requests to dogtag there is a chicken and egg problem.

I worked around it by using the default responder settings in mod_nss ala:

NSSOCSPDefaultResponder on
NSSOCSPDefaultName ocsp

I also had to add the OCSP signing cert to my Apache database:

# certutil -L -d /var/lib/pki-ca/alias/ -n 'ocspSigningCert cert-pki-ca' -a > /tmp/ocsp.pem
# certutil -A -d /etc/httpd/alias -t C,, -n ocsp -a -i /tmp/ocsp.pem

Now it starts ok. It starts because it is talking directly to the CA instead of trying to talk to itself before it is up.

There is still a rather major problem though: this would fail to start on boot because Apache is configured to start before dogtag. So yet another chicken and egg problem. I’ve no easy solution for this just yet.

As a funny side note, the first time I started httpd after configuring the default responder it still didn’t start because the server cert had been revoked!? I’ve had this toy master around for a while, who knows what I’ve done to it. I resubmitted the cert request using certmonger and got a new cert and then it started just fine.

This also assumes that the CA resides on the local box. It means I don’t need to punch any holes through firewalls to make this port available to my master.

This also wouldn’t survive a renewal of the OCSP signing cert. I’d have to manually re-add the updated cert to the Apache NSS database.


I’m pretty sure my original post was against an IPA 3.x server. I had a 4.2.x server lying around so I double-checked the instructions and it still works but the OCSP port needs to be changed from 9180 to 8080:


Ipsilon, HBAC and the pam service

By default Ipsilon configures pam to authenticate using the remote service. This is at least in part because remote already exists on most systems and was easier to setup during initial development.

We see now that an Ipsilon-specific pam service should be used instead. This can be done pretty easily by using the remote service as a template. This will likely be the basis of the Ipsilon-provided service,

If you are using IPA HBAC then regardless of the service you’ll need to ensure that the users that you want to be able to use Federation have access to the configured pam service on the Ipsilon IdP host. It becomes clear pretty quickly when used with HBAC why a separate Ipsilon-specific pam service is desirable.

Attributes in the Ipsilon SSSD info plugin

A SAML assertion may contain attributes about the authenticated user. In Ipsilon these are provided by info plugins. One such plugin retrieves this information from SSSD in conjunction with the Apache mod_lookup_identity plugin. SSSD provides the attributes to mod_lookup_identity when the user authenticates. These are made into environment variables by mod_lookup_identity and added to the SAML Assertion by Ipsilon.

The infosssd plugin must be enabled for this to work and it is only user-configurable at install time currently because root is needed to modify the SSSD and Apache configuration files.

The data flow is: SSSD -> mod_lookup_identity -> env variables -> Ipsilon -> Assertion

The list of attributes is far from being user-configurable as of Ipsilon 1.0: the current list of attributes is hardcoded.

This is what the server installer adds to sssd.conf which defines what attributes to provide to mod_lookup_identity:

ldap_user_extra_attrs =  mail, street, locality, postalCode, telephoneNumber, givenname, sn

services = nss, pam, ssh, ifp

allowed_uids = apache, root
user_attributes = +mail, +street, +locality, +postalCode, +telephoneNumber, +givenname, +sn

Apache has related configuration to make the SSSD values available as environment variables to Ipsilon:

<Location /idp>
  LookupUserAttr locality REMOTE_USER_STATE
  LookupUserAttr street REMOTE_USER_STREET
  LookupUserAttr telephoneNumber REMOTE_USER_TELEPHONENUMBER
  LookupUserAttr givenname REMOTE_USER_FIRSTNAME
  LookupUserAttr mail REMOTE_USER_EMAIL
  LookupUserAttr postalCode REMOTE_USER_POSTALCODE
  LookupUserGroupsIter REMOTE_USER_GROUP

Finally, the infosssd plugin has a mapping from these environment variables into the internal attribute naming conventions:

sssd_mapping = [
    ['REMOTE_USER_GECOS', 'fullname'],
    ['REMOTE_USER_EMAIL', 'email'],
    ['REMOTE_USER_FIRSTNAME', 'givenname'],
    ['REMOTE_USER_LASTNAME', 'surname'],
    ['REMOTE_USER_STREET', 'street'],
    ['REMOTE_USER_STATE', 'state'],
    ['REMOTE_USER_POSTALCODE', 'postcode'],

On top of this, the common Info mapping code has its own hardcoded list of  possible attributes:

        self.standard_attributes = {
            'fullname': 'Full Name',
            'nickname': 'Nickname',
            'surname': 'Last Name',
            'firstname': 'First Name',
            'title': 'Title',
            'dob': 'Date of Birth',
            'email': 'E-mail Address',
            'gender': 'Gender',
            'postcode': 'Postal Code',
            'street': 'Street Address',
            'state': 'State or Province',
            'country': 'Country',
            'phone': 'Telephone Number',
            'language': 'Language',
            'timezone': 'Time Zone',

These are the places one would need to change if one wanted to add an additional attribute to the mapping.


  • A reload of Apache is required after changing an Ipsilon python file
  • Direct changes to Ipsilon python files will be lost on updates

Ipsilon and IPA

Ipsilon is an IdP that supports SAML 2.0. Here I’ll show how the IPA integration works to use IPA as the source of SSL certificates and how to use IPA as the identity backend to Ipsilon.

I have an existing IPA installation that I’ll use as the IPA master. For Ipsilon I’ve got two Openstack VM (1GB RAM, 10GB disk) running Fedora 22.

Install the IdP

Pick one and this is the IdP.  Install the Ipsilon server packages:

$ sudo dnf install ipsilon ipsilon-infosssd ipsilon-saml2 ipsilon-authgssapi ipsilon-tools-ipa freeipa-client --enablerepo=updates-testing

Enroll the machine as an IPA client:

$ sudo ipa-client-install

Install the Ipsilon IdP:

$ sudo kinit admin
$ sudo ipsilon-server-install --ipa=yes --info-sssd=yes --form=yes

Have IPA issue an SSL certificate for Apache:

$ sudo systemctl start certmonger
$ sudo ipa-getcert request -f /etc/pki/tls/certs/server.pem -k /etc/pki/tls/private/server.key -K HTTP/`hostname`

Configure Apache to use this certificate.

$ sudo  /etc/httpd/conf.d/ssl.conf

Modify SSLCertificateFile and SSLCertificateKeyFile to:

SSLCertificateFile /etc/pki/tls/certs/server.pem
SSLCertificateKeyFile /etc/pki/tls/private/server.key

Restart Apache:

$ sudo systemctl restart httpd

Note: As of today there is an SELinux issue preventing Apache from reading ipsilon.conf. I set it to permissive mode.

Test to see that SSL trust is working and the IdP is serving pages:

$ curl https://`hostname`/idp

Scan the output and you should see a link to /idp/login. If you do then it’s working.

Ok, let’s check it out for real now. Fire up a browser and head to https://idp/

You should see a form login. Login using the IPA admin and password. Woo, integration! If you want to really test it, log out, kinit as admin on the machine you launched the browser from (assuming it is an IPA client) and try again and you should get right in.

Install the Service Provider

Let’s a pause a moment and switch over to your other VM onto which we’ll install the Service Provider (SP). The process is similar.

Install the ipsilon and IPA client packages:

$ sudo dnf install ipsilon-client freeipa-client freeipa-admintools mod_ssl --enablerepo=updates-testing

Enroll as an IPA client:

$ sudo ipa-client-install

Install the Ipsilon SP:

$ sudo ipsilon-client-install --saml-idp-metadata --saml-auth /secure

Create an SSL certificate for the SP webserver:

$ sudo kinit admin
$ sudo ipa service-add HTTP/`hostname` --force
$ sudo ipa-getcert request -f /etc/pki/tls/certs/server.pem -k /etc/pki/tls/private/server.key -K HTTP/`hostname`

Configure Apache to use this certificate.

$ sudo  /etc/httpd/conf.d/ssl.conf

Modify SSLCertificateFile and SSLCertificateKeyFile to:

SSLCertificateFile /etc/pki/tls/certs/server.pem
SSLCertificateKeyFile /etc/pki/tls/private/server.key

Restart Apache:

$ sudo systemctl restart httpd

Register the SP

Now we need to register the SP with the IdP. To do this bring up your browser window we opened with the IdP and head to Administration -> Identity Providers -> saml2 -> Manage -> Add New

The name is not important, it is just a unique identifier for the IdP to reference the SP. I tend to use the hostname of the SP.

The metadata can be found on the SP in the filesystem in /etc/httpd/saml2/<fqdn>

For the metadata you can either:

  1. Download the metadata to the machine running the browser, select Browse, find the file and upload it.
  2. Provide the metadata via a URL (
  3. Copy and paste the metadata

The resulting page is the configuration page for the SP. We can leave the defaults for now.

Configure the SP secure pages

Let’s create some secure content to serve on the SP.

$ sudo vi /var/www/html/secure/index.html

Add some content like:

<html><title>Secure</title>Hello there</html>

Now confirm that it requires authentication:

$ curl https://`hostname`/secure/

You should get back a 303 See Other response.

Now try this in your browser. You should be redirected to the IdP to authentication. Do so using the IPA admin credentials. After authenticating you should be redirected back to /secure on the SP and see the text “Hello there”. Success!

Test Logout

Modify the index.html we just created to include a link to

<a href="/saml2/logout?ReturnTo=">Log out</a>

Create a logout landing point (note it is outside the secured area):

$ sudo vi /var/www/html/logged_out.html

with contents:

Congratulations, you've been logged out!
Now try to <a href="/secure/">log back in</a>

You can just refresh the secured page and you should have a Log out link.

Click that and you should see a logout message and a link to log back in. If you click that you will be asked to re-authenticate.

For further exercises, stand up another SP. You’ll find that once you authenticate to one SP you are allowed in with no authentication request on the other one. Similarly, logging out of one logs out of all.

Add more users

So far we’ve limited things to only the IPA admin user. Let’s add a regular IPA user and authenticate as them. On either the SP or the IdP run:

$ kinit admin
$ ipa user-add --first=Test --last=User tuser1 --password
Enter Password again to verify:
$ kinit tuser1
Password for tuser1@EXAMPLE.COM: 
Password expired.  You must change it now.
Enter new password: 
Enter it again:

Now request /secure/ on the SP again and this time authenticate as tuser1. This will work for any IPA user.

Attribute Data

Ipsilon can configure what user information is visible to a given SP and can optionally rename some attributes (called mapping).

A broad way to see what is available is to use Server-Side Includes. Let’s play.

Enable SSI in Apache by editing /etc/httpd/conf.d/ipsilon-saml.conf and adding Options +Includes to the secure Location and a new output filter. It should look like:

    MellonEnable "auth"
    Header append Cache-Control "no-cache"
    Options +Includes

AddOutputFilter INCLUDES .html

Note that you wouldn’t want to do this in production, we’re just playing here.

Restart apache

$ sudo systemctl restart httpd

Configure your secure index.html to display the environment. Add this to the end of the file, before </html>:

<!--#printenv -->

The output is rather ugly, one very long line of output. We’re interested in those attributes prefixed with MELLON_. In my case it is:

MELLON_NAME_ID=tuser1 MELLON_NAME_ID_0=tuser1 MELLON_surname=User MELLON_surname_0=User MELLON_groups=ipausers MELLON_groups_0=ipausers MELLON_givenname=Test MELLON_givenname_0=Test MELLON_fullname=Test User MELLON_fullname_0=Test User MELLON_IDP= MELLON_IDP_0= 

When we installed Ipsilon we enabled the SSSD info plugin. This controls what information is retrieved by SSSD from IPA when a user authenticates. Ipsilon is configured to retrieve the LDAP attributes mail, street, locality, postalCode, telephoneNumber, givenname and surname. We don’t see any address information here because we didn’t configure it for the user, let’s do that now.

$ ipa user-mod --street="123 Main" --city=Baltimore --postalcode=21234 tuser1

Log out of the SP and log back in and you should now have MELLON_ variables for street, state and postalcode.

If you’re wondering why there are _0 versions of these as well it is a mechanism to support multi-valued attributes.

How do I promote an IPA replica to a master?

The short answer is: you don’t, it’s already a master!

All IPA servers are masters, and equals. Some are just more equal than others. The distinguishing factors are: which was the first master installed and does this master have a CA?

In any IPA installation you absolutely want > 1 masters running a CA so you don’t have a single point of failure. When installing a new master this is not done automatically. You need to add the --setup-ca flag, or run ipa-ca-install post-install.

The first IPA master installed is distinguished by two tasks it is responsible for: generating the CRL and renewing the CA subsystem certificates. See the IPA wiki for details on how to switch the master responsible for these at

SubjectAltNames and NSS

NSS is very strict when it comes to validating a certificate hostname. Per RFC 2818 if there is a subjectAltName defined then ONLY the subjectAltName is used to validate the certificate. Based only a little bit of testing, it appears that OpenSSL is a bit more lenient because a certificate that returns an error in NSS worked with s_client.

The error you’d get out of NSS via curl is:

Unable to communicate securely with peer: requested domain name does not match the server's certificate.

So, if you have host and you want to create a certificate to serve that and you’ll need to include two subjectAltName DNSNames in the CSR, one for each host.

With certmonger it might look something like this:

# ipa-getcert request -d /etc/httpd/alias -n Server-Cert -p /etc/httpd/alias/pwdfile.txt -N "" -D "" -K HTTP/ -D ""

Note that this won’t actually work well with IPA by default. You’d also need to tweak ipa-rewrite.conf so the request doesn’t result in a 301 redirect.

FreeIPA 3.3.5 and sudo

The last time I had any real reason to play with sudo and IPA was before sssd got sudo support. I found the previous sudo-ldap debugging quite good, even if sudo itself was rather slow due to lack of caching.

A lot of users seem to have problems getting it setup since older IPA clients will not do this automatically so I thought I’d give it a go. I’m doing this on an up-to-date Fedora 20 system using the following IPA and SSSD:


I started with the sssd-sudo(8) man page which laid out quite clearly the changes I needed to make to /etc/nsswitch.conf and sssd.conf. I restarted sssd and found my user couldn’t sudo at all, which makes sense since I hadn’t added any rules yet.

Ok, so I add a single rule to run any command on any host for a new group I added, sudoers of which my test user is a member. Oh, and be sure that the user is a member of the group before logging in so the groups evaluate properly.

I created the group and sudo rule with:

[admin@ipaserver]$ ipa group-add sudoers
[admin@ipaserver]$ ipa group-add-member sudoers --users=tuser1
[admin@ipaserver]$ ipa sudorule-add --hostcat=all --cmdcat=all sudoers
[admin@ipaserver]$ ipa sudorule-add-user --group=sudoers sudoers

I should also note I still have the HBAC allow_all rule enabled. If you’ve disabled this then you’ll need to grant sudo rights to the users you want to be able to execute it.

Before starting real testing, I created /etc/sudo.conf with these contents:

Debug sudo /var/log/sudo.log all@debug

This gives me a quite verbose log of what is going on. It probably makes more sense to a sudo developer but I can more or less follow along with the number of rules being evaluated, etc.

To double-check that the rule exists we can look in it in LDAP as the IPA admin user:

[admin@ipaserver]$ kinit admin
[admin@ipaserver]$ ldapsearch -LLL -Y GSSAPI -b ou=SUDOers,dc=example,dc=com
SASL/GSSAPI authentication started
SASL username: admin@EXAMPLE.COM
SASL data security layer installed.
dn: ou=sudoers,dc=example,dc=com
objectClass: extensibleObject
ou: sudoers

dn: cn=sudoers,ou=sudoers,dc=example,dc=com
objectClass: sudoRole
objectClass: top
sudoUser: %sudoers
sudoHost: ALL
sudoCommand: ALL
cn: sudoers

Ok, so now I just need to ssh into this box and try sudo -l

[admin@ipaserver]$ ssh tuser1@ipaserver
[tuser1@ipaserver]$ sudo -l
[sudo] password for tuser1: 
User tuser1 may run the following commands on ipaserver:
    (root) ALL

I also want to avoid authentication so I can update the rule to not require it:

[admin@ipaserver]$ ipa sudorule-add-option sudoers --sudooption='!authenticate'

Remember that the rules are cached so changes may not be available immediately, but it worked for me:

[tuser1@ipaserver]$ sudo -l
User tuser1 may run the following commands on ipaserver:
    (root) ALL

A somewhat old, but good, document to read is In particular it has good information how the caching works.