Notifications in devstack

I still haven’t found any good documentation on setting up notifications. Most of the blog entries I’ve found are quite dated and don’t seem to apply to Mitaka-era installs. This is what I came up with (I think) in a devstack environment in /etc/nova/nova.conf:

notification_driver = messagingv2
notification_topics = notifications
notify_on_state_change = vm_state
notify_on_any_change = True

I’m still toying with this but wanted to put something down before I do something dumb like re-install devstack, like I did the last time I had it working and wiped out my configuration.

This is the python I’m using for now:

import json
from oslo_config import cfg
import oslo_messaging

class NotificationEndpoint(object):
    def info(self, ctxt, publisher_id, event_type, payload, metadata):
        print 'notification:'
        print json.dumps(payload, indent=4)
        print publisher_id
        print event_type
        print metadata

transport = oslo_messaging.get_transport(cfg.CONF)
targets = [ oslo_messaging.Target(topic='notifications') ]
endpoints = [ NotificationEndpoint() ]
server = oslo_messaging.get_notification_listener(transport, targets, endpoints)
print "Starting"
#print "Waiting"

FreeIPA, mod_nss and OCSP

I was messing around today on my IPA master and decided to enable OCSP. I set NSSOCP on in nss.conf and restarted and Apache failed to start after quite a long timeout.

After an embarrassingly long pause myself I figured out what the problem was: the IPA master is itself the OCSP responder and since it is proxies requests to dogtag there is a chicken and egg problem.

I worked around it by using the default responder settings in mod_nss ala:

NSSOCSPDefaultResponder on
NSSOCSPDefaultName ocsp

I also had to add the OCSP signing cert to my Apache database:

# certutil -L -d /var/lib/pki-ca/alias/ -n 'ocspSigningCert cert-pki-ca' -a > /tmp/ocsp.pem
# certutil -A -d /etc/httpd/alias -t C,, -n ocsp -a -i /tmp/ocsp.pem

Now it starts ok. It starts because it is talking directly to the CA instead of trying to talk to itself before it is up.

There is still a rather major problem though: this would fail to start on boot because Apache is configured to start before dogtag. So yet another chicken and egg problem. I’ve no easy solution for this just yet.

As a funny side note, the first time I started httpd after configuring the default responder it still didn’t start because the server cert had been revoked!? I’ve had this toy master around for a while, who knows what I’ve done to it. I resubmitted the cert request using certmonger and got a new cert and then it started just fine.

This also assumes that the CA resides on the local box. It means I don’t need to punch any holes through firewalls to make this port available to my master.

This also wouldn’t survive a renewal of the OCSP signing cert. I’d have to manually re-add the updated cert to the Apache NSS database.


I’m pretty sure my original post was against an IPA 3.x server. I had a 4.2.x server lying around so I double-checked the instructions and it still works but the OCSP port needs to be changed from 9180 to 8080:


Ipsilon UI templating

The Ipsilon UI is user-configurable with some caveats.

The template files can be found in /usr/share/ipsilon/templates and the CSS and javascript in /usr/share/ipsilon/ui.

The caveat is that these files may be updated upstream between releases and if the packaging system detects local changes then the local files will not be updated. For rpm-based systems a new file, .rpmnew, will need to be examined and potentially merged into the locally-modified file.


Ipsilon uses PatternFly 2.0 for its UI framework. The Ipsilon-specific configuration is in admin.css and ipsilon.css. Modifying these files directly may be difficult. If custom changes are required then modifying the less files in the upstream source and rebuilding the CSS is probably easier to maintain.

HTML Templates

The HTML templates consist of a number of common files in the root, administrative UI templates in admin, protocol provider-specific files in openid, personal and saml2 and login-specific files in login.

The subdirectory install contains templates used during installation.

Ipsilon uses the python jinja2 engine for HTML templating. This supports some basic conditionals via {%- if %} {%- endif %} blocks, looping via {% for value in values %} and variable substitution via {{ substitute-me }}. See the jinja2 docs for more details.

Customization to functionality in the UI will be limited by what is made available by Ipsilon itself. Some common variables are:

basepath: URI of the idp, e.g.

user.fullname: the login name of the authenticated user. name is an alias for fullname

user.is_admin: boolean, true if the user is an Ipsilon administrator

message: error message when errors are raised

providers: list of SAML2 Service provider objects

Ipsilon, HBAC and the pam service

By default Ipsilon configures pam to authenticate using the remote service. This is at least in part because remote already exists on most systems and was easier to setup during initial development.

We see now that an Ipsilon-specific pam service should be used instead. This can be done pretty easily by using the remote service as a template. This will likely be the basis of the Ipsilon-provided service,

If you are using IPA HBAC then regardless of the service you’ll need to ensure that the users that you want to be able to use Federation have access to the configured pam service on the Ipsilon IdP host. It becomes clear pretty quickly when used with HBAC why a separate Ipsilon-specific pam service is desirable.

info-sssd and pam authentication

We discovered that the info-sssd plugin doesn’t play nicely when the pam auth plugin is used. This is because info-sssd relies on mod_identity_lookup in Apache to lookup the authenticated REMOTE_USER and retrieve the attributes. The pam auth plugin authenticates directly from within Ipsilon so mod_lookup_identity never gets invoked and no attributes are generated.

The solution is to disable the pam auth plugin and use the form plugin instead.

We are going to solve this more gently in the future by providing “login stacks.” Basically a set of known working stacks that can be applied to a given SP as avenues for authentication and info retrieval. We’re not quite there yet.

Ipsilon server debug logging

To enable debugging on the Ipsilon server you need to modify two settings in /etc/ipsilon/idp/ipsilon.conf:

debug = True
log.screen = True

After making these two changes you’ll need to restart the Apache web server:

# service httpd restart

The additional debug information will go into one of the Apache error lgos. On Fedora/RHEL/CentOS this is /var/log/httpd/ssl_error_log.

Attributes in the Ipsilon SSSD info plugin

A SAML assertion may contain attributes about the authenticated user. In Ipsilon these are provided by info plugins. One such plugin retrieves this information from SSSD in conjunction with the Apache mod_lookup_identity plugin. SSSD provides the attributes to mod_lookup_identity when the user authenticates. These are made into environment variables by mod_lookup_identity and added to the SAML Assertion by Ipsilon.

The infosssd plugin must be enabled for this to work and it is only user-configurable at install time currently because root is needed to modify the SSSD and Apache configuration files.

The data flow is: SSSD -> mod_lookup_identity -> env variables -> Ipsilon -> Assertion

The list of attributes is far from being user-configurable as of Ipsilon 1.0: the current list of attributes is hardcoded.

This is what the server installer adds to sssd.conf which defines what attributes to provide to mod_lookup_identity:

ldap_user_extra_attrs =  mail, street, locality, postalCode, telephoneNumber, givenname, sn

services = nss, pam, ssh, ifp

allowed_uids = apache, root
user_attributes = +mail, +street, +locality, +postalCode, +telephoneNumber, +givenname, +sn

Apache has related configuration to make the SSSD values available as environment variables to Ipsilon:

<Location /idp>
  LookupUserAttr locality REMOTE_USER_STATE
  LookupUserAttr street REMOTE_USER_STREET
  LookupUserAttr telephoneNumber REMOTE_USER_TELEPHONENUMBER
  LookupUserAttr givenname REMOTE_USER_FIRSTNAME
  LookupUserAttr mail REMOTE_USER_EMAIL
  LookupUserAttr postalCode REMOTE_USER_POSTALCODE
  LookupUserGroupsIter REMOTE_USER_GROUP

Finally, the infosssd plugin has a mapping from these environment variables into the internal attribute naming conventions:

sssd_mapping = [
    ['REMOTE_USER_GECOS', 'fullname'],
    ['REMOTE_USER_EMAIL', 'email'],
    ['REMOTE_USER_FIRSTNAME', 'givenname'],
    ['REMOTE_USER_LASTNAME', 'surname'],
    ['REMOTE_USER_STREET', 'street'],
    ['REMOTE_USER_STATE', 'state'],
    ['REMOTE_USER_POSTALCODE', 'postcode'],

On top of this, the common Info mapping code has its own hardcoded list of  possible attributes:

        self.standard_attributes = {
            'fullname': 'Full Name',
            'nickname': 'Nickname',
            'surname': 'Last Name',
            'firstname': 'First Name',
            'title': 'Title',
            'dob': 'Date of Birth',
            'email': 'E-mail Address',
            'gender': 'Gender',
            'postcode': 'Postal Code',
            'street': 'Street Address',
            'state': 'State or Province',
            'country': 'Country',
            'phone': 'Telephone Number',
            'language': 'Language',
            'timezone': 'Time Zone',

These are the places one would need to change if one wanted to add an additional attribute to the mapping.


  • A reload of Apache is required after changing an Ipsilon python file
  • Direct changes to Ipsilon python files will be lost on updates

Ipsilon attribute mapping and filtering

A SAML assertion can contain one or more attributes. These are basically pieces of information about the authenticated user. It is important to remember that they are only provided when the user logs in. So, for example, if a user logs in, then you change their e-mail address, the SAML assertion will reflect the original value until that user logs out and back in again.

Ipsilon provides two ways to control what attributes are sent in an assertion and what those attributes are named. This is called attribute mapping and allowed attributes.

Ipsilon supports two levels of mapping and attribute visibility:

  • global level configuration
  • per-SP configuration

These are not additive. If an SP provides its own mapping or list of allowed attributes then that overrides the global setting. It is, though, possible to mix one or the other. So you can have a global mapping and a SP-specific set of allowed attributes.

Info Plugins

It all begins with Info plugins. These retrieve the attributes from the identity source and the available attributes depend entirely on the capabilities of the info plugin. The nss plugin, for example, is limited to those things in <tt>/etc/passwd</tt>. The ldap plugin on the other hand can retrieve any attribute from LDAP.

The list of attributes available from the  SSSD info plugin is configured differently than other info plugins because root is required to manage the SSSD configuration file (<tt>/etc/sssd/sssd.conf</tt>).

The list of attributes is controlled by two variables: ldap_user_extra_attrs and user_attributes. ldap_user_extra_attrs tells sssd what extra attributes to request when it gets a user over LDAP (like in the IPA case). user_attributes controls what extra attributes are available over the SSSD info pipe (D-bus). See sssd-ldap(5) and sssd-ifp(5) for more details.

By default SSSD is configured to retrieve: mail, street, locality, postalCode, telephoneNumber, givenname and surname (sn).

Mapping and Filtering

Mapping allows you to rename an info variable to match the needs of your environment. For example, the user’s e-mail address is defined as mail by Ipsilon but your application may require it to be email. This can be managed via mapping.

Note that mapping an attribute does not remove the original. So if you were to create a mapping from mail -> email you would end up with the following attributes on the SP:


To remove the mail version you would need to use the Allowed Attributes setting to select only those attributes you want sent in the assertion.

The current filtering is a bit limited, providing only a white list of attributes. You can specify all attributes (*) or provide a discrete list of the attributes you want. You cannot specify negative attributes (e.g. all but mail) or a regular expression.

Ipsilon and IPA

Ipsilon is an IdP that supports SAML 2.0. Here I’ll show how the IPA integration works to use IPA as the source of SSL certificates and how to use IPA as the identity backend to Ipsilon.

I have an existing IPA installation that I’ll use as the IPA master. For Ipsilon I’ve got two Openstack VM (1GB RAM, 10GB disk) running Fedora 22.

Install the IdP

Pick one and this is the IdP.  Install the Ipsilon server packages:

$ sudo dnf install ipsilon ipsilon-infosssd ipsilon-saml2 ipsilon-authgssapi ipsilon-tools-ipa freeipa-client --enablerepo=updates-testing

Enroll the machine as an IPA client:

$ sudo ipa-client-install

Install the Ipsilon IdP:

$ sudo kinit admin
$ sudo ipsilon-server-install --ipa=yes --info-sssd=yes --form=yes

Have IPA issue an SSL certificate for Apache:

$ sudo systemctl start certmonger
$ sudo ipa-getcert request -f /etc/pki/tls/certs/server.pem -k /etc/pki/tls/private/server.key -K HTTP/`hostname`

Configure Apache to use this certificate.

$ sudo  /etc/httpd/conf.d/ssl.conf

Modify SSLCertificateFile and SSLCertificateKeyFile to:

SSLCertificateFile /etc/pki/tls/certs/server.pem
SSLCertificateKeyFile /etc/pki/tls/private/server.key

Restart Apache:

$ sudo systemctl restart httpd

Note: As of today there is an SELinux issue preventing Apache from reading ipsilon.conf. I set it to permissive mode.

Test to see that SSL trust is working and the IdP is serving pages:

$ curl https://`hostname`/idp

Scan the output and you should see a link to /idp/login. If you do then it’s working.

Ok, let’s check it out for real now. Fire up a browser and head to https://idp/

You should see a form login. Login using the IPA admin and password. Woo, integration! If you want to really test it, log out, kinit as admin on the machine you launched the browser from (assuming it is an IPA client) and try again and you should get right in.

Install the Service Provider

Let’s a pause a moment and switch over to your other VM onto which we’ll install the Service Provider (SP). The process is similar.

Install the ipsilon and IPA client packages:

$ sudo dnf install ipsilon-client freeipa-client freeipa-admintools mod_ssl --enablerepo=updates-testing

Enroll as an IPA client:

$ sudo ipa-client-install

Install the Ipsilon SP:

$ sudo ipsilon-client-install --saml-idp-metadata --saml-auth /secure

Create an SSL certificate for the SP webserver:

$ sudo kinit admin
$ sudo ipa service-add HTTP/`hostname` --force
$ sudo ipa-getcert request -f /etc/pki/tls/certs/server.pem -k /etc/pki/tls/private/server.key -K HTTP/`hostname`

Configure Apache to use this certificate.

$ sudo  /etc/httpd/conf.d/ssl.conf

Modify SSLCertificateFile and SSLCertificateKeyFile to:

SSLCertificateFile /etc/pki/tls/certs/server.pem
SSLCertificateKeyFile /etc/pki/tls/private/server.key

Restart Apache:

$ sudo systemctl restart httpd

Register the SP

Now we need to register the SP with the IdP. To do this bring up your browser window we opened with the IdP and head to Administration -> Identity Providers -> saml2 -> Manage -> Add New

The name is not important, it is just a unique identifier for the IdP to reference the SP. I tend to use the hostname of the SP.

The metadata can be found on the SP in the filesystem in /etc/httpd/saml2/<fqdn>

For the metadata you can either:

  1. Download the metadata to the machine running the browser, select Browse, find the file and upload it.
  2. Provide the metadata via a URL (
  3. Copy and paste the metadata

The resulting page is the configuration page for the SP. We can leave the defaults for now.

Configure the SP secure pages

Let’s create some secure content to serve on the SP.

$ sudo vi /var/www/html/secure/index.html

Add some content like:

<html><title>Secure</title>Hello there</html>

Now confirm that it requires authentication:

$ curl https://`hostname`/secure/

You should get back a 303 See Other response.

Now try this in your browser. You should be redirected to the IdP to authentication. Do so using the IPA admin credentials. After authenticating you should be redirected back to /secure on the SP and see the text “Hello there”. Success!

Test Logout

Modify the index.html we just created to include a link to

<a href="/saml2/logout?ReturnTo=">Log out</a>

Create a logout landing point (note it is outside the secured area):

$ sudo vi /var/www/html/logged_out.html

with contents:

Congratulations, you've been logged out!
Now try to <a href="/secure/">log back in</a>

You can just refresh the secured page and you should have a Log out link.

Click that and you should see a logout message and a link to log back in. If you click that you will be asked to re-authenticate.

For further exercises, stand up another SP. You’ll find that once you authenticate to one SP you are allowed in with no authentication request on the other one. Similarly, logging out of one logs out of all.

Add more users

So far we’ve limited things to only the IPA admin user. Let’s add a regular IPA user and authenticate as them. On either the SP or the IdP run:

$ kinit admin
$ ipa user-add --first=Test --last=User tuser1 --password
Enter Password again to verify:
$ kinit tuser1
Password for tuser1@EXAMPLE.COM: 
Password expired.  You must change it now.
Enter new password: 
Enter it again:

Now request /secure/ on the SP again and this time authenticate as tuser1. This will work for any IPA user.

Attribute Data

Ipsilon can configure what user information is visible to a given SP and can optionally rename some attributes (called mapping).

A broad way to see what is available is to use Server-Side Includes. Let’s play.

Enable SSI in Apache by editing /etc/httpd/conf.d/ipsilon-saml.conf and adding Options +Includes to the secure Location and a new output filter. It should look like:

    MellonEnable "auth"
    Header append Cache-Control "no-cache"
    Options +Includes

AddOutputFilter INCLUDES .html

Note that you wouldn’t want to do this in production, we’re just playing here.

Restart apache

$ sudo systemctl restart httpd

Configure your secure index.html to display the environment. Add this to the end of the file, before </html>:

<!--#printenv -->

The output is rather ugly, one very long line of output. We’re interested in those attributes prefixed with MELLON_. In my case it is:

MELLON_NAME_ID=tuser1 MELLON_NAME_ID_0=tuser1 MELLON_surname=User MELLON_surname_0=User MELLON_groups=ipausers MELLON_groups_0=ipausers MELLON_givenname=Test MELLON_givenname_0=Test MELLON_fullname=Test User MELLON_fullname_0=Test User MELLON_IDP= MELLON_IDP_0= 

When we installed Ipsilon we enabled the SSSD info plugin. This controls what information is retrieved by SSSD from IPA when a user authenticates. Ipsilon is configured to retrieve the LDAP attributes mail, street, locality, postalCode, telephoneNumber, givenname and surname. We don’t see any address information here because we didn’t configure it for the user, let’s do that now.

$ ipa user-mod --street="123 Main" --city=Baltimore --postalcode=21234 tuser1

Log out of the SP and log back in and you should now have MELLON_ variables for street, state and postalcode.

If you’re wondering why there are _0 versions of these as well it is a mechanism to support multi-valued attributes.