novajoin and the 2016 OpenStack Barcelona Summit

The nova v2 vendordata (dynamic vendordata plugins) came up at the 2016 OpenStack Summit in Barcelona. It was expected at the outset to be a fairly short meeting since the two proposals were seemingly straightfoward: add authentication and caching. It ended up using all 40 minutes but in the end wasn’t all that contentious.

It was decided that authentication would be sent in all cases to the configured vendordata plugins. There was some brief discussion about whether the user token would be included but as I took it, nova would use its own (or some pre-configured) credential. This will require some paste configuration changes in novajoin but authentication should otherwise be supported.

The metadata server will also cache responses. Exactly how long, what, etc is TBD. If I remember I’ll update this post with the gerrit review link once it comes out.

The idea is to commit to master than back port to Newton stable.

The contentious part was related to that user token I mentioned. Adam Young from Keystone wanted that token to be sent along, even if expired, so one could know the roles of the user that kicked things off. The problem of course is that the user is just a snapshot in time. Roles change. Users are deleted. Apparently some users completely hammer on the metadata service today, some as frequently as every few minutes. At some point things could break if that user went away.

I was ambivalent about it. Adam’s point was that it could be used for access control which is a good idea. I think that if the roles were cached instead of the user, that might make more sense. But even then people would complain that they revoked or added access and the user can/can’t do things. It’s a no-win I think. I kept my mouth shut in any case.

In the end this is good news for novajoin. I was quite uncomfortable having unauthenticated requests at all (e.g. metadata requests from an instance) so that’ll go away soon.

The caching will solve the problems I had bending over backwards with the IPA OTP. There could still be problems if the time to enroll the instance > nova metadata cache so I’ll probably leave in my “last update wins” code, but this does make things a bit more predictable and will certainly be faster.

mod_nss and NSS Context part 2

I’ve made quite a lot more progress on migrating mod_nss to using NSS contexts, but I’m not quite done yet.

The server starts and handles some requests, so that’s something. It just freaks out a bit if there are multiple nicknames for the “same” virtual host (bug in SNI code) and it is generally unfriendly when it comes to nicknames.

When multiple NSS databases are initialized at the same time the nicknames become quite odd. The “first” database initialized, something probably not predictable in mod_nss, is a traditional database where references to the nickname without a token are handled.

Other databases use the slot in the nickname, e.g. NSS Certificate DB or in the case of mod_nss, internal. I have a huge hack in place now to try both iterations and it works when there are just two NSS databases and uniquely named nicknames. I think that I’ll need to detect duplicate nicks and blow up early otherwise things just don’t work as they should.

I spent an afternoon hacking on gencert to make it more generic so I can more easily generate multiple NSS certificate databases for testing. I started testing with 3 and like I said, it sorta works.

I’ve run into two difficult problems at the moment:

  1. The SNI code is based only on the vhost id so if there are multiple nicknames for a vhost (on different ports for example) then the wrong nickname could be returned
  2. Even if I have the right nickname looking it up using PK11_FindCertFromNickname() is failing to find it. I’ve tried using Server-Cert and internal:Server-Cert and nothing is found. I’m a bit baffled by this and unfortunately will have to dive into gdb. I say unfortunately because IIRC the PKCS#11 handling is a nightmare of callbacks.

Still, the progress has promise.

I need to double check on this but I’m pretty sure that NSS keeps a single trust table internally so my idea of being able to separate CA trust by vhost seems to be a pipe-dream for now.

mod_nss and NSS Contexts

There has been a long standing need to convert mod_nss from using a single NSS database to support NSS Contexts which will allow for much greater flexibility.

I started hacking on it between other work a couple of days ago and progress has been slow to say the least. It’s amazing the number of assumptions baked into the code that there is a single database. So far I’ve had to:

  • Move the database config option from being treated only as a module config option to a proper Apache server context.
  • Move a lot of initialization code around that verifies the NSS database filesystem read permissions and make it iterable
  • Convert nss_pcache from validating the token password to just storing it. I now do a NSS_NoDB_Init() because there could be multiple sources and I don’t want to deal with that headache. This shifts the burden of identifying a bad password a bit later but it should be handled ok

I’ve run into another roadblock similar to the database config option being treated as global: the password dialog is also treated as a global. It really needs to be handled per-server context. And this is where I am.

Fortunately it has mostly been a matter of reorganizing some structures and fixing all uses of them, so it’s been more grunt work than anything else.

On the bright side I’ve got it passing the in-tree tests by specifying a single NSS database. Under the hood it’s using NSS contexts and the new nss_pcache backend.

Something I’ll need to look into is how to release this new code. I’m not 100% sure it isn’t going to blow up on people, depending on how they have things configured. This may also represent mod_nss 2.0, we’ll see. It won’t be much of a difference functionally, with the exception that different VS would be able to have different CA trust, but internally it’ll be quite different.

So stay tuned.