SSL certificate automation and Let’s Encrypt integration

[andersk]

We would like to be able to accept certificates from ​Let’s Encrypt. Unfortunately, because Let’s Encrypt issues certificates with a 90 day lifetime, this would significantly increase our regular maintenance burden. So we need to develop some kind of automation before allowing this:

1. CSR generation. Leaving this as a manual process is probably acceptable for now, since it only needs to happen once per host: CSRs are reusable as long as the key doesn’t change. Automating CSRs is the topic of #241.

2. Interface for the user to provide certificates. The user could upload or paste their certificate into Pony which would store it into the vhost’s LDAP record. It would be good to have a token-based API to let the user automate this without client certificates.

3. Validation of provided certificates. This is necessary because an invalid certificate can cause Apache to fail to start (although that’s less of an issue with #52). It would also be nice to warn the user about common errors like incomplete or misordered chains.

4. Making Apache use the certificates. An ideal solution is #52, but that requires some nontrivial Apache development. Until then, we could write a daily cronjob to copy out the certificates and reload Apache. In fact, the cronjob might as well reify the vhost configurations too, so that we can ditch all the per-site configuration in svn.

5. Helping users automate Let’s Encrypt renewals. This might take the form of a script for the user to serve at .well-known/acme-challenge and a line to for the user to add to their crontab.

See also ​help.mit.edu #3519679. An alternative idea mentioned there is storing certificates as files in the corresponding lockers rather than in LDAP; however, this would make it impossible for the user to get feedback from validation, and would conflict with #52.

[achernya]

Proposed steps to implement this, given that #52 is well in progress.

1. Update the LDAP schema to include the certificate. Update reify-vhost.py to write out the certificates from LDAP. Upload existing certificates into LDAP and stop versioning the certificates in SVN.

2. Write a tool that performs Let's Encrypt issuance and puts the result into ldap. I manually issued {www,}achernya.com and did some copy-pasting; ideally this could probably be done as a certbot plugin that can talk straight to ldap and also write files out to AFS. This would involve repointing .well-known/acme to a directory in the Scripts locker (or similar).

3. Cronjob to renew the certificates. Possibly this is just "certbot renew" if we're going with the plugin.

4. Pony UI, for whitelisted users only, to opt into (2). This can happen any time after (2) is complete.

5. Replace the reified vhosts with mod_vhost_ldap. This step can happen any time after (1) is complete. See #52 for details.

[dukhovni]

Working on a CGI script to talk to ldap and link certbot to respond to ACME challenges.

[bpchen]

If I'm understanding correctly that this is different from what's above, will poke at generating CSRs, possibly with certbot, possibly not?

[bpchen]

Dropping some links and notes from a little research I did today:

​acme_tiny.py is a nice short implementation we could consider using. In particular, it's really easy to specify an account key, which we must sign all requests to Let's Encrypt with, so we want to share the account key across servers; this behavior might be more desirable than certbot, which I think tries to handle the account key for you somewhat behind-the-scenes.

(The ​README also has useful things like the openssl snippet for CSR generation)

I think the only thing we might need to tweak in acme_tiny.py is the logic where it writes the challenge file (maybe we want to put it into LDAP instead??), and maybe not even that. The HTTP ("http-01") challenge that acme_tiny.py implements is also quite simple and specified in ​section 7.2 of the ACME spec. It only depends on the challenge token that the Let's Encrypt server sends us, and concatenating it with a hash of the account key.

[dukhovni]

I was talking to andreser, and he mentioned that ​https://github.com/andres-erbsen/simple-https-server might be useful. It's a short Go script that's like python's SimpleHTTPServer but for HTTPS, automatically getting certs signed by Let's Encrypt on the fly. A lot of the actual work is done by Go's letsencrypt library, which looks pretty handy. Not sure how this fits in with what we're currently planning, though.

[andersk]

The LDAP schema now includes scriptsVhostCertificate and scriptsVhostCertificateKeyFile (r2787, r2788):

dn: scriptsVhostName=feed.mit.edu,ou=VirtualHosts,dc=scripts,dc=mit,dc=edu
objectClass: top
objectClass: scriptsVhost
scriptsVhostName: feed.mit.edu
scriptsVhostAlias: feed
scriptsVhostAccount: uid=feed,ou=People,dc=scripts,dc=mit,dc=edu
scriptsVhostDirectory: .
scriptsVhostCertificate: MIIFqjCCBJKgAwIBAgIRANEdkdUX1hR+1P+Gym1cY/MwDQYJKoZIhvcNAQELBQAwdjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAk1JMRIwEAYDVQQHEwlBbm4gQXJib3IxEjAQBgNVBAoTCUludGVybmV0MjERMA8GA1UECxMISW5Db21tb24xHzAdBgNVBAMTFkluQ29tbW9uIFJTQSBTZXJ2ZXIgQ0EwHhcNMTQxMDA2MDAwMDAwWhcNMTcxMDA1MjM1OTU5WjCB5jELMAkGA1UEBhMCVVMxDjAMBgNVBBETBTAyMTM5MQswCQYDVQQIEwJNYTESMBAGA1UEBxMJQ2FtYnJpZGdlMR0wGwYDVQQJExQ3NyBNYXNzYWNodXNldHRzIEF2ZTEuMCwGA1UEChMlTWFzc2FjaHVzZXR0cyBJbnN0aXR1dGUgb2YgVGVjaG5vbG9neTEqMCgGA1UECxQhSW5mb3JtYXRpb24gU2VydmljZXMgJiBUZWNobm9sb2d5MRQwEgYDVQQLEwtQbGF0aW51bVNTTDEVMBMGA1UEAxMMZmVlZC5taXQuZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxJo8HD63ei1mpkNGQVv3wXXGrQUiDON1FhsJ6iYiCFmxmOKNvFssPlQ48uIBJBNCqLh8EkmnmecSI5kDPlDGy/rq2lZC7OrSfcsNzTP9fXHi5kvYoOS6XuVuLf/yDglp7T/7qcrMPXX4KBDcaILnEH9Y8Leg8UBVf2xGgSF+Pe62oTR7BX8+g9TUUp6pdycdwr6JCwJaRKnokys2CksYyOlVdOZByV0ZVHZjVC4uOwn72GfLJEdni7wYZ76tgWXW2c1l3j09wL47BfBtDq3W9STke5HS2SRvWh/U2tDLIzO+mzBR1mrkk+gs8XGC919jFXQzBqDNrmUmrtT6YrSAHwIDAQABo4IBwDCCAbwwHwYDVR0jBBgwFoAUHgWjd49sluJbh0umtIascQAM5zgwHQYDVR0OBBYEFPuoJjTiDIZLN0PM6/KN7akyBw1kMA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjBnBgNVHSAEYDBeMFIGDCsGAQQBriMBBAMBATBCMEAGCCsGAQUFBwIBFjRodHRwczovL3d3dy5pbmNvbW1vbi5vcmcvY2VydC9yZXBvc2l0b3J5L2Nwc19zc2wucGRmMAgGBmeBDAECAjBEBgNVHR8EPTA7MDmgN6A1hjNodHRwOi8vY3JsLmluY29tbW9uLXJzYS5vcmcvSW5Db21tb25SU0FTZXJ2ZXJDQS5jcmwwdQYIKwYBBQUHAQEEaTBnMD4GCCsGAQUFBzAChjJodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vSW5Db21tb25SU0FTZXJ2ZXJDQV8yLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAXBgNVHREEEDAOggxmZWVkLm1pdC5lZHUwDQYJKoZIhvcNAQELBQADggEBAC5BLa3fvJGa7oVLbMUwK6pnqLGeClw5fkHRKG13w1MXMhUfG1Y51p5QDZrH7ep+6R+fdOVmXAXngs8fng32MMO+Wx29Tap1bWuLD8oT+5aUCR5wOnqBHguNRGWhY/ZJuAEAoA0CLFO2P8kW0baGOsvs7FGPGu1GcRfwem0xzTQqFW7VDgFhVA5U4NT+ekM5vv+j0Gv9NB70VTV1qzlHsyJYnXk6dtigUbsVmK3vqjRdHSY+DKHOlCknP9LhgjVGftAFg+7fOCO2M/oYi362k2KbXadeETbeqbnBoJV5CrlbglbvN7kwfMVOK3vVj5w78jGVBoINVvWcTCaCpjwUZ10= MIIF+TCCA+GgAwIBAgIQRyDQ+oVGGn4XoWQCkYRjdDANBgkqhkiG9w0BAQwFADCBiDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQxMDA2MDAwMDAwWhcNMjQxMDA1MjM1OTU5WjB2MQswCQYDVQQGEwJVUzELMAkGA1UECBMCTUkxEjAQBgNVBAcTCUFubiBBcmJvcjESMBAGA1UEChMJSW50ZXJuZXQyMREwDwYDVQQLEwhJbkNvbW1vbjEfMB0GA1UEAxMWSW5Db21tb24gUlNBIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJwb8bsvf2MYFVFRVA+exU5NEFj6MJsXKZDmMwysE1N8VJG06thum4ltuzM+j9INpun5uukNDBqeso7JcC7vHgV9lestjaKpTbOc5/MZNrun8XzmCB5hJ0R6lvSoNNviQsil2zfVtefkQnI/tBPPiwckRR6MkYNGuQmm/BijBgLsNI0yZpUn6uGX6Ns1oytW61fo8BBZ321wDGZq0GTlqKOYMa0dYtX6kuOaQ80tNfvZnjNbRX3EhigsZhLI2w8ZMA0/6fDqSl5AB8f2IHpTeIFken5FahZv9JNYyWL7KSd9oX8hzudPR9aKVuDjZvjs3YncJowZaDuNi+L7RyMLfzcCAwEAAaOCAW4wggFqMB8GA1UdIwQYMBaAFFN5v1qqK0rPVIDh2JvAnfKyA2bLMB0GA1UdDgQWBBQeBaN3j2yW4luHS6a0hqxxAAznODAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwGwYDVR0gBBQwEjAGBgRVHSAAMAgGBmeBDAECAjBQBgNVHR8ESTBHMEWgQ6BBhj9odHRwOi8vY3JsLnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwdgYIKwYBBQUHAQEEajBoMD8GCCsGAQUFBzAChjNodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQWRkVHJ1c3RDQS5jcnQwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggIBAC0RBjjW29dYaK+qOGcXjeIT16MUJNkGE+vrkS/fT2ctyNMU11ZlUp5uH5gIjppIG8GLWZqjV5vbhvhZQPwZsHURKsISNrqOcooGTie3jVgU0W+0+Wj8mN2knCVANt69F2YrA394gbGAdJ5fOrQmL2pIhDY0jqco74fzYefbZ/VS29fR5jBxu4uj1P+5ZImem4Gbj1e4ZEzVBhmO55GFfBjRidj26h1oFBHZ7heDH1Bjzw72hipu47Gkyfr2NEx3KoCGMLCj3Btx7ASn5Ji8FoU+hCazwOU1VX55mKPU1I2250LoRCASN18JyfsD5PVldJbtyrmz9gn/TKbRXTr80U2q5JhyvjhLf4lOJo/UzL5WCXEDSmyj4jWG3R7Z8TED9xNNCxGBMXnMete+3PvzdhssvbORDwBZByogQ9xL2LUZFI/ieoQp0UM/L8zfP527vWjEzuDN5xwxMnhi+vCToh7J159o5ah29mP+aJnvujbXEnGanrNxHzu+AGOePV8hwrGGG7hOIcPDQwkuYwzN/xT29iLp/cqf9ZhEtkGcQcIImH3boJ8ifsCnSbu0GB9L06Yqh7lcyvKDTEADslIaeSEINxhO2Y1fmcYFX/Fqrrp1WnhHOjplXuXE0OPa0utaKC25Aplgom88L2Z8mEWcyfoB7zKOfD759AN7JKZWCYwk MIIFdzCCBF+gAwIBAgIQE+oocFv07O0MNmMJgGFDNjANBgkqhkiG9w0BAQwFADBvMQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowgYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgBJlFzYOw9sIs9CsVw127c0n00ytUINh4qogTQktZAnczomfzD2p7PbPwdzx07HWezcoEStH2jnGvDoZtF+mvX2do2NCtnbyqTsrkfjib9DsFiCQCT7i6HTJGLSR1GJk23+jBvGIGGqQIjy8/hPwhxR79uQfjtTkUcYRZ0YIUcuGFFQ/vDP+fmyc/xadGL1RjjWmp2bIcmfbIWax1Jt4A8BQOujM8Ny8nkz+rwWWNR9XWrf/zvk9tyy29lTdyOcSOk2uTIq3XJq0tyA9yn8iNK5+O2hmAUTnAU5GU5szYPeUvlM3kHND8zLDU+/bqv50TmnHa4xgk97Exwzf4TKuzJM7UXiVZ4vuPVb+DNBpDxsP8yUmazNt925H+nND5X4OpWaxKXwyhGNVicQNwZNUMBkTrNN9N6frXTpsNVzbQdcS2qlJC9/YgIoJk2KOtWbPJYjNhLixP6Q5D9kCnusSTJV882sFqV4Wg8y4Z+LoE53MW4LTTLPtW//e5XOsIzstAL81VXQJSdhJWBp/kjbmUZIO8yZ9HE0XvMnsQybQv0FfQKlERPSZ51eHnlAfV1SoPv10Yy+xUGUJ5lhCLkMaTLTwJUdZ+gQek9QmRkpQgbLevni3/GcV4clXhB4PY9bpYrrWX1Uu6lzGKAgEJTm4Diup8kyXHAc/DVL17e8vgg8CAwEAAaOB9DCB8TAfBgNVHSMEGDAWgBStvZh6NLQm9/rEJlTvA73gJMtUGjAdBgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wEQYDVR0gBAowCDAGBgRVHSAAMEQGA1UdHwQ9MDswOaA3oDWGM2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9BZGRUcnVzdEV4dGVybmFsQ0FSb290LmNybDA1BggrBgEFBQcBAQQpMCcwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggEBAJNl9jeDlQ9ew4IcH9Z35zyKwKoJ8OkLJvHgwmp1ocd5yblSYMgpEg7wrQPWCcR23+WmgZWnRtqCV6mVksW2jwMibDN3wXsyF24HzloUQToFJBv2FAY7qCUkDrvMKnXduXBBP3zQYzYhBx9G/2CkkeFnvN4ffhkUyWNnkepnB2u0j4vAbkN9w6GAbLIevFOFfdyQoaS8Le9Gclc1Bb+7RrtubTeZtv8jkpHGbkD4jylW6l/VXxRTrPBPYer3IsynVgviuDQfJtl7GQVoP7o81DgGotPmjw7jtHFtQELFhLRAlSv0ZaBIefYdgWOWnU914Ph85I6p0fKtirOMxyHNwu8=
scriptsVhostCertificateKeyFile: scripts-2048.key

The format of scriptsVhostCertificate is a space-separated list of base-64 encoded DER blobs (also known as PEM minus the -----(BEGIN|END) CERTIFICATE----- markers and minus newlines), one for each certificate in the chain.

jakobw imported all our custom certificates into LDAP.

I wrote a script /etc/httpd/export-scripts-certs to generate the Apache configuration and PEM files from LDAP (r2791), and removed the certificates from version control (r2792).

Then I wrote a script locker/sbin/vhostcert for converting certificate chains between PEM format and the LDAP format (r2793). It also automatically puts the chain in the right order and deletes the self-signed root.

[jakobw]

I made a modified version ot acme_tiny.py that we should be cool with using, and I wrote the skeleton of a script that would do the renewing.

Can someone take a look at it?

​https://github.mit.edu/jakobw/acme_tiny

EDIT - I looked at this 6 months later, and it doesn't work, and I am confused at 6-months-ago-me.

[andersk]

I added sanity checks to /etc/httpd/export-scripts-certs to make sure we don’t generate configuration that will make Apache fail to load (r2813, r2821). That will let us run it automatically, although we’ll still want to limit the frequency of Apache reloads to avoid connection delays.

[andersk]

/etc/httpd/export-scripts-certs now reloads Apache itself if anything changed (r2825), and is now run automatically 38 minutes after every hour.

[jakobw]

for actually doing the LE cert getting, a plan to implement it using the acme_tiny.py script linked by brian above:

1. tell apache to send .well-known/challenge in everyone's hostnames to the challenge folder (I think this is a small change)
2. give pony permission to sudo a script that runs andersk's CSR script followed by acme_tiny.py (I am willing to write said script)
3. have a pony endpoint where users can go and it will run said script and load the resulting cert into LDAP (I am willing to implement this)
4. automate renewals: cron task that runs the above script for all enrolled domains every 80 days (having a cron task able to sudo seems sketchy to me but maybe this is the right answer - looking for feedback.)

Does this sound reasonable? This is basically a slightly more specific rewriting of achernya's 2-4 above. If someone says it looks okay, I'll do 2 and 3.

[andersk]

Replying to jakobw:

.well-known/challenge can be served by a script (such as a Pony webapp endpoint); it doesn’t need to be a physical folder. This way we can generate challenge responses dynamically as they are requested.

Pony already has sudo permissions on gencsr-pony. Why does acme_tiny need sudo permissions?

Please don’t try to renew all our certificates simultaneously—that sounds like a great way to run into Let’s Encrypt quotas with very little time left to fix things. We should have an hourly or daily job that searches for certificates less than 30 days from expiration and renews those ones.

[jakobw]

acme_tiny, or whatever else implements LE http-01, needs to be able to sign things using an "authorized key pair" for the account, in order to do certificate management. Said authorized key pair seems like a thing we probably want to restrict access to, but I guess it doesn't need to be only root.

[jakobw]

w/r/t .well-known/challenge, it needs to be able to serve a specific response determined by a request already sent to the LE server.

1. We send request to LE server for verification
2. They send a challenge
3. We do some computation on the challenge and host the response at .well-known/challenge
4. We send them a "we're ready"
5. They send a request to .well-known/challenge/ahashdeterminedbythechallengeabove and get back our response.
6. We send them a request for a cert.

This means storing it somewhere like the AFS or LDAP (or in the SQL db?). A directory in the pony locker seems easiest to me, but I don't know how easy it is to update LDAP schema and whether we want to do that.

[jakobw]

I implemented the pony portion of this in ​https://github.com/jakob223/scripts-pony/tree/letsencrypt - I will write a cron script for it tomorrow evening. It currently still uses the filesystem to store challenges and assumes that the account key is also in the filesystem somewhere.

[jakobw]

I ~implemented 2, 3, and 4 from the above list in ​https://github.com/jakob223/scripts-pony/tree/letsencrypt and I'd love a code review at some point. There are still some things that need ironing out.

[andersk]

By necessity, I made a bunch of ​progress this week toward automatic mitcert integration (#380), some of which will be relevant here. Pony now has the permissions needed to install certificates to LDAP, and there’s now a scripts.cert library with useful functions (some of which were extracted from vhostcert). Also find_expiring_certs.py has much of the logic we’ll want to decide which certificates to renew.

[andersk]

[deleted]