Opened 9 years ago
Last modified 7 years ago
#419 new enhancement
SSL certificate automation and Let’s Encrypt integration
Reported by: | andersk | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | |
Component: | pony | Keywords: | |
Cc: |
Description
We would like to be able to accept certificates from Let’s Encrypt. Unfortunately, because Let’s Encrypt issues certificates with a 90 day lifetime, this would significantly increase our regular maintenance burden. So we need to develop some kind of automation before allowing this:
- CSR generation. Leaving this as a manual process is probably acceptable for now, since it only needs to happen once per host: CSRs are reusable as long as the key doesn’t change. Automating CSRs is the topic of #241.
- Interface for the user to provide certificates. The user could upload or paste their certificate into Pony which would store it into the vhost’s LDAP record. It would be good to have a token-based API to let the user automate this without client certificates.
- Validation of provided certificates. This is necessary because an invalid certificate can cause Apache to fail to start (although that’s less of an issue with #52). It would also be nice to warn the user about common errors like incomplete or misordered chains.
- Making Apache use the certificates. An ideal solution is #52, but that requires some nontrivial Apache development. Until then, we could write a daily cronjob to copy out the certificates and reload Apache. In fact, the cronjob might as well reify the vhost configurations too, so that we can ditch all the per-site configuration in svn.
- Helping users automate Let’s Encrypt renewals. This might take the form of a script for the user to serve at .well-known/acme-challenge and a line to for the user to add to their crontab.
See also help.mit.edu #3519679. An alternative idea mentioned there is storing certificates as files in the corresponding lockers rather than in LDAP; however, this would make it impossible for the user to get feedback from validation, and would conflict with #52.
Change History (17)
comment:1 Changed 8 years ago by achernya
comment:2 Changed 8 years ago by dukhovni
Working on a CGI script to talk to ldap and link certbot to respond to ACME challenges.
comment:3 Changed 8 years ago by bpchen
If I'm understanding correctly that this is different from what's above, will poke at generating CSRs, possibly with certbot, possibly not?
comment:4 Changed 8 years ago by bpchen
Dropping some links and notes from a little research I did today:
acme_tiny.py is a nice short implementation we could consider using. In particular, it's really easy to specify an account key, which we must sign all requests to Let's Encrypt with, so we want to share the account key across servers; this behavior might be more desirable than certbot, which I think tries to handle the account key for you somewhat behind-the-scenes.
(The README also has useful things like the openssl snippet for CSR generation)
I think the only thing we might need to tweak in acme_tiny.py is the logic where it writes the challenge file (maybe we want to put it into LDAP instead??), and maybe not even that. The HTTP ("http-01") challenge that acme_tiny.py implements is also quite simple and specified in section 7.2 of the ACME spec. It only depends on the challenge token that the Let's Encrypt server sends us, and concatenating it with a hash of the account key.
comment:5 Changed 8 years ago by dukhovni
I was talking to andreser, and he mentioned that https://github.com/andres-erbsen/simple-https-server might be useful. It's a short Go script that's like python's SimpleHTTPServer but for HTTPS, automatically getting certs signed by Let's Encrypt on the fly. A lot of the actual work is done by Go's letsencrypt library, which looks pretty handy. Not sure how this fits in with what we're currently planning, though.
comment:6 Changed 8 years ago by andersk
The LDAP schema now includes scriptsVhostCertificate and scriptsVhostCertificateKeyFile (r2787, r2788):
dn: scriptsVhostName=feed.mit.edu,ou=VirtualHosts,dc=scripts,dc=mit,dc=edu objectClass: top objectClass: scriptsVhost scriptsVhostName: feed.mit.edu scriptsVhostAlias: feed scriptsVhostAccount: uid=feed,ou=People,dc=scripts,dc=mit,dc=edu scriptsVhostDirectory: . scriptsVhostCertificate: MIIFqjCCBJKgAwIBAgIRANEdkdUX1hR+1P+Gym1cY/MwDQYJKoZIhvcNAQELBQAwdjELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAk1JMRIwEAYDVQQHEwlBbm4gQXJib3IxEjAQBgNVBAoTCUludGVybmV0MjERMA8GA1UECxMISW5Db21tb24xHzAdBgNVBAMTFkluQ29tbW9uIFJTQSBTZXJ2ZXIgQ0EwHhcNMTQxMDA2MDAwMDAwWhcNMTcxMDA1MjM1OTU5WjCB5jELMAkGA1UEBhMCVVMxDjAMBgNVBBETBTAyMTM5MQswCQYDVQQIEwJNYTESMBAGA1UEBxMJQ2FtYnJpZGdlMR0wGwYDVQQJExQ3NyBNYXNzYWNodXNldHRzIEF2ZTEuMCwGA1UEChMlTWFzc2FjaHVzZXR0cyBJbnN0aXR1dGUgb2YgVGVjaG5vbG9neTEqMCgGA1UECxQhSW5mb3JtYXRpb24gU2VydmljZXMgJiBUZWNobm9sb2d5MRQwEgYDVQQLEwtQbGF0aW51bVNTTDEVMBMGA1UEAxMMZmVlZC5taXQuZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxJo8HD63ei1mpkNGQVv3wXXGrQUiDON1FhsJ6iYiCFmxmOKNvFssPlQ48uIBJBNCqLh8EkmnmecSI5kDPlDGy/rq2lZC7OrSfcsNzTP9fXHi5kvYoOS6XuVuLf/yDglp7T/7qcrMPXX4KBDcaILnEH9Y8Leg8UBVf2xGgSF+Pe62oTR7BX8+g9TUUp6pdycdwr6JCwJaRKnokys2CksYyOlVdOZByV0ZVHZjVC4uOwn72GfLJEdni7wYZ76tgWXW2c1l3j09wL47BfBtDq3W9STke5HS2SRvWh/U2tDLIzO+mzBR1mrkk+gs8XGC919jFXQzBqDNrmUmrtT6YrSAHwIDAQABo4IBwDCCAbwwHwYDVR0jBBgwFoAUHgWjd49sluJbh0umtIascQAM5zgwHQYDVR0OBBYEFPuoJjTiDIZLN0PM6/KN7akyBw1kMA4GA1UdDwEB/wQEAwIFoDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjBnBgNVHSAEYDBeMFIGDCsGAQQBriMBBAMBATBCMEAGCCsGAQUFBwIBFjRodHRwczovL3d3dy5pbmNvbW1vbi5vcmcvY2VydC9yZXBvc2l0b3J5L2Nwc19zc2wucGRmMAgGBmeBDAECAjBEBgNVHR8EPTA7MDmgN6A1hjNodHRwOi8vY3JsLmluY29tbW9uLXJzYS5vcmcvSW5Db21tb25SU0FTZXJ2ZXJDQS5jcmwwdQYIKwYBBQUHAQEEaTBnMD4GCCsGAQUFBzAChjJodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vSW5Db21tb25SU0FTZXJ2ZXJDQV8yLmNydDAlBggrBgEFBQcwAYYZaHR0cDovL29jc3AudXNlcnRydXN0LmNvbTAXBgNVHREEEDAOggxmZWVkLm1pdC5lZHUwDQYJKoZIhvcNAQELBQADggEBAC5BLa3fvJGa7oVLbMUwK6pnqLGeClw5fkHRKG13w1MXMhUfG1Y51p5QDZrH7ep+6R+fdOVmXAXngs8fng32MMO+Wx29Tap1bWuLD8oT+5aUCR5wOnqBHguNRGWhY/ZJuAEAoA0CLFO2P8kW0baGOsvs7FGPGu1GcRfwem0xzTQqFW7VDgFhVA5U4NT+ekM5vv+j0Gv9NB70VTV1qzlHsyJYnXk6dtigUbsVmK3vqjRdHSY+DKHOlCknP9LhgjVGftAFg+7fOCO2M/oYi362k2KbXadeETbeqbnBoJV5CrlbglbvN7kwfMVOK3vVj5w78jGVBoINVvWcTCaCpjwUZ10= MIIF+TCCA+GgAwIBAgIQRyDQ+oVGGn4XoWQCkYRjdDANBgkqhkiG9w0BAQwFADCBiDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQxMDA2MDAwMDAwWhcNMjQxMDA1MjM1OTU5WjB2MQswCQYDVQQGEwJVUzELMAkGA1UECBMCTUkxEjAQBgNVBAcTCUFubiBBcmJvcjESMBAGA1UEChMJSW50ZXJuZXQyMREwDwYDVQQLEwhJbkNvbW1vbjEfMB0GA1UEAxMWSW5Db21tb24gUlNBIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJwb8bsvf2MYFVFRVA+exU5NEFj6MJsXKZDmMwysE1N8VJG06thum4ltuzM+j9INpun5uukNDBqeso7JcC7vHgV9lestjaKpTbOc5/MZNrun8XzmCB5hJ0R6lvSoNNviQsil2zfVtefkQnI/tBPPiwckRR6MkYNGuQmm/BijBgLsNI0yZpUn6uGX6Ns1oytW61fo8BBZ321wDGZq0GTlqKOYMa0dYtX6kuOaQ80tNfvZnjNbRX3EhigsZhLI2w8ZMA0/6fDqSl5AB8f2IHpTeIFken5FahZv9JNYyWL7KSd9oX8hzudPR9aKVuDjZvjs3YncJowZaDuNi+L7RyMLfzcCAwEAAaOCAW4wggFqMB8GA1UdIwQYMBaAFFN5v1qqK0rPVIDh2JvAnfKyA2bLMB0GA1UdDgQWBBQeBaN3j2yW4luHS6a0hqxxAAznODAOBgNVHQ8BAf8EBAMCAYYwEgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwGwYDVR0gBBQwEjAGBgRVHSAAMAgGBmeBDAECAjBQBgNVHR8ESTBHMEWgQ6BBhj9odHRwOi8vY3JsLnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwdgYIKwYBBQUHAQEEajBoMD8GCCsGAQUFBzAChjNodHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQWRkVHJ1c3RDQS5jcnQwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggIBAC0RBjjW29dYaK+qOGcXjeIT16MUJNkGE+vrkS/fT2ctyNMU11ZlUp5uH5gIjppIG8GLWZqjV5vbhvhZQPwZsHURKsISNrqOcooGTie3jVgU0W+0+Wj8mN2knCVANt69F2YrA394gbGAdJ5fOrQmL2pIhDY0jqco74fzYefbZ/VS29fR5jBxu4uj1P+5ZImem4Gbj1e4ZEzVBhmO55GFfBjRidj26h1oFBHZ7heDH1Bjzw72hipu47Gkyfr2NEx3KoCGMLCj3Btx7ASn5Ji8FoU+hCazwOU1VX55mKPU1I2250LoRCASN18JyfsD5PVldJbtyrmz9gn/TKbRXTr80U2q5JhyvjhLf4lOJo/UzL5WCXEDSmyj4jWG3R7Z8TED9xNNCxGBMXnMete+3PvzdhssvbORDwBZByogQ9xL2LUZFI/ieoQp0UM/L8zfP527vWjEzuDN5xwxMnhi+vCToh7J159o5ah29mP+aJnvujbXEnGanrNxHzu+AGOePV8hwrGGG7hOIcPDQwkuYwzN/xT29iLp/cqf9ZhEtkGcQcIImH3boJ8ifsCnSbu0GB9L06Yqh7lcyvKDTEADslIaeSEINxhO2Y1fmcYFX/Fqrrp1WnhHOjplXuXE0OPa0utaKC25Aplgom88L2Z8mEWcyfoB7zKOfD759AN7JKZWCYwk MIIFdzCCBF+gAwIBAgIQE+oocFv07O0MNmMJgGFDNjANBgkqhkiG9w0BAQwFADBvMQswCQYDVQQGEwJTRTEUMBIGA1UEChMLQWRkVHJ1c3QgQUIxJjAkBgNVBAsTHUFkZFRydXN0IEV4dGVybmFsIFRUUCBOZXR3b3JrMSIwIAYDVQQDExlBZGRUcnVzdCBFeHRlcm5hbCBDQSBSb290MB4XDTAwMDUzMDEwNDgzOFoXDTIwMDUzMDEwNDgzOFowgYgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpOZXcgSmVyc2V5MRQwEgYDVQQHEwtKZXJzZXkgQ2l0eTEeMBwGA1UEChMVVGhlIFVTRVJUUlVTVCBOZXR3b3JrMS4wLAYDVQQDEyVVU0VSVHJ1c3QgUlNBIENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAgBJlFzYOw9sIs9CsVw127c0n00ytUINh4qogTQktZAnczomfzD2p7PbPwdzx07HWezcoEStH2jnGvDoZtF+mvX2do2NCtnbyqTsrkfjib9DsFiCQCT7i6HTJGLSR1GJk23+jBvGIGGqQIjy8/hPwhxR79uQfjtTkUcYRZ0YIUcuGFFQ/vDP+fmyc/xadGL1RjjWmp2bIcmfbIWax1Jt4A8BQOujM8Ny8nkz+rwWWNR9XWrf/zvk9tyy29lTdyOcSOk2uTIq3XJq0tyA9yn8iNK5+O2hmAUTnAU5GU5szYPeUvlM3kHND8zLDU+/bqv50TmnHa4xgk97Exwzf4TKuzJM7UXiVZ4vuPVb+DNBpDxsP8yUmazNt925H+nND5X4OpWaxKXwyhGNVicQNwZNUMBkTrNN9N6frXTpsNVzbQdcS2qlJC9/YgIoJk2KOtWbPJYjNhLixP6Q5D9kCnusSTJV882sFqV4Wg8y4Z+LoE53MW4LTTLPtW//e5XOsIzstAL81VXQJSdhJWBp/kjbmUZIO8yZ9HE0XvMnsQybQv0FfQKlERPSZ51eHnlAfV1SoPv10Yy+xUGUJ5lhCLkMaTLTwJUdZ+gQek9QmRkpQgbLevni3/GcV4clXhB4PY9bpYrrWX1Uu6lzGKAgEJTm4Diup8kyXHAc/DVL17e8vgg8CAwEAAaOB9DCB8TAfBgNVHSMEGDAWgBStvZh6NLQm9/rEJlTvA73gJMtUGjAdBgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMBAf8wEQYDVR0gBAowCDAGBgRVHSAAMEQGA1UdHwQ9MDswOaA3oDWGM2h0dHA6Ly9jcmwudXNlcnRydXN0LmNvbS9BZGRUcnVzdEV4dGVybmFsQ0FSb290LmNybDA1BggrBgEFBQcBAQQpMCcwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZIhvcNAQEMBQADggEBAJNl9jeDlQ9ew4IcH9Z35zyKwKoJ8OkLJvHgwmp1ocd5yblSYMgpEg7wrQPWCcR23+WmgZWnRtqCV6mVksW2jwMibDN3wXsyF24HzloUQToFJBv2FAY7qCUkDrvMKnXduXBBP3zQYzYhBx9G/2CkkeFnvN4ffhkUyWNnkepnB2u0j4vAbkN9w6GAbLIevFOFfdyQoaS8Le9Gclc1Bb+7RrtubTeZtv8jkpHGbkD4jylW6l/VXxRTrPBPYer3IsynVgviuDQfJtl7GQVoP7o81DgGotPmjw7jtHFtQELFhLRAlSv0ZaBIefYdgWOWnU914Ph85I6p0fKtirOMxyHNwu8= scriptsVhostCertificateKeyFile: scripts-2048.key
The format of scriptsVhostCertificate is a space-separated list of base-64 encoded DER blobs (also known as PEM minus the -----(BEGIN|END) CERTIFICATE----- markers and minus newlines), one for each certificate in the chain.
jakobw imported all our custom certificates into LDAP.
I wrote a script /etc/httpd/export-scripts-certs to generate the Apache configuration and PEM files from LDAP (r2791), and removed the certificates from version control (r2792).
Then I wrote a script locker/sbin/vhostcert for converting certificate chains between PEM format and the LDAP format (r2793). It also automatically puts the chain in the right order and deletes the self-signed root.
comment:7 Changed 8 years ago by jakobw
I made a modified version ot acme_tiny.py that we should be cool with using, and I wrote the skeleton of a script that would do the renewing.
Can someone take a look at it?
https://github.mit.edu/jakobw/acme_tiny
EDIT - I looked at this 6 months later, and it doesn't work, and I am confused at 6-months-ago-me.
comment:8 Changed 8 years ago by andersk
comment:9 Changed 8 years ago by andersk
/etc/httpd/export-scripts-certs now reloads Apache itself if anything changed (r2825), and is now run automatically 38 minutes after every hour.
comment:10 follow-up: ↓ 11 Changed 7 years ago by jakobw
for actually doing the LE cert getting, a plan to implement it using the acme_tiny.py script linked by brian above:
- tell apache to send .well-known/challenge in everyone's hostnames to the challenge folder (I think this is a small change)
- give pony permission to sudo a script that runs andersk's CSR script followed by acme_tiny.py (I am willing to write said script)
- have a pony endpoint where users can go and it will run said script and load the resulting cert into LDAP (I am willing to implement this)
- automate renewals: cron task that runs the above script for all enrolled domains every 80 days (having a cron task able to sudo seems sketchy to me but maybe this is the right answer - looking for feedback.)
Does this sound reasonable? This is basically a slightly more specific rewriting of achernya's 2-4 above. If someone says it looks okay, I'll do 2 and 3.
comment:11 in reply to: ↑ 10 Changed 7 years ago by andersk
Replying to jakobw:
.well-known/challenge can be served by a script (such as a Pony webapp endpoint); it doesn’t need to be a physical folder. This way we can generate challenge responses dynamically as they are requested.
Pony already has sudo permissions on gencsr-pony. Why does acme_tiny need sudo permissions?
Please don’t try to renew all our certificates simultaneously—that sounds like a great way to run into Let’s Encrypt quotas with very little time left to fix things. We should have an hourly or daily job that searches for certificates less than 30 days from expiration and renews those ones.
comment:12 Changed 7 years ago by jakobw
acme_tiny, or whatever else implements LE http-01, needs to be able to sign things using an "authorized key pair" for the account, in order to do certificate management. Said authorized key pair seems like a thing we probably want to restrict access to, but I guess it doesn't need to be only root.
comment:13 Changed 7 years ago by jakobw
w/r/t .well-known/challenge, it needs to be able to serve a specific response determined by a request already sent to the LE server.
- We send request to LE server for verification
- They send a challenge
- We do some computation on the challenge and host the response at .well-known/challenge
- We send them a "we're ready"
- They send a request to .well-known/challenge/ahashdeterminedbythechallengeabove and get back our response.
- We send them a request for a cert.
This means storing it somewhere like the AFS or LDAP. A directory in the pony locker seems easiest to me, but I don't know how easy it is to update LDAP schema and whether we want to do that.
comment:14 Changed 7 years ago by jakobw
I implemented the pony portion of this in https://github.com/jakob223/scripts-pony/tree/letsencrypt - I will write a cron script for it tomorrow evening. It currently still uses the filesystem to store challenges and assumes that the account key is also in the filesystem somewhere.
comment:15 Changed 7 years ago by jakobw
I ~implemented 2, 3, and 4 from the above list in https://github.com/jakob223/scripts-pony/tree/letsencrypt and I'd love a code review at some point. There are still some things that need ironing out.
comment:16 Changed 7 years ago by andersk
By necessity, I made a bunch of progress this week toward automatic mitcert integration (#380), some of which will be relevant here. Pony now has the permissions needed to install certificates to LDAP, and there’s now a scripts.cert library with useful functions (some of which were extracted from vhostcert). Also find_expiring_certs.py has much of the logic we’ll want to decide which certificates to renew.
comment:17 Changed 7 years ago by andersk
[deleted]
Proposed steps to implement this, given that #52 is well in progress.