Using ACME for automated SSL/TLS certificate management

Background

The Automated Certificate Management Environment (ACME), is a protocol that makes it possible to automate the issuance and renewal of certificates.  ACME is primarily used to obtain domain validated (DV) certificates and is commonly done using the free Let's Encrypt certificate authority (CA) service.  Alternatively, organization validated (OV) certs may be obtained when using the Sectigo / InCommon CA's ACME service.   There is no substantive difference between the types of certificates; however your choice of CA may be influenced by your server's network configuration.

Recommendations

Internet-available sites

If your web server is open to the Internet (even if it requires authentication), the simplest and most secure option for automating certificates using ACME is to select one of the free CAs that provide ACME support.  For example, both Let's Encrypt and ZeroSSL provide free and trusted SSL certificates as long as your web server can be reached via HTTP on the Internet.

Firewalled sites

If you manage web servers that are not open to the Internet, or if you manage a mix of open/firewalled servers and want a consistent configuration and/or process then you can use the Sectigo/InCommon CA's ACME service.  The InCommon CA provides OV certificates via ACME but requires a more complex configuration than the public CAs listed above.

Protecting ACME secrets

Unlike Let's Encrypt implementation of ACME, Sectigo's implementation provides ACME secrets that cover many domains, not a single domain. This means the ACME secrets need to be handled with exceptional care because if they are compromised they can be used to request an unlimited number of certificates for any of the delegated domains assigned to the ACME account.

As a result, secrets should not be shared over insecure communication methods or stored in environments that are not expected to be secure. Access to systems that contain ACME secrets should be limited to only those who require it. One recommended way to do this is to configure a centralized server that will request and share out the certificates as needed.

Centralized vs Decentralized Management

When using Sectigo's ACME service, it is highly encouraged that users implement a centralized solution for certificate requests and distribution, especially if you are responsible for requesting a large number of certificates.  This has the benefit of protecting your ACME secrets by limiting the number of locations they are saved.

A centralized management approach requires configuring a server/container to act as a central server that will request all certificates, and scripts or configuration management tools (e.g. Puppet, Ansible) to distribute and install certificates from the central server to the application hosts. If you run Windows IIS another potential solution is to use IIS Centralized SSL Certificate Support.

ACME Clients

Sectigo recommends Certbot as their supported ACME client.  The Electronic Frontier Foundation provides excellent instructions for installing Certbot on your server.  Please visit https://certbot.eff.org/instructions and follow the instructions based on your web server software and operating system.

In addition, "unsupported" clients such as win-acme are known to work well.

For reverse proxies, Apache now supports ACME natively.  The Caddy web server is performant and easy to configure with built-in ACME and support.  Certbot has integrations for Nginx.

Process for Let’s Encrypt or ZeroSSL

For more information see:

https://letsencrypt.org/getting-started/

https://help.zerossl.com/hc/en-us/categories/360005948793-Getting-Started

Process for Sectigo/InCommon

Prerequisites

• You must be a Departmental Certificate Administrator (DCA) for your department
• For each FQDN requiring an SSL certificate, the domain must be delegated to your department
• A technical comfort to research how certbot (or other ACME clients) work in your environment

Overview

This process involves creating an ACME account for each host or group of hosts for an application.  The goal is to limit the number of domains assigned to a given ACME account.  While you may use a single ACME account for all of your delegated domains, it creates a shared-fate risk if an ACME secret were exposed.  See Protecting ACME secrets above. The process outlined below attempts to remove the shared-fate risk associated with EAB keys.  For example:

Application myapp1.dept.berkeley.edu may have several domains:

myapp1.dept.berkeley.edu
host1.myapp1.dept.berkeley.edu
host2.myapp1.dept.berkeley.edu

You also manage myapp2.dept.berkeley.edu with a similar set of domain names.   In this case you would create 2 ACME accounts, one for each application: dept_myapp1 and dept_myapp2.  You would then scope each ACME account to the relevant domains and distribute the EAB key for myapp1 only to the systems hosting that site.

Steps

1. Log into the Cert-Manager site.
2. Navigate to Domains and expand berkeley.edu to see your list of delegated domains.
3. Further expand the domain for which you will configure ACME.  If you do not see the FQDNs of the certificates you will be creating via ACME you will need to add them. 
   
   NOTE:  This is normally not required for manual certificate requests as long as you have the top-level domain.  To secure your ACME accounts, you need to add all of the FQDNs as listed domains
   
   
4. In this example we will use ACME to issue certificates to app1.training.berkeley.edu but it's not listed, so we will add it now.
   
   
   
   
5. Click the green plus button in the upper right corner of the screen and enter the FQDN.  Then check the box for SSL Certificate next to your department and Save the domain.
   
   
6. Add any additional FQDNs that will be requested by ACME.
7. Navigate to Enrollment > ACME.
8. Select the checkbox next to https://acme.sectigo.com/v2/InCommonRSAOV.
9. Select Accounts.
10. Click the green plus button in the upper right corner of the screen to add a new ACME account.
11. Enter a name for the account, for this example something like training_app1 would make sense.
12. Note that your department is automatically filled in.
13. Next, click the plus sign next to Domains.
    
    
    
    
14. Now add any FQDNs for this ACME account / endpoint.  Note that wildcard domains should never be assigned to an ACME account.  In this example both app1.training.berkeley.edu and acme-test01.training.berkeley.edu are used by the application so have been added to this ACME account.  Save the account.
    
    
    
    
15. The next screen will display the ACME account's External Account Binding (EAB) details.  Please treat these as sensitive secrets!  These should be stored in some sort of password manager or vault and only exist in the ACME configuration for your system(s).  You will need the ACME URL, Key ID, and HMAC key for your ACME client.  See the Minimum Security Standards for Networked Devices (MSSND), NIST guidelines NIST SP 800-63B, Sec. 5.1.1.2, and UC Encryption Key and Certificate Management Standard for guidance.
    
    
16. You can now configure an ACME client that supports EAB.  Currently the only client supported by Sectigo / InCommon is Certbot.
17. The ACME setup isn't complete until you connect to the InCommon/Sectigo ACME server with an ACME client.  See the section "Connect to the Sectigo ACME Service" for more details.

Install Certbot in your environment

Sectigo recommends Certbot as their supported ACME client.  The Electronic Frontier Foundation hosts an excellent set of instructions for installing certbot on your server.  Please visit https://certbot.eff.org/instructions and follow the instructions based on your web server software and operating system.

Connect to the Sectigo ACME Service

In order to connect to the Sectigo ACME Service you'll need to use the command line sequence appropriate for your operating system and selected ACME client.  This is an example using certbot.

Credentials
 
ACME URL: https://acme.sectigo.com/v2/InCommonRSAOV
Key ID: insert  key ID
HMAC Key: insert HMAC key

Certbot requires admin privileges. Here are example commands that will use certbot to connect to the InCommon ACME service, complete your account setup, and get your first certificate.

Depending on your platform, web server, ACME plug-ins, etc, this may not be the way you obtain subsequent certs. See the certbot documentation at https://certbot.eff.org/. 
 
Linux
 
sudo certbot certonly --standalone --non-interactive --agree-tos --email {requestor@berkeley.edu} --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid {Key ID} --eab-hmac-key {HMAC Key} --domain {domain name used as CN for the cert} --cert-name {short friendly name for cert}
 
Windows
 
Start a CMD window as an admin account, then:
 
certbot certonly --standalone --non-interactive --agree-tos --email {requestor@berkeley.edu} --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid {Key ID} --eab-hmac-key {HMAC Key} --domain {domain name used as CN for the cert} --cert-name {short friendly name for cert}
 
Note: Any domains you reference in your ACME request must first be added to InCommon certificate services (if they don't already exist there), then they must be delegated to your department, and then delegated to your departmental ACME account.
This isn't the way the Let's Encrypt service works, but it is the way the InCommon ACME service works.