A knowledge base article about Using ACME for automated SSL/TLS certificate management provided by the UC Berkeley IT Service Hub - Knowledge Portal

Background

The Automated Certificate Management Environment (ACME), is a protocol that makes it possible to automate the issuance and renewal of certificates. ACME is primarily used to obtain domain validated (DV) certificates and is commonly done using the free Let's Encrypt certificate authority (CA) service. Alternatively, organization validated (OV) certs may be obtained when using the Sectigo / InCommon CA's ACME service. There is no substantive difference between the types of certificates; however your choice of CA may be influenced by your server's network configuration.

	ACME + Sectigo / InCommon	ACME + Let’s Encrypt
Maximum certificate validity	365 days	90 days
Who can use it?	Berkeley Departments	Anyone
Obtain and install certificates fully automatically	Yes	Yes
Renew certificates fully automatically	Yes	Yes
Does each ACME client need to successfully complete a challenge for Domain Control Validation?	No, domains are pre-validated when they are delegated	Yes
ACME credentials used to initially set up a new server or device	Obtained and installed manually	Obtained automatically
Central management and reporting	Yes, for departments that have access to the InCommon Certificate Manager web app	No
University-wide support contract	Yes	No
Staging environment	No	Yes

Recommendations

Internet-available sites

If your web server is open to the Internet (even if it requires authentication), the simplest and most secure option for automating certificates using ACME is to select one of the free CAs that provide ACME support. For example, both Let's Encrypt and ZeroSSL provide free and trusted SSL certificates as long as your web server can be reached via HTTP on the Internet.

Firewalled sites

If you manage web servers that are not open to the Internet, or if you manage a mix of open/firewalled servers and want a consistent configuration and/or process then you can use the Sectigo/InCommon CA's ACME service. The InCommon CA provides OV certificates via ACME but requires a more complex configuration than the public CAs listed above.

Protecting ACME secrets

Unlike the Let's Encrypt implementation of ACME, Sectigo's implementation provides ACME secrets that can cover many domains, not just a single domain. This means the ACME secrets need to be handled with exceptional care because if they are compromised they can be used to request an unlimited number of certificates for any of the delegated domains assigned to the ACME account.

As a result, secrets must not be shared over insecure communication methods or stored in environments that are not expected to be secure. Access to systems that contain ACME secrets must be limited to only those who require it. One recommended way to do this is to configure a centralized server that will request and share out the certificates as needed.

Important: Per the Minimum Security Standards for Networked Devices (MSSND), keys stored on servers must be made resistant to offline attacks according to NIST guidelines NIST SP 800-63B, Sec. 5.1.1.2. In addition, the requirements of the UC Encryption Key and Certificate Management Standard are applicable to the External Account Binding (EAB) credentials as well as the generated certificate key pairs.

Additionally, keys issued by Sectigo (or any Enterprise CA) that have the ability to request certificates for multiple applications are classified as P4 shared-fate keys and must be protected accordingly.

Centralized vs Decentralized Management

When using Sectigo's ACME service, it is highly encouraged that users implement a centralized solution for certificate requests and distribution, especially if you are responsible for requesting a large number of certificates. This has the benefit of protecting your ACME secrets by limiting the number of locations they are saved.

A centralized management approach requires configuring a server/container to act as a central server that will request all certificates, and scripts or configuration management tools (e.g. Puppet, Ansible) to distribute and install certificates from the central server to the application hosts. If you run Windows IIS, another potential solution is to use IIS Centralized SSL Certificate Support.

ACME Clients

Sectigo recommends Certbot as their supported ACME client. The Electronic Frontier Foundation provides excellent instructions for installing Certbot on your server. Please visit Certbot Instructions and follow the instructions based on your web server software and operating system.

In addition, "unsupported" clients such as win-acme are known to work well.

For reverse proxies, Apache now supports ACME natively. The Caddy web server is performant and easy to configure with built-in ACME and support. Certbot has integrations for Nginx.

Process for Let’s Encrypt or ZeroSSL

For more information see:

Let's Encrypt Getting Started Guide

ZeroSSL Getting Started Guide

Process for Sectigo/InCommon

Prerequisites

You must be a Departmental Certificate Administrator (DCA) for your department
For each FQDN requiring an SSL certificate, the domain must be delegated to your department
A technical comfort to research how certbot (or other ACME clients) work in your environment

Overview

This process involves creating an ACME account for each host or group of hosts for an application. The goal is to limit the number of domains assigned to a given ACME account. While you may use a single ACME account for all of your delegated domains, it creates a shared-fate risk if an ACME secret were exposed. See Protecting ACME secrets above. The process outlined below attempts to remove the shared-fate risk associated with EAB credentials. For example:

Application myapp1.dept.berkeley.edu may have several domains:

myapp1.dept.berkeley.edu
host1.myapp1.dept.berkeley.edu
host2.myapp1.dept.berkeley.edu

You also manage myapp2.dept.berkeley.edu with a similar set of domain names. In this case you would create 2 ACME accounts, one for each application: dept_myapp1 and dept_myapp2. You would then scope each ACME account to the relevant domains and distribute the EAB credential for myapp1 only to the systems hosting that site.

Important: Sectigo offers multiple ACME enrollment endpoints. In most cases use either one of the Sectigo Public ACME endpoints: https://acme.sectigo.com/v2/InCommonECCOV or https://acme.sectigo.com/v2/InCommonRSAOV. The steps below document using these endpoints because they are able to scope your EAB credentials and offer the most secure configuration when configured correctly.

The Universal ACME endpoint should generally NOT be used unless you implemented a centralized management solution and you are NOT deploying EAB credentials to individual servers. This is because the Universal ACME endpoint does not allow you to limit which of your delegated domains are available via the EAB credentials.

Steps

Log into the Cert-Manager site.
Navigate to Domains and expand berkeley.edu to see your list of delegated domains.
Further expand the domain for which you will configure ACME. If you do not see the FQDNs of the certificates you will be creating via ACME you will need to add them.

NOTE: This is normally not required for certificate requests as long as you have the top-level delegated (e.g. deptA.berkeley.edu). However, to secure your ACME accounts, you need to add all of the FQDNs for each host or application as listed domains.
In this example we will use ACME to issue certificates to app1.training.berkeley.edu but it's not listed, so we will add it now.
Click the green plus button in the upper right corner of the screen and enter the FQDN. Then check the box for SSL Certificate next to your department and Save the domain.
Add any additional FQDNs that will be requested by ACME.
Navigate to Enrollment > ACME.
Select the checkbox next to https://acme.sectigo.com/v2/InCommonRSAOV or https://acme.sectigo.com/v2/InCommonECCOV.
Select Accounts.
Click the green plus button in the upper right corner of the screen to add a new ACME account.
Enter a name for the account, for this example something like training_app1 would make sense.
Note that your department is automatically filled in.
Next, click the plus sign next to Domains.
Now add any FQDNs for this ACME account / endpoint. Note that wildcard domains should never be assigned to an ACME account. In this example both app1.training.berkeley.edu and acme-test01.training.berkeley.edu are used by the application so have been added to this ACME account. Save the account.
The next screen will display the ACME account's External Account Binding (EAB) details. Treat these as sensitive secrets! These should be stored in some sort of password manager (e.g. LastPass) or vault and only exist in the ACME configuration for your system(s). You will need the ACME URL, Key ID, and HMAC key for your ACME client. See the Minimum Security Standards for Networked Devices (MSSND), NIST guidelines NIST SP 800-63B, Sec. 5.1.1.2, and UC Encryption Key and Certificate Management Standard for requirements and guidance on protecting these secrets.
You can now configure an ACME client that supports EAB. Currently the only client supported by Sectigo / InCommon is Certbot.
The ACME setup isn't complete until you connect to the InCommon/Sectigo ACME server with an ACME client. See the section "Connect to the Sectigo ACME Service" for more details.

Install Certbot in your environment

Sectigo recommends Certbot as their supported ACME client. The Electronic Frontier Foundation hosts an excellent set of instructions for installing certbot on your server. Please visitCertbot Instructions and follow the instructions based on your web server software and operating system.

Connect to the Sectigo ACME Service

In order to connect to the Sectigo ACME Service you'll need to use the command line sequence appropriate for your operating system and selected ACME client. This is an example using certbot.

Credentials
 
ACME URL: https://acme.sectigo.com/v2/InCommonRSAOV
Key ID: insert  key ID
HMAC Key: insert HMAC key

Certbot requires admin privileges. Here are example commands that will use certbot to connect to the InCommon ACME service, complete your account setup, and get your first certificate.

Depending on your platform, web server, ACME plug-ins, etc, this may not be the way you obtain subsequent certs. See the certbot documentation at https://certbot.eff.org/. 
 
Linux
 
sudo certbot certonly --standalone --non-interactive --agree-tos --email {requestor@berkeley.edu} --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid {Key ID} --eab-hmac-key {HMAC Key} --domain {domain name used as CN for the cert} --cert-name {short friendly name for cert}
 
Windows
 
Start a CMD window as an admin account, then:
 
certbot certonly --standalone --non-interactive --agree-tos --email {requestor@berkeley.edu} --server https://acme.sectigo.com/v2/InCommonRSAOV --eab-kid {Key ID} --eab-hmac-key {HMAC Key} --domain {domain name used as CN for the cert} --cert-name {short friendly name for cert}
 
Note: Any domains you reference in your ACME request must first be added to InCommon certificate services (if they don't already exist there), then they must be delegated to your department, and then delegated to your departmental ACME account.
This isn't the way the Let's Encrypt service works, but it is the way the InCommon ACME service works.