Part 3 Service-to-service communications
6 Securing east/west traffic with certificates
6.1 Why use mTLS?
TLS protects communications between two parties for confidentiality and integrity. Using TLS to secure data in transit has been a practice for several years. Recently, because of increased cybersecurity threats, it has become a mandatory practice in any business that has serious concerns about data security.
6.1.1 Building trust between a client and a server with a certificate authority
6.1.2 Mutual TLS helps the client and the server to identify each other
6.1.3 HTTPS is HTTP over TLS
6.2 Creating certificates to secure access to microservices
If your microservice endpoints aren’t public, you don’t need to have a public CA sign the corresponding certificates. You can use your own CA, trusted by all the microservices in your deployment.
6.2.1 Creating a certificate authority
6.2.2 Generating keys for the Order Processing microservice
6.2.3 Generating keys for the Inventory microservice
6.2.4 Using a single script to generate all the keys
6.3 Securing microservices with TLS
6.3.1 Running the Order Processing microservice over TLS
curl -v -k https://localhost:6443/orders/11
using –k in the curl command to instruct curl to avoid trust validation. Ideally, you shouldn’t do this in a production deployment.
6.3.2 Running the Inventory microservice over TLS
6.3.3 Securing communications between two microservices with TLS
6.4 Engaging mTLS
Now you have two microservices communicating with each other over TLS, but it’s one-way TLS. Only the calling microservice knows what it communicates with, and the recipient has no way of identifying the client. This is where you need mTLS.
6.5 Challenges in key management
6.5.1 Key provisioning and bootstrapping trust
6.5.2 Certificate revocation
6.6 Key rotation
Key rotation is more challenging in a microservices deployment with an increased number of services spinning on and off. Automation is the key to addressing this problem.
6.7 Monitoring key usage
We discuss the observability of a system under three categories, which we call the three pillars of observability: logging, metrics, and tracing.
Monitoring a microservices deployment is challenging, as many service-to-service interactions occur. We use tools like Zipkin, Prometheus, and Grafana in a microservices deployment to monitor key use.
Summary
There are multiple options in securing communications among microservices, including mutual TLS (mTLS) and JSON Web Tokens (JWTs).
Transport Layer Security protects communications between two parties for confidentiality and integrity. Using TLS to secure data in transit has been a practice for several years.
mTLS is the most popular way of securing interservice communications among microservices.
TLS is also known as one-way TLS, mostly because it helps the client identify the server it’s talking to, but not the other way around. Two-way TLS, or mTLS, fills this gap by helping the client and server identify themselves to each other.
Key management in a microservices deployment is quite challenging, and we need to be concerned about bootstrapping trust and provisioning keys and certificates to workloads or microservices, key revocation, key rotation, and key use monitoring.
Certificate revocation can happen for two main reasons: the corresponding private key is compromised, or the private key of the CA that signed the certificate is compromised.
Using a certificate revocation list (CRL), defined in RFC 2459, was among one of the very first approaches suggested to overcome issues related to certificate revocation.
Unlike CRL, the Online Certificate Status Protocol (OCSP) doesn’t build one
bulky list of all revoked certificates. Each time the TLS client application sees a certificate, it has to talk to the corresponding OCSP endpoint and check whether the certificate is revoked.
OCSP stapling makes OCSP a little better. It takes the overhead of talking to the OCSP endpoint from the TLS client and hands it over to the server.
The approach suggested by short-lived certificates ignores certificate revocation, relying instead on expiration.
All the keys provisioned into microservices must be rotated before they expire.
Observability is an essential ingredient of a typical microservices deployment. It’s about how well you can infer the internal state of a system by looking at the external outputs. Monitoring is about tracking the state of a system.