TLS protects communications between two parties for confidentiality and integrity. Using TLS to secure data in transit has been a practice for several years. Recently, because of increased cybersecurity threats, it has become a mandatory practice in any business that has serious concerns about data security.
6.1.1 Building trust between a client and a server with a certificate authority 6.1.2 Mutual TLS helps the client and the server to identify each other
6.1.3 HTTPS is HTTP over TLS
6.2 Creating certificates to secure access to microservices
If your microservice endpoints aren’t public, you don’t need to have a public CA sign the corresponding certificates. You can use your own CA, trusted by all the microservices in your deployment.
6.2.1 Creating a certificate authority 6.2.2 Generating keys for the Order Processing microservice 6.2.3 Generating keys for the Inventory microservice
6.2.4 Using a single script to generate all the keys
6.3 Securing microservices with TLS
6.3.1 Running the Order Processing microservice over TLS
curl -v -k https://localhost:6443/orders/11 using –k in the curl command to instruct curl to avoid trust validation. Ideally, you shouldn’t do this in a production deployment.
6.3.2 Running the Inventory microservice over TLS 6.3.3 Securing communications between two microservices with TLS
6.4 Engaging mTLS
Now you have two microservices communicating with each other over TLS, but it’s one-way TLS. Only the calling microservice knows what it communicates with, and the recipient has no way of identifying the client. This is where you need mTLS.
6.5 Challenges in key management
6.5.1 Key provisioning and bootstrapping trust
6.5.2 Certificate revocation
6.6 Key rotation
Key rotation is more challenging in a microservices deployment with an increased number of services spinning on and off. Automation is the key to addressing this problem.
6.7 Monitoring key usage
We discuss the observability of a system under three categories, which we call the three pillars of observability: logging, metrics, and tracing.
Monitoring a microservices deployment is challenging, as many service-to-service interactions occur. We use tools like Zipkin, Prometheus, and Grafana in a microservices deployment to monitor key use.
Summary There are multiple options in securing communications among microservices, including mutual TLS (mTLS) and JSON Web Tokens (JWTs).
Transport Layer Security protects communications between two parties for confidentiality and integrity. Using TLS to secure data in transit has been a practice for several years.
mTLS is the most popular way of securing interservice communications among microservices.
TLS is also known as one-way TLS, mostly because it helps the client identify the server it’s talking to, but not the other way around. Two-way TLS, or mTLS, fills this gap by helping the client and server identify themselves to each other.
Key management in a microservices deployment is quite challenging, and we need to be concerned about bootstrapping trust and provisioning keys and certificates to workloads or microservices, key revocation, key rotation, and key use monitoring.
Certificate revocation can happen for two main reasons: the corresponding private key is compromised, or the private key of the CA that signed the certificate is compromised.
Using a certificate revocation list (CRL), defined in RFC 2459, was among one of the very first approaches suggested to overcome issues related to certificate revocation.
Unlike CRL, the Online Certificate Status Protocol (OCSP) doesn’t build one bulky list of all revoked certificates. Each time the TLS client application sees a certificate, it has to talk to the corresponding OCSP endpoint and check whether the certificate is revoked.
OCSP stapling makes OCSP a little better. It takes the overhead of talking to the OCSP endpoint from the TLS client and hands it over to the server.
The approach suggested by short-lived certificates ignores certificate revocation, relying instead on expiration.
All the keys provisioned into microservices must be rotated before they expire.
Observability is an essential ingredient of a typical microservices deployment. It’s about how well you can infer the internal state of a system by looking at the external outputs. Monitoring is about tracking the state of a system.
3 Securing north/south traffic with an API gateway
In an ideal world, the microservices developer should worry only about the business functionality of a microservice, and the rest should be handled by specialized components with less hassle.
The API Gateway and Service Mesh are two architectural patterns that help us reach that ideal.
The API Gateway pattern is mostly about edge security, while the Service Mesh pattern deals with service-to-service security. Or, in other words, the API Gateway deals with north/south traffic, while the Service Mesh deals with east/west traffic.
3.1 The need for an API gateway in a microservices deployment
3.1.1 Decoupling security from the microservice
One key aspect of microservices best practices is the single responsibility principle. This principle, commonly used in programming, states that every module, class, or function should be responsible for a single part of the software’s functionality. Under this principle, each microservice should be performing only one particular function.
Executing all these steps becomes a problem because the microservice loses its atomic characteristics by performing more operations than it’s supposed to.
The coupling of security and business logic introduces unwanted complexity and maintenance overhead to the microservice.
（1）Changes in the security protocol require changes in the microservice
（2）Scaling up the microservice results in more connections to the authorization server
3.1.2 The inherent complexities of microservice deployments make them harder to consume
3.1.3 The rawness of microservices does not make them ideal for external exposure
3.2 Security at the edge
3.2.1 Understanding the consumer landscape of your microservices
Applications running within the organization’s computing infrastructure may consume both internal-facing and external-facing microservices.
3.2.2 Delegating access
3.2.3 Why not basic authentication to secure APIs?
The username and password are static, long-living credentials. No restrictions on what the application can do.
3.2.4 Why not mutual TLS to secure APIs?
mTLS solves one of the problems with basic authentication by having a lifetime for its certificates. The certificates used in mTLS are time-bound, and whenever a certificate expires, it’s no longer considered valid.
However, just as in basic authentication, mTLS fails to meet access delegation requirements we discussed in section 3.2.2 in a microservices deployment.
Therefore, mTLS is mostly used to secure communication between a client application and a microservice, or communications among microservices. In other words, mTLS is mostly used to secure communications among systems.
3.2.5 Why OAuth 2.0?
To understand why OAuth 2.0 is the best security protocol for securing your microservices at the edge, first you need to understand your audience. You need to figure out who wants access to your resources, for what purpose, and for how long.
3.3 Setting up an API gateway with Zuul
3.3.1 Compiling and running the Order Processing microservice
3.3.2 Compiling and running the Zuul proxy
3.3.3 Enforcing OAuth 2.0–based security at the Zuul gateway
Enforcing token validation at the zuul gateway
A filter can be one of four types :
Prerequest filter—Executes before the request is routed to the target service Route filter—Can handle the routing of a message Post-request filter—Executes after the request has been routed to the target service Error filter—Executes if an error occurs in the routing of a request
Self-validation of tokens without integrating with an authorization server
But the gateway component relies heavily on the authorization server to enable access to your microservices, so the gateway component is coupled with another entity.
The way to deal with this problem is to find a mechanism that enables the gateway to validate tokens by itself without the assistance of an authorization server.
JWTs are designed to solve this problem . A JSON Web Signature (JWS) is a JWT signed by the authorization server. By verifying the signature of the JWS, the gateway knows that this token was issued by a trusted party and that it can trust the information contained in the body.
Pitfalls of self-validating tokens and how to avoid them
If one of these tokens is prematurely revoked, the API gateway won’t know that the token has been revoked, because the revocation happens at the authorization server end, and the gateway no longer communicates with the authorization server for the validation of tokens.
In practice, however, applications with longer user sessions have to keep refreshing their access tokens when the tokens carry a shorter expiration.
Another problem with the self-contained access token is that the certificate used to verify a token signature might have expired.
To solve this problem, you need to make sure that whenever a certificate is renewed, you deploy the new certificate on the gateway.
Then the gateway can fetch the token issuer’s certificate dynamically from an endpoint exposed by the authorization server to do the JWT signature validation, and check whether that certificate is signed by a certificate authority it trusts.
3.4 Securing communication between Zuul and the microservice
3.4.1 Preventing access through the firewall
3.4.2 Securing the communication between the API gateway and microservices by using mutual TLS
But mTLS verification happens at the Transport layer of the microservice and doesn’t propagate up to the Application layer.
Summary The API Gateway pattern is used to expose microservices to client applications as APIs.
The API gateway helps to expose microservices of different flavors by using a consistent and easy-to-understand interface to the consumers of these microservices.
We do not have to expose all microservices through the API gateway. Some microservices are consumed internally only, in which case they will not be exposed through the gateway to the outside world.
Protocols such as basic authentication and mutual TLS are not sufficient to secure APIs, and microservices are exposed to the outside world via APIs.
OAuth 2.0 is the de facto standard for securing APIs and microservices at the edge.
OAuth 2.0 is an extensible authorization framework, which has a wide array of grant types. Each grant type defines the protocol indicating how a client application would obtain an access token to access an API/microservice.
We need to choose the right OAuth 2.0 grant type for our client applications based on their security characteristics and trustworthiness.
An access token can be a reference token or a self-contained token (JWT). If it is a reference token, the gateway has to talk to the issuer (or the authorization server) always to validate it. For self-contained tokens, the gateway can perform the validation by verifying the token signature.
A self-contained access token has its own pitfalls, and one way to get around token revocation is to have short-lived JWTs for self-contained access tokens.
The communication between the gateway and the microservice can be protected with either firewall rules or mutual TLS—or a combination of both.
All samples in this chapter use HTTP (not HTTPS) endpoints to spare you from having to set up proper certificates and to make it possible for you to inspect messages being passed on the wire (network), if required. In production systems, we do not recommend using HTTP for any endpoint.
4 Accessing a secured microservice via a single-page application
We believe in completing an end-to-end architecture with a microservices deployment, from data to screen. And SPAs are the most used client application type.
4.1 Running a single-page application with Angular
4.1.1 Building and running an Angular application from the source code
4.1.2 Looking behind the scenes of a single-page application
4.2 Setting up cross-origin resource sharing
4.2.1 Using the same-origin policy
URL consists of the URI scheme, hostname, and port.
The same-origin policy exists to prevent a malicious script on one website from accessing data on other websites unintentionally. The same-origin policy applies only to data access, not to CSS, images, and scripts, so you could write web pages that consist of links to CSS, images, and scripts of other origins.
4.2.2 Using cross-origin resource sharing
Web browsers have an exception to the same-origin policy: cross-origin resource sharing (CORS), a specification that allows web browsers to access selected resources on different origins;
Web browsers use the OPTIONS HTTP method along with special HTTP headers to determine whether to allow or deny a cross-origin request. Let’s see how the protocol works.
You can observe this request, known as a preflight request, by inspecting it on the Network tab of your browser’s developer tools. The request includes the following HTTP headers: Access-Control-Request-Headers Access-Control-Request-Method Origin
The server responds to this preflight request with the following headers: Access-Control-Allow-Credentials Access-Control-Allow-Headers Access-Control-Allow-Methods Access-Control-Allow-Origin Access-Control-Max-Age
4.2.3 Inspecting the source that allows cross-origin requests
@CrossOrigin(origins = “http://localhost:4200”)
4.2.4 Proxying the resource server with an API gateway
4.3 Securing a SPA with OpenID Connect
OpenID Connect (OIDC) is an identity layer built on top of OAuth 2.0.
Summary Single-page applications perform better by reducing network chattiness as they perform all rendering on the web browser and by reducing the workload on the web server.
The SPA architecture brings simplicity to microservices architectures because they do not require complex web application-hosting facilities such as JBoss or Tomcat.
The SPA architecture abstracts out the data layer from the user experience layer.
SPAs have security restrictions and complexities due to the same-origin policy on web browsers.
The same-origin policy ensures that scripts running on a particular web page can make requests only to services running on the same origin.
The same-origin policy applies only to data access, not to CSS, images, and scripts, so you can write web pages that consist of links to CSS, images, and scripts of other origins.
OpenID Connect is an identity layer built on top of OAuth 2.0. Most SPAs use OpenID Connect to authenticate users.
Because SPAs may consume APIs (data sources) from multiple trust domains, a token obtained from one trust domain may not be valid for another trust domain. We need to build token-exchange functionality when a SPA hops across multiple trust boundaries.
All samples in this chapter used HTTP (not HTTPS) endpoints to spare you from having to set up proper certificates and to make it possible for you to inspect messages being passed on the wire (network), if required. In production systems, we do not recommend using HTTP for any endpoint.
5 Engaging throttling, monitoring, and access control
5.1 Throttling at the API gateway with Zuul
5.1.1 Quota-based throttling for applications
5.1.2 Fair usage policy for users
5.1.3 Applying quota-based throttling to the Order Processing microservice
5.1.4 Maximum handling capacity of a microservice
5.1.5 Operation-level throttling
5.1.6 Throttling the OAuth 2.0 token and authorize endpoints
5.1.7 Privilege-based throttling
5.2 Monitoring and analytics with Prometheus and Grafana
The modern term for monitoring and analytics is known as observability.
Prometheus is a popular open source monitoring tool for microservices. It helps us keep track of system metrics over a given time period and can be used to determine the health of a software system. Metrics include memory usage and CPU consumption. Grafana is an open source data visualization tool. It can help you build dashboards to visualize the metrics being provided by Prometheus or any other data source. At this time of writing, Grafana is the most popular data-visualizing tool in the market.
5.2.1 Monitoring the Order Processing microservice
As you can see, Grafana gives you a much more user-friendly view of the metrics exposed by the Order Processing microservice.
5.2.2 Behind the scenes of using Prometheus for monitoring
5.3 Enforcing access-control policies at the API gateway with Open Policy Agent
The API gateway here is acting as a policy enforcement point. OPA is a lightweight general-purpose policy engine that has no dependency on microservices. You can use OPA to define fine-grained access-control policies and enforce those policies at different places in a microservices deployment.
5.3.1 Running OPA as a Docker container 5.3.2 Feeding the OPA engine with data 5.3.3 Feeding the OPA engine with access-control policies 5.3.4 Evaluating OPA policies 5.3.5 Next steps in using OPA
Summary Quota-based throttling policies for applications help to monetize APIs/microservices and to limit a given application from overconsuming APIs/microservices.
Fair-usage policies need to be enforced on applications to ensure that all users get a fair quota of requests.
User privilege-based throttling is useful for allowing different quotas for users with different privilege levels.
An API gateway can be used to apply throttling rules in a microservices deployment.
Prometheus is the most popular open source monitoring tool available as of this writing.
Grafana helps to visualize the data being recorded by Prometheus.
Open Policy Agent (OPA) helps control access to a microservices deployment.
OPA data, OPA input data, and OPA policies are used together to apply various access-control rules.
All samples in this chapter used HTTP (not HTTPS) endpoints to spare you from having to set up proper certificates and to make it possible for you to inspect messages being passed on the wire (network), if required. In production systems, we do not recommend using HTTP for any endpoint.
1.1 How security works in a monolithic application
In most monolithic applications, security is enforced centrally, and individual components need not worry about carrying out additional checks unless there is a desperate requirement to do so. As a result, the security model of a monolithic application is much more straightforward than that of an application built around microservices architecture.
1.2 Challenges of securing microservices
Mostly because of the inherent nature of microservices architecture, security is challenging.
1.2.1 The broader the attack surface, the higher the risk of attack
1.2.2 Distributed security screening may result in poor performance
These repetitive, distributed security checks and remote connections could contribute heavily to latency and considerably degrade the performance of the system.
1.2.3 Deployment complexities make bootstrapping trust among microservices a nightmare
Managing a large-scale microservices deployment with thousands of services would be extremely challenging if you didn’t know how to automate.
Fortunately, things didn’t happen that way, and that’s why we believe that microservices and containers are a match made in heaven. Microservices and containers (Docker) were born at the right time to complement each other nicely.
1.2.4 Requests spanning multiple microservices are harder to trace
Observability is a measure of what you can infer about the internal state of a system based on its external outputs. Logs, metrics, and traces are known as the three pillars of observability.
1.2.5 Immutability of containers challenges how you maintain service credentials and access-control policies
The whole purpose of expecting servers to be immutable in a microservices deployment is to make deployment clean and simple. At any point, you can kill a running container and create a new one with the base configuration without worrying about runtime data. （ scale horizontally easier）
1.2.6 The distributed nature of microservices makes sharing user context harder
The challenge is to build trust between two microservices so that the receiving microservice accepts the user context passed from the calling microservice.
1.2.7 Polyglot architecture demands more security expertise on each development team
In a multiteam environment, in which each team develops its own set of microservices, each team has the flexibility to pick the optimal technology stack for its requirements. This architecture, which enables the various components in a system to pick the technology stack that is best for them, is known as a polyglot architecture.
A polyglot architecture makes security challenging. Because different teams use different technology stacks for development, each team has to have its own security experts.
1.3 Key security fundamentals
How much you should worry about security isn’t only a technical decision, but also an economic decision. The level of security you need depends on the assets you intend to protect.
1.3.1 Authentication protects your system against spoofing
1.3.2 Integrity protects your system from data tampering
Systems protected for integrity don’t ignore this possibility; they introduce measures so that if a message is altered, the recipient can detect and discard the request. The most common way to protect a message for integrity is to sign it.
Along with the data in transit, the data at rest must be protected for integrity. Of all your business data, audit trails matter most for integrity checks.
One way is to periodically calculate the message digests of audit trails, encrypt them, and store them securely.
1.3.3 Nonrepudiation: Do it once, and you own it forever
Nonrepudiation is an important aspect of information security that prevents you from denying anything you’ve done or committed.
Even in the digital world, a signature helps you achieve nonrepudiation; in this case, you use a digital signature.
You also need to make sure that you record transactions along with the timestamp and the signature—and maintain those records for a considerable amount of time.
1.3.4 Confidentiality protects your systems from unintended information disclosure
1.3.5 Availability: Keep the system running, no matter what
1.3.6 Authorization: Nothing more than you’re supposed to do
Authentication helps you learn about the user or the requesting party. Authorization determines the actions that an authenticated user can perform on the system.
1.4 Edge security
1.4.1 The role of an API gateway in a microservices deployment
APIs have also become many companies’ main revenue-generation channel. The key role of the API gateway in a microservices deployment is to expose a selected set of microservices to the outside world as APIs and build quality-of-service (QoS) features. These QoS features are security, throttling, and analytics.
1.4.2 Authentication at the edge
Certificate-based authentication protects an API at the edge with mutual Transport Layer Security (mTLS).
OAuth 2.0, which is an authorization framework for delegated access control, is the recommended approach for protecting APIs when one system wants to access an API on behalf of another system or a user.
1.4.3 Authorization at the edge
In addition to figuring out who the requesting party is during the authentication process, the API gateway could enforce corporatewide access-control policies, which are probably coarse-grained. More fine-grained access-control policies are enforced at the service level by the microservice itself .
1.4.4 Passing client/end-user context to upstream microservices
But you need a way to protect the communication channels between the gateway and the corresponding microservice, as well as a way to pass the initial client/user context. User context carries basic information about the end user, and client context carries information about the client application. This information probably could be used by upstream microservices for service-level access control.
You have a couple of options: pass the user context in an HTTP header, or create a JWT with the user data. The first option is straightforward but raises some trust concerns when the first microservice passes the same user context in an HTTP header to another microservice. The second microservice doesn’t have any guarantee that the user context isn’t altered. But with JWT, you have an assurance that a man in the middle can’t change its content and go undetected, because the issuer of the JWT signs it.
1.5 Securing service-to-service communication
The security model that you develop to protect service-to-service communication should consider the communication channels that cross trust boundaries, as well as how the actual communication takes place between microservices: synchronously or asynchronously.
In most cases, synchronous communication happens over HTTP. Asynchronous communication can happen over any kind of messaging system.
1.5.1 Service-to-service authentication
You have three common ways to secure communications among services in a microservices deployment: trust the network, mTLS, and JWTs.
trust the network
The trust-the-network approach is an old-school model in which no security is enforced in service-to-service communication; rather, the model relies on network-level security .
mTLS
Mutual TLS , in fact, this method is the most common form of authentication used today. Each microservice in the deployment has to carry a public/private key pair and uses that key pair to authenticate to the recipient microservices via mTLS.
JWT
Unlike mTLS, JWT works at the application layer, not at the transport layer. JWT is a container that can carry a set of claims from one place to another.
1.5.2 Service-level authorization
Two approaches are used to enforce authorization at the service level: the centralized policy decision point (PDP) model and the embedded PDP model.
This method creates a lot of dependency on the PDP and also increases the latency because of the cost of calling the remote PDP endpoint. In some cases, the effect on latency can be prevented by caching policy decisions at the service level, but other than cache expiration time, there’s no way to communicate policy update events to the service. In practice, policy updates happen less frequently, and cache expiration may work in most cases.
The challenge with embedded PDPs is how to get policy updates from the centralized policy administration point (PAP).
1.5.3 Propagating user context among microservices
When one microservice invokes another microservice, it needs to carry both the enduser identity and the identity of the microservice itself. When one microservice authenticates to another microservice with mTLS or JWT, the identity of the calling microservice can be inferred from the embedded credentials.
There are three common ways to pass the end-user context from one microservice to another microservice: Send the user context as an HTTP header. Use a JWT. Use a JWT issued by an external STS that is trusted by all the microservices in the deployment. （ This is the most secure approach. ）
1.5.4 Crossing trust boundaries
In terms of security, when one microservice talks to another microservice, and both microservices are in the same trust domain, each microservice may trust one STS in the same domain or a certificate authority in the same domain. Based on this trust, the recipient microservice can validate a security token sent to it by a calling microservice. Typically, in a single trust domain, all the microservices trust one STS and accept only security tokens issued by that STS.
Summary Securing microservices is quite challenging with respect to securing a monolithic application, mostly because of the inherent nature of the microservices architecture.
A microservices security design starts by defining a process to streamline development and engage security-scanning tools to the build system, so that we can discover the code-level vulnerabilities at a very early stage in the development cycle.
We need to worry about edge security of a microservices deployment and securing communications among microservices.
Edge security is about authenticating and authorizing requests coming into the microservices deployment from client applications, at the edge, probably with an API gateway.
Securing communications among microservices is the most challenging part.We discussed multiple techniques in this chapter, and which you choose will depend on many factors, such as the level of security, the type of communication (synchronous or asynchronous), and trust boundaries.
2 First steps in securing microservices
2.1 Building your first microservice
2.1.1 Downloading and installing the required software INSTALLING THE JDK INSTALLING APACHE MAVEN INSTALLING CURL INSTALLING THE GIT COMMAND-LINE TOOL
2.1.3 Compiling the Order Processing microservice mvn clean install mvn spring-boot:run
2.1.4 Accessing the Order Processing microservice 2.1.5 What is inside the source code directory?
Often, a resource represents an object or entity that you intend to inspect or manipulate. When mapped to HTTP, a resource is usually identified by a request URI, and an action is represented by an HTTP method;
2.1.6 Understanding the source code of the microservice
2.2 Setting up an OAuth 2.0 server
2.2.1 The interactions with an authorization server
It’s recommended that the client communicate with the microservice over HTTPS and send the token in an HTTP header instead of a query parameter. Because query parameters are sent in the URL, those can be recorded in server logs. Hence, anyone who has access to the logs can see this information.
2.2.2 Running the OAuth 2.0 authorization server 2.2.3 Getting an access token from the OAuth 2.0 authorization server 2.2.4 Understanding the access token response
2.5 Performing service-level authorization with OAuth 2.0 scopes
A privilege describes the actions you’re permitted to perform on a resource.
More often than not, your role or roles in an organization describe which actions you’re permitted to perform within that organization and which actions you’re not permitted to perform. A privilege may also indicate status or credibility.
Likewise, a privilege is an indication of the level of access that a user or an application possesses in a system.
In the world of OAuth 2.0, privilege is mapped to a scope. A scope is way of abstracting a privilege. A privilege can be a user’s role, membership status, credibility, or something else. It can also be a combination of a few such attributes. You use scopes to abstract the implication of a privilege. A scope declares the privilege required by a calling client application to grant access to a resource.
2.5.1 Obtaining a scoped access token from the authorization server
2.5.2 Protecting access to a microservice with OAuth 2.0 scopes
{ “error”:”insufficient_scope”, “error_description”:”Insufficient scope for this resource”, “scope”:”write” }
Summary OAuth 2.0 is an authorization framework, which is widely used in securing microservices deployments at the edge.
OAuth 2.0 supports multiple grant types. The client credentials grant type, which we used in this chapter, is used mostly for system-to-system authentication.
Each access token issued by an authorization server is coupled with one or more scopes. Scopes are used in OAuth 2.0 to express the privileges attached to an access token.
OAuth 2.0 scopes are used to protect and enforce access-control checks in certain operations in microservices.
All samples in this chapter used HTTP (not HTTPS) endpoints to spare you from having to set up proper certificates and to make it possible for you to inspect messages being passed on the wire (network), if required. In production systems, we do not recommend using HTTP for any endpoint.
