Clumio announces $75M Series D and 4X YoY growth in ARR

// 29 Jul 2022

Leveraging AWS API Gateway in Clumio

Pritesh Lahoti, Principal Engineer
ShareTwitterfacebookLinkedin

The need for an API Gateway

In a microservices-based architecture for an enterprise-level system like Clumio, it is impractical to have a public endpoint per microservice. One of the major problems is the tight coupling this introduces between the client and the server. In addition, each microservice needs to implement and maintain the common pieces like logging, tracing, security (authentication, authorization, rate-limiting).

An API Gateway dwells between a client and a server to proxy requests and responses between the two.

Having an API Gateway provides a unified entry-point for all the external clients and a framework that takes care of all the common pieces mentioned above. This also enables each microservice to implement its own communication protocol that best fits its requirements, independent of the Gateway.

AWS API Gateway

In addition to the above goodness provided by API Gateways, we also had additional requirements that went into choosing one:

  • Having a mechanism to integrate with the API Gateway in a manner that has no/minimal friction for introducing new APIs and maintaining existing APIs, so as to enable developers to focus primarily on their business logic
  • Given an API Gateway could potentially be a single point of failure, it should be a fully managed offering that is resilient and highly scalable
  • Support for Websockets

AWS API Gateway ticks all these requirements and offers good integration with other AWS services for building on top of it.

Source: https://aws.amazon.com/api-gateway/

Leveraging AWS API Gateway

Swagger specification

Swagger provides a powerful representation for RESTful APIs.

We have employed the go-swagger tool which is capable of generating a Swagger specification from annotated Go code.

As a result, developers only need to annotate their REST API as per the above specification and let our framework (mentioned later) take care of integrating the same with AWS API Gateway.

Integrating with AWS API Gateway

At Clumio, all our AWS infrastructure is managed as code via Terraform. AWS API Gateway can be configured using the Terraform resources that it offers.

We use the aws_api_gateway_rest_api resource that allows us to feed the entire Swagger specification into the AWS API Gateway. The AWS API Gateway, however, cannot process a raw Swagger specification. It expects various extensions (like x-amazon-apigateway-integration) to be part of the Swagger specification so that it could configure an API seamlessly. This is where we have implemented a framework that injects the required extensions into a raw Swagger specification to make it work with the AWS API Gateway!

VPC Link Integration type

AWS API Gateway offers various integrations with your backend servers for proxying the incoming API requests.

As all our microservices are bound to a VPC for tighter security, we use the VPC Link integration. This enables the AWS API Gateway to access private API endpoints within the VPC securely.

Custom Domain Names and API Mappings

By default, AWS API Gateway generates a unique domain name for the API, something like aabbccdd12.execute-api.us-west-2.amazonaws.com. For an enterprise company, we would obviously like to customize the public domain name of the APIs for our consumers. AWS API Gateway allows us to do so via Custom Domain Names.

In addition, we also create separate APIs in API Gateway corresponding to some of our most dense API base paths. All these APIs are then glued together via API Mappings as below. This allows us to scale out our APIs and yet have a single public-facing domain name for them.

Source: https://aws.amazon.com/blogs/compute/using-multiple-segments-in-amazon-api-gateway-base-path-mapping/

Lambda Authorizer, Usage Plans and API Keys

As mentioned earlier, one of the primary use-cases for an API Gateway is to enforce authentication and authorization right at the beginning of the API request lifecycle, so that only the relevant traffic hits the backend servers.

We leverage Lambda Authorizer for authentication and authorization, while the Usage Plans and API Keys are used to enforce rate-limiting/throttling for our APIs.

Source: https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html

We have implemented a Token-based Lambda Authorizer which expects the following input:

The authorizationToken (in the form of a JWT) can be used to implement the required authentication/authorization logic within the Lambda Authorizer to return the output as below:

The authentication/authorization decision is governed by the value of Effect (Allow|Deny) and the throttling decision is governed by the API key identifier in usageIdentifierKey. The API Key has to be associated with a Usage Plan which dictates the request quote to enforce (in terms of requests per second). We issue an API key for each client and hence are able to enforce throttling per client.

Another interesting thing in the output is context which can have any key-value pairs as required by the application. We use this heavily to capture the client information which we then pass throughout the API request lifecycle. This eliminates the need to lookup client details, while the request navigates through multiple microservices at the backend.

Conclusion

We hope you gained insight into how we leverage AWS API Gateway at Clumio and also a sneak-peek into the broader Engineering efforts that we carry out. We give utmost attention to security, scalability and most importantly adopting developer-friendly processes. This makes it easier to maintain products as we mature and scale, while being extensible enough to cater to new functionalities.

If this interests you, feel free to write to us and we are open to get into further technicalities. Do check out our Careers page, if you would like to be a part of the future of Data Protection in the cloud!

This post was originally published on Clumio Engineering on Medium. Subscribe to our blog for exclusive content from our engineers.

eering team.