When building a startup with limited funding and a small team, you probably want to focus on aspects that differentiate you from the competition and offload everything else. If this is the case, you might want to consider Serverless Architecture, which is becoming more popular and mature every year.
The four primary benefits of this architecture are:
No infrastructure to manage – Someone else manages your infrastructure.
Auto-scaling – The ability of your system to automatically allocate resources depending on the load.
Pay-as-you-go – You pay only when you use the resources.
High availability – The ability to replace a resource in case of failure and continue operating.
The Problem with Testing Serverless
This is great, but as always, there is a caveat. Serverless does not mean that there is no server, but rather that someone else manages it, giving you limited control. This introduces various challenges when it comes to testing your system.
A common scenario for some developers is:
Make a change in the source code of an AWS Lambda function.
Deploy a new version of the function.
Test the function.
Repeat.
This process is very time-consuming and significantly extends the feedback loop.
Of course, there are different techniques and tools to alleviate this problem, but they come with their own issues. For example, when using the Serverless Framework, you can successfully invoke your function locally, but the function deployed on AWS may fail due to missing IAM permissions. You can use LocalStack to simulate AWS locally, but how confident can you be that your code integrates correctly with real services?
It is essential to have a solid strategy when testing your Serverless Architecture—or any system, for that matter. Let’s look at how Apptimia tests Serverless Architecture in an AWS-based system for one of our customers.
Case Study: Testing Serverless Architecture on AWS
A simplified version of the system architecture is illustrated in the following diagram:
The AWS API Gateway ingests HTTP traffic, authorizes requests using AWS Cognito, and forwards them to the Lambda REST API. The Lambda function can get or put data in PostgreSQL on RDS or schedule tasks for Lambda Workers using an SNS Topic or an SQS Queue. Workers run in the background, communicating with each other or with other AWS services like AWS S3.
So how can we test this system in a way that is effective and provides confidence?
First, let’s remind ourselves how Testing Pyramid looks:
As we move from the bottom to the top, the number of tests decreases, the feedback loop slows down, but confidence increases. How does this correspond with our architecture?
Unit Tests
We can definitely use unit tests to test the domain logic of our Lambda functions. Unit tests provide very fast feedback. However, domain logic may be coupled with infrastructure code in ways that make unit testing difficult.
Let’s look at a concept that helps with this: Hexagonal Architecture.
Hexagonal Architecture allows us to separate the domain layer from infrastructure code using port interfaces, which are implemented as adapters to the infrastructure code. This way, we can move all our infrastructure code into suitable adapters and replace real adapters with mocks during unit testing.
By mocking all infrastructure code, running ~1600 unit tests for our system locally (Node.js with Jest) takes less than 8 seconds. Additionally, during development, you typically run only a small subset of the test suite, which executes in milliseconds. That’s a huge difference compared to the previously mentioned process of deploying after each change.
Integration Tests
Now, let’s move on to integration tests. We have two types of integration tests:
Adapter integration tests – Tests integration with infrastructure.
Lambda integration tests – Tests integration between all modules, along with real adapters.
All integration tests can be run locally without deploying the code to AWS. The question is: how do we handle infrastructure in our tests? Some pieces of infrastructure, like PostgreSQL (which runs on RDS that is not a serverless service), can be run locally. But what about other AWS services?
There are tools like LocalStack, which simulate AWS locally without requiring internet access. Another option is to use AWS directly, which requires deployment but provides more confidence by testing real services. This is the approach we took.
A typical integration test in our project follows the Arrange-Act-Assert pattern:
Arrange - Create a piece of AWS infrastructure using the AWS SDK, such as an S3 bucket or an SNS topic. Then, upload a fixture file to the bucket if necessary.
Act - Execute a function that interacts with the infrastructure, such as deleting a file from S3.
Assert - Use the AWS SDK to verify the expected result (e.g., confirm the file was deleted). Then, tear down the AWS resources (e.g., remove the S3 bucket).
This approach works very well. Running 236 integration tests takes 2 minutes and 20 seconds, which can be reduced by using multiple PostgreSQL test databases to run tests in parallel. Failures due to network issues are rare. Developers do not interfere with each other when running tests locally, as all resources are created with special prefixes, ensuring stable integration tests.
End-to-End Tests
Even with integration tests, there are still aspects that remain untested. For example, how can we be sure that a Lambda function has the correct permissions to access an S3 file? What about RDS configuration? And, perhaps most importantly, API Gateway and Cognito integration with the Lambda REST API?
That’s where we rely on end-to-end (E2E) tests, which test the system as a whole.
Deploying the entire system and running tests that exercise all components is time-consuming and less stable than other types of tests. To save developer time, our E2E tests (implemented in Cypress) are primarily executed in the Continuous Delivery pipeline. To optimize efficiency and stability, we limit E2E tests to a few critical User Journeys. An example user journey may look like this: a user logs in, uploads some data, processes the data, downloads the results, and logs out.
Since less critical user paths are not covered, some errors may slip through if they were not caught in unit or integration tests. However, this is a small price to pay for reasonably fast and stable E2E tests, especially with a good monitoring system and the ability to deploy patches quickly.
Manual Tests
What about manual testing and Exploratory Tests? Since all our infrastructure is described as code using the Serverless Application Model, we can easily create a new environment for manual tests. Additionally, we can deploy an exact copy of the production infrastructure without worrying about high costs, as serverless pricing is usage-based, and a single developer’s traffic is typically minimal.
Load Tests
One category of testing we haven’t covered yet is Load Testing. While we have not implemented these tests yet, here’s how we would approach them:
Limiting execution frequency – With Serverless, we pay as we go. While this is beneficial for manual testing, it becomes a challenge for load testing, as generating a high volume of traffic could result in significant costs.
Focusing on user journeys – Using user journeys for load testing seems to be a better approach than hammering individual endpoints, as the latter is more likely to stress AWS’s scaling capabilities rather than your actual system.
Considering AWS Lambda throttling – It is crucial to consider AWS Lambda’s throttling behavior when ramping up traffic during tests.
Using proper tools – Tools like Artillery or Locust can be helpful for effective load testing.
This is one of the many production-grade cloud systems we have built for our customers at Apptimia. If you're building scalable cloud processing solutions and need help, get in touch with us!
Bartosz Ch.
Lead Software Engineer at Apptimia