API Gateway Limitations Issue

We live in a world where many everyday tasks are carried out through digital applications. A lot of those applications thaht are hosted on AWS, are accessed via an AWS API Gateway and need to upload and store data in an S3 bucket. The AWS API Gateway comes with multiple benefits like scalability, security, seamless integration with other AWS services, and performance optimization, but large files can hit the limits of the API Gateway. In this post, we'll explore how presigned S3 URLs can help us the limitation of an API Gateway.

AWS API Gateway has a few limitations, including a 10 MB maximum payload size for REST APIs and a 30-second timeout for request processing. These limits can be problematic when handling large file or long-running uploads.

Note: Full API Gateway limits description can be found here.

Let's imagine the following architecture design. A containerized application backend is receiving user upload requests through an API gateway. The API Gateway and the containerized deployment will mitigate increased load during peak hours by scaling the app seamlessly.

However, there is an issue when the user tries to upload a file larger than 10 MB or has a slow internet connection and exceeds the 30 seconds request limit. In both cases the upload will not be successful and the functionality of the app will be impacted, causing disruptions and bad user experience.

S3 Presigned URLs

The above architecture could be sufficient in some cases, but there is a better way to achieve the functionality result without compromising cloud security. S3 buckets provide a native and secure direct upload option via a presgined link. The presigned URls are URLs that grant temporary, secure access to specific objects stored in an Amazon S3 bucket. They allow users to upload or download files without needing AWS credentials, by signing the URL with a specified expiration time and permissions.

Uploading directly via an S3 presigned link does not have the same time or size limits as those of the API Gateway. There are still limits that should be consider, however, they are configurable when generating the link. In general, S3 presigned URLs support large file uploads and downloads, making them more suitable for handling large files or slower internet connections.

Mitigating the API Gateway Limitiations

Now that we know the benefits of the presigned URLs, we can re-desgin the file upload process by using an S3 upload URL as seen below:

When a user sends an upload request, instead of initiating the upload process, the application backend generates an S3 upload URL via the AWS SDK, signed precisely for the given file and the specific user. After being generated, the URL is returned to the user. At this point it is up to the user-facing logic to handle the file upload. In this way upload data is no loger flowing through the API gateway, but it's handled directly between the user and S3.

Why use the advanced architecture?

This upgraded architecture provides numerous advantages:

1. Enhanced Security: Pre-signed URLs restrict file uploads to authorized users, ensuring secure access without compromising ease of use.

2. Eliminates File Size Restrictions: With direct uploads to S3, the limitations imposed by API Gateway on file size are removed.

3. Better Scalability: The design can support a greater volume of uploads, all while maintaining simplicity and avoiding added complexity.

4. Improves Speed and Efficiency: Direct uploads accelerate the process, minimizing latency and enhancing the overall performance of file transfers.