Cloud Architecture: Uploading to object storage without a server, Part 1

Last week I was working on a feature where I needed to allow a user to upload a file. These files are typically under 200kb, and I was having no issue passing the information in a POST body and processing it with a serverless function. However when I tried a larger file of 20mb I hit an error. DARN.

I was passing the file as a string in an AppSync event body, which has a limit of 6mb. Okay, what if I make an API Gateway endpoint and send it there and then to S3? Only slightly better, as Lambda integrations are limited to a 10mb request body. The solution was to upload the file directly to object storage from the frontend. For this AWS S3 has Presigned URLs.

Presigned URLs grant temporary access to a specific resource or operation. They can help ensure that we are following the principle of least privilege as we build distributed applications.

Resources I'll be demonstrating:

• React App made with Vite to request presigned URL and upload a file (covered in part 2)

• API Gatway & Lambda for generating and configuring the presigned URL

• S3 bucket for storing the file

• AWS Cloud Development Kit (CDK) for resource definition and configuration

This writeup assumes you have an AWS account and a default credentials profile. To check, run

cat ~/.aws/credentials

You should see something like this

putting this together I was on node v18.19.1

Alright…. let's get to it! Here’s a diagram of the application flow:

First, we need to configure a CDK application. Make a new directory for our code:

mkdir upload-serverless && cd upload-serverless

Then run the following command.

cdk init app --language=typescript

If you need to install cdk you can do so with npm install -g aws-cdk

Great! You should see something like the following

Couple setup steps before we get to the code

• First, create a .env file in the root containing your 12 digit AWS account ID and your preferred region. Mine looks like this.

  AWS_ACCOUNT_ID=123123123123
  REGION='us-west-1'

• Next we need to install our packages. Run

  yarn add dotenv aws-cdk-lib@2.96.2 && yarn add aws-cdk-lib@2.96.2 --dev

• delete package-lock.json

We can now start defining the resources we want to deploy.

Our infrastructure code will be placed inside lib/upload-serverless-stack.ts. I’m going to share the complete file then go over each piece in detail.

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as apiGateway from 'aws-cdk-lib/aws-apigateway';
import * as dotenv from 'dotenv';
dotenv.config();

export class UploadServerlessStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const presignedUrlLambdaRole = new iam.Role(this, 'PresignedUrlLambdaRole', {
      assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com')
    });

    presignedUrlLambdaRole.addToPolicy(
      new iam.PolicyStatement({
        actions: ['ssm:GetParameters'],
        resources: [
          `arn:aws:ssm:${process.env.REGION}:${process.env.AWS_ACCOUNT_ID}:parameter/ACCESS_KEY_ID`,
          `arn:aws:ssm:${process.env.REGION}:${process.env.AWS_ACCOUNT_ID}:parameter/SECRET_ACCESS_KEY`,
          `arn:aws:ssm:${process.env.REGION}:${process.env.AWS_ACCOUNT_ID}:parameter/BUCKET_NAME`
        ]
      })
    );

    // Construct for serverless function
    const presignedUrlLambda = new lambda.Function(
      this,
      'presignedUrlLambda',
      {
        runtime: lambda.Runtime.NODEJS_18_X,
        handler: 'index.handler',
        code: lambda.Code.fromAsset('src/lambdas/presignedUrlLambda/dist'),
        role: presignedUrlLambdaRole,
        timeout: cdk.Duration.seconds(10)
      }
    );

    const utilityApi = new apiGateway.RestApi(this, 'UtilityApi', {
      defaultCorsPreflightOptions: {
        allowOrigins: ['*'],
        allowMethods: apiGateway.Cors.ALL_METHODS
      }
    });

    utilityApi.root
      .addResource('put-object-presigned') //The name of our endpoint
      .addMethod(
        'GET',
        new apiGateway.LambdaIntegration(presignedUrlLambda)
      );


    new s3.Bucket(this, 'Bucket');
  }
}

Whew, that’s a lot. Let’s break it down piece by piece.

At the top of our file we have imports and a CDK Stack definition.

Here we’re doing a couple things

• First, we create a new IAM role with the name of PresignedUrlLambdaRole. This role will be assumed by the Lambda resource that we’re about to create.

• Add Policy Actions to the Lambda Role to allow access to AWS Systems Manager (SSM) Parameter Store. We will be storing sensitive values required by S3 to generate the presigned URL.

• Next, we define our Node.js Lambda function. We specify the Node version we want to use, the type of handler we want to use, where the code for this function lives, the role of the function, and how long we want the function to be able to run before it times out. Let’s finish up our CDK code before we jump into exactly what this Lambda function will be doing.

  

  Finally:

• Creating an API Gateway and assigning its construct to the variable utilityApi

  • This API allows all origins to make requests with [*]

  • This API accepts all HTTP methods with apiGateway.Cors.ALL_METHODS

• Adding a resource to our new utilityApi

  • Exposes endpoint put-object-presigned

  • Associate it with a GET HTTP method

  • Add a Lambda integration to the resource. This means our GET HTTP endpoint response will be fulfilled by the serverless function presignedUrlLambda.

  • When we make GET requests to this endpoint, the Lambda serverless function will be responsible for retrieving a presigned URL from S3, producing a response body, and responding with the appropriate status code (200, 400, etc.)

• Creating the bucket that will store our files

Put on your Gordon Freeman glasses, and let’s dive into the Lambda code.

We need to create our directory structure. In the root directory run:

mkdir -p src/lambdas/presignedUrlLambda && cd src/lambdas/presignedUrlLambda

Now we need to create the code for our serverless Lambda function. Create three files here.

touch package.json && touch handler.ts && touch tsconfig.json

Inside of the Lambda folder’s package.json paste the following and run yarn

{
  "name": "presigned-url-lambda",
  "version": "1.0.0",
  "license": "",
  "description": "Description of your Lambda function",
  "scripts": {
    "test": "echo \"No tests available\"",
    "build": "esbuild handler.ts --bundle --minify --sourcemap --platform=node --target=es2020 --outfile=dist/index.js"
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.556.0",
    "@aws-sdk/client-ssm": "^3.556.0",
    "@aws-sdk/s3-request-presigner": "^3.556.0",
    "@types/node-fetch": "^2.6.7",
    "esbuild": "^0.19.3",
    "uuid": "^9.0.1"
  },
  "devDependencies": {
    "@types/uuid": "^9.0.4",
    "terser": "^5.20.0",
    "typescript": "^5.2.2"
  }
}

tsconfig.json

{
    "compilerOptions": {
      "target": "esNext",
      "strict": true,
      "preserveConstEnums": true,
      "noEmit": false,
      "sourceMap": false,
      "module":"CommonJS",
      "moduleResolution": "node",
      "outDir": "./dist",
      "esModuleInterop": true, 
      "rootDir": "../",
      "skipLibCheck": true,
      "forceConsistentCasingInFileNames": true, 
      "isolatedModules": true, 
    },
    "include": ["handler.ts"],
    "exclude": ["node_modules"]
  }

And last, inside handler.ts paste the following

import { getSignedUrl } from '@aws-sdk/s3-request-presigner';
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { GetParametersCommandOutput, SSM } from '@aws-sdk/client-ssm';
import { v4 as uuidv4 } from 'uuid';

type PresignedURLResponse = {
    statusCode: number;
    headers: Record<string, string>;
    body: string;
};

export const handler = async (): Promise<PresignedURLResponse> => {
    try {
        const ssm = new SSM({ region: process.env.REGION });

        const result: GetParametersCommandOutput = await ssm.getParameters({
            Names: ['ACCESS_KEY_ID', 'SECRET_ACCESS_KEY', 'BUCKET_NAME'],
            WithDecryption: true
        });

        if (
            !result ||
            !result.Parameters ||
            !result.Parameters[0].Value ||
            !result.Parameters[1].Value ||
            !result.Parameters[2].Value
        )
            throw new Error('Retrieval of SSM parameters failed');

        const ssmParams: { [key: string]: any } = {};
        result.Parameters.forEach((p) => {
            if (p.Name && p.Value) {
                ssmParams[p.Name] = p.Value;
            }
        });

        const client = new S3Client({
            region: process.env.REGION,
            credentials: {
                accessKeyId: ssmParams.ACCESS_KEY_ID,
                secretAccessKey: ssmParams.SECRET_ACCESS_KEY
            }
        });

        const command = new PutObjectCommand({
            Bucket: ssmParams.BUCKET_NAME,
            Key: uuidv4()
        });

        const url = await getSignedUrl(client, command, { expiresIn: 15 });
        return {
            statusCode: 200,
            headers: {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Origin': '*'
            },
            body: JSON.stringify({ url })
        };
    } catch (err) {
        console.error('Error: ', err);
        return {
            statusCode: 400,
            headers: {
                'Content-Type': 'application/json',
                'Access-Control-Allow-Origin': '*'
            },
            body: "Bad request"
        };
    }
};

To overview quickly:

• First we create a client for SSM, the secure parameter store that will house our sensitive strings. We want to retrieve the following values 'ACCESS_KEY_ID', 'SECRET_ACCESS_KEY', 'BUCKET_NAME'

  • (In a production scenario AWS Secrets Manager would be a better place to store the first two as it offers more robust encryption and rotation features at a greater cost. The process of retrieval is largely the same.)

  

• Next, we create an S3 Client with the credentials we retrieved from SSM, as well as instantiate a new PutObjectCommand. If we were going to submit a file to S3 from this Lambda we could do so with this client and command - the client would simply perform the command. However since we want our frontend to do that operation on its own, we want to use these two variables as arguments for getSignedUrl.

  

• We have our presigned URL! If no errors throw we will return this value back to the client.

• Finally, install these sub dependencies

  yarn add @aws-sdk/client-sso-oidc @aws-sdk/client-sts

Awesome, our Lambda directory now has what it needs for the Lambda function to be built. Earlier when we defined our Lambda resource we said that our code was going to live in 'src/lambdas/presignedUrlLambda/dist'. To create this dist directory and transpile our Typescript code into minified JavaScript, run:

yarn && yarn build

Inside of dist you should see a file called index.js. This file contains the code that will fulfill GET requests to our endpoint in API Gateway.

Finally, we can head back to our root directory and run

yarn cdk deploy --profile default

This will deploy our new stack to AWS! As part of a new stack deployment the CLI process will ask you to confirm your modifications, you should see something like this.

After our stack finishes deploying there are several things to do in the AWS console.

• We need to populate our Parameter store values that will be retrieved by the Lambda function.

  • Head to AWS Systems Management, and under Application Management click Parameter Store

  • We need to store three values. ACCESS_KEY_ID, SECRET_ACCESS_KEY, BUCKET_NAME

  • Find the first two in your credentials file, cat ~/.aws/credentials

  • Find your bucket name in S3, something like uploadserverlessstack-bucket839-ozntltq

  • Store them as SecureString

• Finally, head to API Gateway, click Stages on the left, and find our invoke URL. We can pass it to cURL to retrieve our first presigned URL:

curl --location 'https://YOUR_UNIQUE_API_ID.execute-api.YOUR_REGION.amazonaws.com/prod/put-object-presigned'

If successful our endpoint should respond with something like the following:

For the next 15 seconds this URL allows a web client to store a file directly in S3.

In the next entry we’ll create the frontend and see this architecture in action.

Stay tuned!