Develop and Deploy a REST API in AWS With API GW + LAMBDA + S3 FOR FREE!!

11 min readOct 14, 2020

In this article and some others upcoming, I pretend bring closer into AWS technologies a very global community of IT engineers regardless if they are Developers and/or System Administrator, I will try to use a very clear language on all layers so everyone who has a bachelor degree on information technology would understand completely.

Before beginning with this guide I guess that the person who reads this article has the knowledge and the understanding about how Cloud Computing works and which are their Service Models and the differences.

Here comes a lot of theory so grab your cup of coffee , you could skip the theory and go to the HANDS ON section, but you should consider read the theory, because the strategy behind adopting the cloud technologies successfully is understanding the main purpose of each service and its features.

At the end of this article I hope the readers understand at a basic level the following AWS services:

AWS LAMBDA.
AWS API GATEWAY.
AWS S3 and Select Feature.

AWS LAMBDA

AWS Lambda is a Service categorized as FaaS in the cloud computing Service Models, it is better known as Serverless Computing.

the main difference between an application running in a PaaS model and a Serverless Based (FaaS) Application is when you use a PaaS solution you expect that your app is always running regardless of the middleware (Jboss — Websphere -Tomcat) and or runtime used.

When you use a Serverless solutions like AWS LAMBDA your application only runs while its invoked, so the cloud provider starts a container with the needed runtime to run your code, each time your lambda is invoked AWS will create a new container to run the code, this container just die when the function return a result or an exception. This is how your application could become on a highly scalable solution.

In the following image explain a little bit better the process of AWS bringing up new lambdas to support the Function workload.

Principal Considerations while designing and developing a Lambda Function:

The function must be stateless, this is because a lambda can’t guarantee the data persistence when it finish the execution, so you can integrate your function with services like AWS DynamoDB or AWS S3.
The approach of developing a Serverless Application is get a Solution highly decoupled and very scalable.
AWS Lambda could fail so you need to consider a strategy to persist the data and process it after a system failure (Fault Tolerant), for this you could use a service like AWS SQS.
You will be billed for the consumption only, so you need a very clean code that return a result fast or fail fast if is required. you can see here the pricing of AWS Lambda.
You can have as much functions as needed and you will pay only for the consumption so you should apply the Single Responsibility Principle while defining the lambdas of your application architecture.

AWS S3

Amazon Simple Storage Service is a storage solution based on objects this is high scalable on demand storage, the service provide you an API to communicate, with this service you don’t need to worry about durability and resilience because these features are built in with the service.

By now i will not explain too much about Object Storage system, this will require another complete post, but you must understand that this service doesn’t not works like a FileSystem or a NAS (nfs, ftp, smb).

Main things of AWS S3 to mention:

AWS S3 use Buckets (Like a folder) to store the data, the buckets must be unique within the region and can be publicly accessible or private.
The way to access a bucket is through an URL like this one <your-bucket-name>.s3.<region>.amazonaws.com
The files are stored within the Buckets with an unique key and you can access it using an URL like this one https://bucket-name.s3.us-east-1.amazonaws.com/test.txt

AWS API GATEWAY

AWS API Gateway is a service categorized as a PaaS the main purpose of implementing an API Gateway is to expose RESTful API’s (only HTTP based) even you can integrate WEBSOCKET as long as it communicates over HTTP.

Maybe some of you could ask “Why should I use an API Gateway instead of exposing directly my API?” for some people the answer is just obviously, but lets see some points about the pros of using AWS API Gateway.

Publicly accessible through Internet.
Native integration with other AWS services.
Single entry and administration point.
Routing based on PATH.
Transformations on REQUEST and on RESPONSE.
Authentication based on OAuth and OpenID.
Create quotas based on a possible service plan for your API.
Create Policies to handle access to the API or some specified methods based on source IP address and other criteria.

The list could become more and more long and there are too some cons about using AWS api gateway but at this point we will not talk about this.

HANDS ON…

The Problem

Let’s purpose a hypothetical problem, think that you have a Legacy software developed on an old framework and language version, so this system generates some outputs as CSV files or another plain formats, the point is the system doesn’t store this info on a DB or expose some API to generate those reports.

The business ask you to integrate this system reports with other systems as soon as possible, there is some ways to accomplish with this requirement:

Create an ETL to load the data on other systems (I don’t like too much this one).
Update the source code and develop the needed logic to store the info on a DB and create a RestFull API to expose this info. (if you choose this option you have 2 topics that hurts 1- What about the time that could take update this old app? 2- You need to develop too the API to expose the info).
Create an interface on each software to import the CSV (This is not the best way to do it but is an option)

Let’s see an alternative using AWS Services

We will expose an API through AWS API Gateway and this will be integrated with a LAMBDA that will execute a Select with SQl syntax on a CSV file stored in AWS S3.

We will create the API Gateway first than all.

Follow the steps:

1.Go to the AWS Console → Services (Top left) → API Gateway.

2. In thee API Gateway section just click on build in “REST API”.

3. While creating the REST API just Specify a name for Resource.

4. Now we have an API Gateway we can start creating the HTTP methods and Resources.

5. Select the created resource and add a HTTP Method

6. We will create a MOCK just to try and validate our API Gateway.

7. Select the resource METHOD and go to the Integration Response Mehtod.

8. Drop down the options from the default method response, in the MApping templates select the default Content-Type and paste in the template anything and click the save button.

9. At this point we had been designing our API Gateway configuration, it’s time to deploy this config, select in the Actions “Deploy API” in the Deployment Stage select “New Stage” and in the “Stage name” you can set a name for the stage example: DEV, TEST, PROD.

Important: After any change on API definition, to apply the change you must Deploy the API on an existent or new stage.

10. After deploy the API can go to stages section and there you will se an Invoke URL, this will be the URL of your API gateway.

The URL will be composed in the following format always “https://API-GW-ID.execute-api.THE-AWS-REGION.amazonaws.com/THE-STAGE-NAME".

11. Lets try our mock just executing the stage URL + our resource in our browser(by default any web browser will execute a GET operation)

Now our API GATEWAY is deployed and working! lets go to configure and develop our LAMBDA, we will integrate API GATEWAY and LAMBDA later.

You can clean the resource created for the mock.

Now, let’s create a Lambda

1.Go to the AWS Console → Services (Top left) → LAMBDA.

2. Select the option Create Function → Author from scratch → specify a function name By default, Lambda will create an execution role with permissions to upload logs to Amazon CloudWatch Logs. You can customize this default role later when adding triggers.”

After the lambda creation you will see the following default sample code:

exports.handler = async (event) => {
    // TODO implement
    const response = {
        statusCode: 200,
        body: JSON.stringify('Hello from Lambda!'),
    };
    return response;
};

3. Now we must to integrate AWS API Gateway with the LAMBDA, select the “Add trigger” option

4. Now select API Gateway as trigger source, select the previously created API Rest and the stage, the security just leave it open we will secure our api later.

Now the AWS Lambda service has created a new API Gateway resource in this case with the same name of the LAMBDA.

Before test the “Hello from Lambda” remember DEPLOY the API after any change in the API Gateway console.

Now let’s test the lambda from the API endpoint.

GREAT!!

AWS S3 Bucket creation and Select Feature

1.Go to the AWS Console → Services (Top left) → S3 → Click on Create Bucket

2. Specify a name for your bucket, remember the bucket name must be unique so is possible that if you put common words like ‘’test’ this will be already in use, Select the region where you want to create the bucket.

3. Now just click on next button until the bucket get created, we will leave all configs as it is, by default AWS will create the bucket private blocking all public access.

For this case, I will download the historical COLCAP index pricing of the Colombian stock market, this info is publicly accessible via BANREP site here.

Let’s modify the file first rows and leave those as headers.

4. Upload the the file into the bucket and click on the uploaded file and select the tab “Select from”.

Añadir texto alternativo

In the preview Section Click on Next Button And let’s execute the following query.

select * from s3object s WHERE s.”Date” = ‘13–10–2020’

You can go to de AWS S3 Select command documentation to see how to create a more complex queries and you should always see the services and features limitations

Now that we know how AWS S3 Select works let’s create a LAMBDA to integrate AWS API Gateway with S3 and expose the result of the SQL query as a JSON.

Integrating AWS LAMBDA with AWS S3

1.First we need to give enough permissions to LAMBDA to be able query the content from the object in the bucket.

How to assign this permission? remember when you were creating the LAMBDA that i mention the Lambda will create an execution role with permissions to upload logs to Amazon CloudWatch Logs, so let’s modify this role and assign a new inline policy.

Go to the AWS Console → Services (Top left) → IAM → Click on Roles section.

Now you will see a role with the lambda name

To add the permissions you have 2 options attach an existing policy there is a lot a policies designed by AWS or add a policy inline.

If you choose Attach policy just add the managed policy AmazonS3ReadOnlyAccess”

If you choose the Inline policy you should generate the policy you can do it with a JSON specification or with the console, the JSON will look like this one.

2. Now you can write your code using the AWS SDK in this case I will leave you a sample code in JavaScript let’s see the documentation AWS SDK for Javascript.

You can download this code from GitHub here

The Code will look like this.

Important things to consider while writing your own code:

You must to understand how AWS API Gateway sends the Client request to the lambda so you will know how to handle the Path Params, Query Params, the Body in case of a POST request, everything related with the HTTP request. let’s check the docs here.
You must to return always a response with the following format, see in the Sample Code the return statement.

When returning a JSON you must convert the object in a string you can use JSON.stringify with NodeJS.

3. Let’s test the API from the Browser or Postnam.

In the code we capture the param “date” to concatenate it with the SQLQuery.