Localstack S3 and Java Lambda Example in Docker
Published
Updated
Introduction
The LocalStack project allows you to create an AWS cloud stack on your local development machine. It grants you an incredible number of different features that are available with a standard AWS instance. Additionally, you can interact with this application stack as though it is a real AWS cloud server.
As you can imagine, this can be incredibly useful for a variety of use cases, such as creating mock services or rapidly prototyping cloud native applications without having to worry about any costs incurred by creating infrastructure. While tools such as Moto exist allowing you to mock AWS services, LocalStack takes it a step further and allows you to have process isolation and fully coupleable services that can interact with one another.
As part of this article, I will guide you and create a Java Lambda which will execute via a step function. The step function is triggered upon a file landing within S3 and will read data in from the aforementioned file.
Licensing and Features
There are two offering tiers of LocalStack available, a free community edition and a pro edition which costs approximately $20 USD a month per license, but adds a significant number of features which may prove valuable to enterprises and organizations.
As this article is targeted towards the Community Edition, here are a few of the available features:
- AWS Certificate Manager (ACM)
- Amazon API Gateway
- CloudFormation
- CloudWatch
- DynamoDB
- Elastic Compute Instances (EC2)
- ElasticSearch
- S3 Object Storage
- Simple Notification Service (SNS)
- Message Queuing Service (SQS)
- Step Functions
- and many more…
Getting Started
You will need the following items installed or configured.
- Python v3 installation
- Maven installation
- OpenJDK 15 or greater
- AWS Command Line Interface (CLI)
- Docker
If you are a Mac user, all of these can easily be installed with Homebrew:
# Install Python v3
brew install python3
# Install Maven via Homebrew
brew install maven
#Install OpenJDK latest via Homebrew
brew install openjdk
# Install Docker, the machine images, and the VM drivers with homebrew
brew install docker
brew install docker-machine
brew install docker-machine-driver-xhyve
You can install the latest version of LocalStack using the Python Package Manager (pip) using the following command:
pip3 install localstack
Creating the Docker Compose Image
The following docker-compose.yml YAML file is for our Docker Compose image and it will pull the localstack/localstack image from Docker Hub.
I have placed comments next to the Docker port number bindings so you can easily distinguish which port number correlates with what service.
version: '2.2'
services:
localstack:
container_name: "LocalStack-Demo"
image: localstack/localstack
network_mode: bridge
ports:
- "4566:4566" # Edge Port
- "4569:4569" # Dynamo DB Port
- "4571:4571" # Elasticsearch Port
- "4572:4572" # S3 Port
- "4574:4574" # Lambda Port
- "8080:8080" # Web-UI Port
Starting LocalStack via Docker Compose
Open a terminal window and traverse to the directory which contains your docker-compose.yml file. Execute the following to initialize and start your LocalStack services:
docker-compose up
Preparing our Lambda Code
Please note, the aws-lambda-java-* libraries are not a part of the AWS API and are distributed separately from the aws-java-sdk dependency group. Please see my article for more information regarding dependency management within the AWS Java SDK.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.codetinkering</groupId>
<artifactId>localstack-example</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<properties>
<java.version>8</java.version>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bom</artifactId>
<version>1.11.939</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-lambda-java-events</artifactId>
<version>2.2.9</version>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-s3</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.2</version>
<configuration>
<createDependencyReducedPom>false</createDependencyReducedPom>
</configuration>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
The Maven shade plugin will bundle all the dependencies into a single, monolithic jar file. This is necessary for Lambda execution.
Creating a Bucket Event Handler in Java
Create a class called BucketHandler which will serve as an implementation of the AWS RequestHandler interface. This class will be later bound to any file creation events generated by our S3 bucket. It will print the contents of any files added to the console.
package example;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import java.io.IOException;
import java.io.InputStream;
public class BucketHandler implements RequestHandler<S3Event, String> {
public String handleRequest(S3Event s3Event, Context context) {
// Pull the event records and get the object content type
String bucket = s3Event.getRecords().get(0).getS3().getBucket().getName();
String key = s3Event.getRecords().get(0).getS3().getObject().getKey();
S3Object obj = prepareS3().getObject(new GetObjectRequest(bucket, key));
try (InputStream stream = obj.getObjectContent()) {
// TODO: Do something with the file contents here
stream.transferTo(System.out);
System.out.println();
} catch (IOException ioe) {
//throw ioe;
ioe.printStackTrace();
}
return obj.getObjectMetadata().getContentType();
}
...
The following method instantiates the AWS Client Builder and configures the endpoint to our LocalStack instance. The credentials provider could potentially be replaced with alternative source such as an HSM or a secure credential storage.
...
public final String AWS_REGION = "us-east-1";
public final String S3_ENDPOINT = "http://localhost:4566";
private AmazonS3 prepareS3() {
BasicAWSCredentials credentials = new BasicAWSCredentials("foo", "bar");
AwsClientBuilder.EndpointConfiguration config =
new AwsClientBuilder.EndpointConfiguration(S3_ENDPOINT, AWS_REGION);
AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard();
builder.withEndpointConfiguration(config);
builder.withPathStyleAccessEnabled(true);
builder.withCredentials(new AWSStaticCredentialsProvider(credentials));
return builder.build();
}
}
Compile your code with Maven using the following command:
clean compile package
Initializing Services via AWS CLI
Now that our LocalStack has been initialized and is running, we will want to prepare our various AWS services within it. As part of this exercise, we are going to prepare the following:
- Create an S3 Bucket
- Create a serverless Lambda Function which will execute a jar file
- Establish a put-bucket-notification which will trigger our Lambda
Creating an S3 Bucket with LocalStack
Using the AWS CLI, the following command will create an S3 bucket called mybucket.
aws s3 mb s3://mybucket --endpoint-url http://localhost:4566
If you haven’t used the AWS CLI before, you will likely receive an error Unable to locate credentials. Simply execute the following export commands in your console and re-run the make bucket command. Don’t worry what the values are, LocalStack disregards any provided credentials. The AWS CLI only requires that something is set.
export AWS_ACCESS_KEY_ID=foo
export AWS_SECRET_ACCESS_KEY=bar
If you receive an error Connection was closed before we received a valid response from endpoint URL, try using port 4566 instead. Sometimes you can encounter this issue if you are attempting to connect directly to the s3 endpoint instead of the edge service.
Creating a Lambda within LocalStack
Run the following command to create a Java Lambda function within LocalStack. It will reference your compiled jar file from the previous step. Change function-name to an appropriate value to match your function.
aws lambda create-function \
--endpoint-url http://localhost:4566 \
--function-name examplelambda \
--runtime java8 \
--handler example.BucketHandler \
--region us-east-1 \
--zip-file fileb://localstack-example-1.0-SNAPSHOT.jar \
--role arn:aws:iam::12345:role/ignoreme
After creating the Lambda, you will receive a confirmation and details in JSON format.
{
"FunctionName": "examplelambda",
"FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:examplelambda",
"Runtime": "java8",
"Role": "arn:aws:iam::12345:role/ignoreme",
"Handler": "lambda.S3EventHandler",
...
}
Create a JSON file called s3hook.json with the following contents. This file will be used to bind the lambda to the S3 Object Creation event.
{
"LambdaFunctionConfigurations": [
{
"Id": "1234567890123",
"LambdaFunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:examplelambda",
"Events": [
"s3:ObjectCreated:*"
]
}
]
}
Registering our Lambda to S3 bucket events
Finally, we need to bind the lambda to the put-bucket-notification event within LocalStack. Any time that an object is created within our mybucket S3 bucket, it will invoke our earlier Java code.
aws s3api put-bucket-notification-configuration --bucket mybucket --notification-configuration file://s3hook.json --endpoint-url http://localhost:4566
Invoking a Lambda within LocalStack
Create a simple text file called samplefile.txt and run the following command to transfer the file to S3, thereby triggering the Lambda:
aws s3 cp samplefile.txt s3://mybucket/samplefile.txt --endpoint-url http://localhost:4566
AWS SDK and Lambda ClassCastException
java.lang.ClassCastException: class com.amazonaws.services.s3.event.S3EventNotification$S3EventNotificationRecord cannot be cast to class com.amazonaws.services.lambda.runtime.events.models.s3.S3EventNotification$S3EventNotificationRecord
I encountered this strange error and it appears the aws-lambda-java-events SDK version 3.x.x is incompatible with the S3Event object passed by the Request Handler. To solve this, simply downgrade the version of the aws-lambda-java-events library to version 2.2.9. You can find more details on this issue in Github Issue #2852.
Checkout this project from Github
git clone https://github.com/code-tinkering/localstack-exampleDownload Zip