AWS DAX Overview and Sample Application using Encryption At Rest

This is an overview with a sample application using AWS DAX – DynamoDB Accelerator. Further below is a sample application that uses DAX, DynamoDB and EC2 – all with encryption at rest.

DAX is a fully managed, in-memory cache for DynamoDB. It reduces DynamoDB response times from milliseconds to microseconds. As with other AWS services, DAX is a fully managed, highly available service. Some of its key benefits are:

Extreme Performance – consistent signle-digit milisecond latency
Highly Scalable – can select from different node cluster types and scale from there
Fully Managed
East of Use – original code used to interact with DynamoDB does not change. Refactor needed on the initial connection part to DynamoDB which is replaced with DAX.
Flexible – a single DAX cluster can service multiple DynamoDB tables
Secure – can use IAM, integrates with CloudWatch and supports VPC, there is also support for server-side encryption at rest

DAX is particularly beneficial in situations when you have unpredictable workloads. Prior to DAX this was done using client side caches. The problem with this is that often users will over provision those caches. DAX is scalable therefore it saves costs on these over provisioned situations. Another use case for DAX is when high throughput is needed – for example in live gaming (twitch) where hundreds of thousands are accessing data simultaneously and they need that real time performance.

DAX is only available as an EC2-VPC platform (no EC2-Classic). One limitation of DAX is that it maintains metadata of all attribute names it caches, even for those items that have expired or been evicted. Due to this, applications with varying data sets that have unbounded number of attribute names could exhaust the DAX cache over time. When does DAX remove cache data? There are two ways

TTL – time to live which can be configured (default is 5 minutes)
Least Recently Used (LRU) – those items being accessed the least are removed, also can be configured; the LRU algorithm is part of the DAX service

DAX is required to run in a VPC. The diagram below shows an example architecture of DAX being accessed by an EC2 application in the same VPC.

The sample application below will show how we use the DAX client to interacts with DynamoDB through DAX. DAX supports all I/O operations into DynamoDB, which are listed below. DAX does not support the creation, deletion or updates of tables. The operations DAX supports are:

GetItem
BatchGetItem
Query
Scan
BatchWriteItem
UpdateItem
DeleteItem
PutItem

During read operations the data will be returned from the DAX cluster if available. If it does not exist it will be read from DynamoDB and cached in DAX. For Query operations we can specifically call for cache queries so that it only retrieves what is in cache. For write operations, DAX will write to DynamoDB first and then to it’s cache cluster. The write operation is not completed until both write locations are done successfully.

To interact with DAX we access it through the cluster endpoint. This includes a port number.

myDAXcluster.2cmrwl.clustercfg.dax.use1.cache.amazonaws.com:8111

A DAX cluster can have one or more nodes. AWS suggests running with at least 3 nodes in a cluster. A single cluster can support up to 10 nodes with one primary node. The primary node is responsible for fulling DAX requests and handling operations with DynamoDB. For high availability, we can spread these nodes into different Availability Zones. However, note that DAX clusters and nodes can only interact with DynamoDB tables in the same region. (A single region can have several availability zones).

When creating or setting up DAX we can do it through the AWS CLI or Console. Below is a sample application deployed in EC2 that utilizes DAX.

Sample Application

This sample application is a .Net Core Web API application that is hosted in an EC2 Linux instance and connects to DynamoDB via DAX. To get this application running from scratch, we go through the following steps:

Create IAM User for Application
Create KMS Key for Encryption
Create and configure a DynamoDB table
Create and configure DAX
Create and configure an EC2
Deploy and Run application in EC2

Note that this .Net Core application requires the following AWS SDKs in order for it to work:

AWSSDK.DynamoDB
AWSSDK.DAX

These can be found on the following S3 site or downloaded through NuGet

http://dax-sdk.s3-website-us-west-2.amazonaws.com/

The complete source code can be found here:

https://github.com/TBD

Create IAM User

First step is to create an IAM account that will have access to the AWS services we need as well as owning the keys that will be used by the application. From IAM I go to user and create a new user. This account will be for programmatic acces only.

This account will need access to DAX and DynamoDB. I can set this up by attaching existing policies directly to the account. You will see an option for this in IAM. A sample policy for these services can be found here:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.client.create-user-policy.html

From IAM we paste the policy template shown in the link above.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dax:*"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "dynamodb:*"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ]
}

Below is a screen shot of how it looks in IAM

Create KMS Encryption Key

For this sample application I will be using encryption in all the data stores. To use encryption I must first setup a key in KMS. From the KMS dashboard I select “Create New Key” and set it up as shown below.

As with all resources in AWS I set the following tags.

In the KMS create wizard we are prompted to select IAM users that can manage and use this key. I select the same user account created in the section above.

Create and Configure a DynamoDB Table

For my sample application I use a single DynamoDB table. I create a new table from the DynamoDB dashboard. I have a partition key “pk” and sort key called “sk”. There is also a “data” column that contains the actual data but it is not shown on the Create Table page. Note that by default, DynamoDB enables encryption at rest. This is somewhat a new feature in AWS as part of new GDPR regulations set forth in 2018.

To see the encryption option you need to uncheck the “Use default settings” option.

DynamoDB supports encryption at rest using a default key or a user specified KMS key. The default option lets AWS create and manage the key that encrypts the database.

Create and Configure DAX

With the DynamoDB database created we are ready to create a DAX cluster. In the same DynamoDB dashboard there are links to the DAX dashboard on the left side navigation. I go to it and select the “Create new Cluster” button. I’m prompted with a form and fill it in as shown below.

For this sample application I selected the smallest Node Type available – dax.t2.small. I could have also gone with a single node (primary node only) cluster size but choose the smallest multi-cluster configuration which is 3. AWS recommends that in production you use at a minimum 3 node cluster sizes. Note that in DAX, the encryption is an option. By default it is selected but it can be unchecked. Lastly, I create a new IAM role that allows DAX to access DynamoDB. It will only have access to the Widgets table that was created in the section above.

As DAX creates multi-node clusters, it needs to have a subnet group configured to organize these nodes. I create a new subnet group as shown above. This subnet group still resides in my VPC and I use the same VPC as part of my security group to determine what resources can access DAX.

Create and Configure EC2

The next step is creating an EC2 instance that will host our application. I created an EC2 instance using the out-of-the-box AMI Linux with .Net Core (pre-baked) image. (Search for “ami .net” in the Ec2 search and it should be the first result).

During EC2 creating there is a step to enable encryption of the underlying EBS volume. Unfortunately the main root volume (default) cannot be encrypted at the time of EC2 instance creation. This needs to be done post creation. For now we just select the default settings for volume. See the end of this section below on encrypting the root volume.

Another item to note during the EC2 creation process are security groups which define who can access the EC2. The security group must be in the same VPC as the Ec2. The security group defines what types of access are allowed (SSH, FTP, etc) and the IP addresses the access is allowed from. More details about what security groups are and how they are used can be found here:

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html

During the EC2 creation process we can create a new security group or use an existing one. For this sample application I selected an existing security group that allows access from within the enterprise only.

When creating the EC2 we need to take note of the new user account it creates. We’ll be using it to SSH into the Ec2. As part of the EC2 launch process it will give an option to create a new key-pair. Download this (.pem file). Please refer to the EC2 AWS Documentation if more details are needed in creating key-pairs for EC2 access. For this sample application I choose to reuse an existing key I already owned.

That’s the last configuration step in creating the EC2. Once it launches and running, we need to go back and set encryption for the root volume.

Unfortunately AWS does not provide an encryption option when initially creating an EC2. There are methods of creating AMI copies on an encrypted volume which can then be used to create an EC2 instance (see link below for this method), however for this sample application I’ve already created an unencrypted volume so I need to go back and encrypt it. However, another unfortunate thing, AWS doesn’t have an easy way of encrypting existing volumes. The following steps are what I took to get a final encrypted volume for my EC2.

First we need to stop the EC2 if it is running.

Next we go to the EBS volumes page (link on the left navigation from EC2 Dashboard) and we need to detach the root volume. It should be in an “available” state once detachment has completed.

With the volume detached we can create a snapshot of it. Select “Action” button and the “Create Snapshot”. Enter a description and create the snapshot.

The snapshot can be found in the “Snapshots” page. There is a link to it in the left navigation under “ELASTIC BLOCK STORE”. Select the created snapshot and under the “Actions” button select “Copy”.

Here we can finally see the option to encrypt. I select it and select the key I created in the previous KMS section.

Once the encrypted copy of the snapshot is available, we can finally make a volume out of it. Select it and under the “Action” button select “Create Volume”.

In the following popup we can see that this volume has encryption enabled. Ensure that the Availability Zone is the same as the original EC2 instance (it should have copied over from the original snapshot copies). Create the volume and wait until it becomes available in the “Volumes” page (link in the left navigation). Select it and under the “Actions” button select “Attach Volume”.

In the popup we need to select the original EC2 instance created at the top of this section. Enter it’s instance id and the device name. You should be able to find both of these back on the “Instances” page under EC2 dashboard when highlighting the original EC2 instance that was created.

With the newly encrypted volume attached back onto the EC2 instance, we are ready to fire it up and test it out. From the “Instances” page select the EC2 instance and start it up. When it becomes available, we can test it out by trying to SSH into it. From a terminal, run the ssh command and include the key-pair file that was generated at the end of the EC2 creation step from beginning part of this section. The SSH command should look something like:

ssh -i my-keypair.pem ec2-user@public-dns-name

*Note that if you are unable to connect, confirm your firewall is not blocking the SSH connection.

Hopefully you are able to successfully SSH using the key. The following commands can be run to see the volume status.

[ec2-user@ip-00-00-00-00 ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
[ec2-user@ip-00-00-00-00 ~]$ sudo file -s /dev/xvda1
/dev/xvda1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)

Before moving on to deploying our application, we need to configure accesses. We want our application to be able to access DyanmoDB/DAX. There are few different ways of doing this. We could embed them into our application and reference them in each DynamoDB call. Or, I can configure them into the EC2 instance itself. This allows me to run my application freely without having to reference these keys. As long as my application is running inside this EC2, it will be able to use the keys and access DynamoDB. This is the option I took for this sample application. Complete instructions on configuring Ec2 can be found here:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.client.configure-ec2-instance.html

Deploy and Run Application in EC2

With the infrastructure all configured we’re finally ready to deploy our application. This application is based on .Net Core 2.1 and is a basic MVC based web application. That means it will run the web server using the built in Kestral server. Full sample code can be found here:

https://github.com/johnlee/daxdemo

First thing we need to do is deploy the app into EC2. There are a few different ways of doing this, such as through Visual Studio, FTP/SCP. I decided to take what I thought was the easiest approach for this sample and SCP the files onto the EC2. This can be done using the certificate (key-pair) we generated during the Ec2 launch process. I build up the app and zip it up into a file called daxdemo.zip. I transfer the that file onto EC2 with the command:

scp -i myec2cert.pem daxdemo.zip ec2-user@ec2-00000-000-000.us-west-2.compute.amazonaws.com

Next I SSH onto the EC2 server. This can be done with a similar command as above but calling ssh instead:

ssh -i myec2cert.pem ec2-user@ec2-00000-000-000.us-west-2.compute.amazonaws.com

Once on EC2 I need to unzip the file, then run the dotnet steps to run the application. Below is an example of how I did that on the EC2.

[ec2-user@ip-000-000-000-000 ~] mkdir daxdemo
[ec2-user@ip-000-000-000-000 ~] mv daxdemo.zip daxdemo
[ec2-user@ip-000-000-000-000 ~] cd daxdemo
[ec2-user@ip-000-000-000-000 ~] unzip daxdemo
[ec2-user@ip-000-000-000-000 ~] dotnet build
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 48.64 ms for /home/ec2-user/daxdemo/trydax.csproj.
  trydax -> /home/ec2-user/daxdemo/bin/Debug/netcoreapp2.1/trydax.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.38
[ec2-user@ip-000-000-000-000 ~] dotnet run

Note that this application tries to run specifically on port 8080. Therefore to access it through the web browser, we need to enter the Url plus the 8080 port number.

http://ec2-54-69-225-101.us-west-2.compute.amazonaws.com:8080/

Also note that this application requires an access key defined in the appsettings.json file. There are a few different ways to handle application access into DAX and DynamoDB. The way I have it currently coded uses a user account I created through IAM to grant access to both DAX and DynamoDB. This is done through policies. You can read more about how this is done specifically for DAX and DynamoDB here. There are many different approaches depending on your application’s requirements.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.access-control.html

Another way to control access for the application is on EC2 itself. We can configure the access key on EC2 and then all applications running on that EC2 would share the same policies. Further details of this approach can be found here:

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DAX.client.configure-ec2-instance.html