Introduction to AWS CDK
Malte Hallström / November 20, 2021
11 min read •
Background / Initiative
In order to future-proof the architecture and also keep it under version control in a clear and easy-to-understand way, I decided to give CDK a go.
Infrastructure requirements
The project that this was built around is weather-scraper
, a recurring job that contained the following steps:
- Download weather data
- Parse and enrich said data
- Run DB migrations and put data into database tables
- Clean irrelevant/old data
The goal was to turn this whole project into a container that could be deployed on ECS and run with regular intervals, perhaps 2-3 times / hour. In order to make this possible, a docker image was created that supported environment variables through docker-compose
(for local use) and through injection using ECS (explained later).
With some help, I was able to set up a basic configuration that integrated the following AWS services:
- ECS
- ECR
- RDS
- SM
- CloudWatch
Prerequisites
In order to get started, you need to be signed into the AWS SDK. You also have to install the AWS CDK CLI by running yarn global add aws-cdk
Also, make sure that you specify a default region by running aws configure
and following the steps. I used eu-west-1
.
The basics
The first thing I did was to create a CDK app by installing the CDK CLI () and then issuing cdk init app --language typescript
. Doing so left me with a folder structure like this:
The stack
The cdk-init-stack.ts
file was pretty basic to begin with, containing only this:
import * as cdk from "@aws-cdk/core";
export class CdkInitStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
// The code that defines your stack goes here
}
}
The stack defines a set of Amazon services. Some services depend on others, in which case variable references can be used within a stack in order to link them together.
The app
The cdk-init.ts
file was also quite empty, with the exception of some comments:
#!/usr/bin/env node
import 'source-map-support/register'
import * as cdk from '@aws-cdk/core'
import { CdkInitStack } from '../lib/cdk-init-stack'
const app = new cdk.App()
new CdkInitStack(app, 'CdkInitStack', {
/* If you don't specify 'env', this stack will be environment-agnostic.
* Account/Region-dependent features and context lookups will not work,
* but a single synthesized template can be deployed anywhere. */
/* Uncomment the next line to specialize this stack for the AWS Account
* and Region that are implied by the current CLI configuration. */
// env: { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION },
/* Uncomment the next line if you know exactly what Account and Region you
* want to deploy the stack to. */
// env: { account: '123456789012', region: 'us-east-1' },
/* For more information, see https://docs.aws.amazon.com/cdk/latest/guide/environments.html */
})
The app's purpose, as I later learned, is to tie the stacks together.
The configuration
The cdk.json
file had the following default content:
{
"app": "npx ts-node --prefer-ts-exts bin/cdk-init.ts",
"context": {
"@aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
"@aws-cdk/core:enableStackNameDuplicates": true,
"aws-cdk:enableDiffNoFail": true,
"@aws-cdk/core:stackRelativeExports": true,
"@aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
"@aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
"@aws-cdk/aws-kms:defaultKeyPolicies": true,
"@aws-cdk/aws-s3:grantWriteWithoutAcl": true,
"@aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
"@aws-cdk/aws-rds:lowercaseDbIdentifier": true,
"@aws-cdk/aws-efs:defaultEncryptionAtRest": true,
"@aws-cdk/aws-lambda:recognizeVersionProps": true,
"@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true
}
}
Useful commands
When using CDK, many common operations are done through the CDK CLI. From the documentation:
cdk deploy
deploy this stack to your default AWS account/regioncdk diff
compare deployed stack with current statecdk synth
emits the synthesized CloudFormation template
Customizing the template to fit the project
In order to satisfy the [[#Infrastructure requirements]], the existing files had to change.
First, all dependencies needed were installed and imported. This was the result:
import * as cdk from "@aws-cdk/core";
import * as ecs from "@aws-cdk/aws-ecs";
import * as ecr from "@aws-cdk/aws-ecr";
import * as ec2 from "@aws-cdk/aws-ec2";
import * as secretsmanager from "@aws-cdk/aws-secretsmanager";
import * as logs from "@aws-cdk/aws-logs";
Creating the main stack
The first step was to define the class that would represent our stack. In order to make it modular and compatible with different environments, it was also made sure that it was possible to pass parameters to the class on initialization.
Here is the resulting base class along with an interface that describes the options, placed in a file called cdk-stack.ts
:
export interface WeatherScraperStackProps extends cdk.StackProps {
environment: "development" | "staging" | "production";
ecrRepo: ecr.Repository;
vpcId: string;
postgresSecurityGroupIds: string[];
}
export class WeatherScraperStack extends cdk.Stack {
securityGroup: ec2.SecurityGroup;
constructor(
scope: cdk.Construct,
id: string,
props: WeatherScraperStackProps
) {
super(scope, id, props);
}
}
Where:
ecrRepo
will be a reference to a repository on the Elastic Container Registry. This is needed in order to reference and pull the latest docker images.vpcId
will be the ID of the Virtual Private Cloud where the database lives. This is needed in order to create security rules that can then be added to the security groups specified inpostgresSecurityGroupIds
postgresSecurityGroupIds
will be a list of IDs to the specific security groups within the VPC that are associated with the database that we need to connect to.
The task definition
AWS Task Definitions are used to uniquely identify tasks that can be run in ECS. They can reference one or more containers, and can be configured with a literal sea of parameters in typical AWS fashion.
In order to add a task definition to the stack, the following code was used:
const taskDefinition = new ecs.Ec2TaskDefinition(this, "WeatherScraperTask", {
networkMode: ecs.NetworkMode.AWS_VPC,
});
The second parameter of the Ec2TaskDefinition
constructor is the unique id
of the task definition, which here will be the name of the project.
The networkMode
is the Docker networking mode to use for the containers in the task. The valid values are none
, bridge
, awsvpc
, and host
.
The security rules
As previously mentioned, we need to configure security rules in order to securely connect to RDS (in this case a Postgres database).
To do that, we first create a referene to the VPC in question using this bit of code:
const vpc = ec2.Vpc.fromLookup(this, 'Vpc', { vpcId: props.vpcId })
this.securityGroup = new ec2.SecurityGroup(this, 'WeatherScraperSecurityGroup', { vpc })```
Were:
- `props.vpcId` is explained in [[#Creating the main stack]]
- The second argument of `ec2.SecurityGroup` is the unique id of the security group that will be created.
- `vpc` is a direct reference to a VPC, in this case the one where the database is located.
This is not all, however, since we still have not added any security rules to the database. In order to do that, we can use this code:
```typescript
props.postgresSecurityGroupIds.forEach((pgSgId) => {
const pgSecurityGroup = ec2.SecurityGroup.fromSecurityGroupId(this, `PostgresSecurityGroup${pgSgId}`, pgSgId)
pgSecurityGroup.addIngressRule(this.securityGroup, ec2.Port.tcp(5432), `Weather Scraper ${props.environment}`)
})
Where:
postgresSecurityGroupIds
is explained in [[#Creating the main stack]]SecurityGroup.fromSecurityGroupId
imports an existing security group by ID. It is assumed that the group allows all outbound traffic, and thus it is only possible to add Ingress rules.addIngressRule
adds an inbound rule to the already existing security group. By specifying thesecurityGroup
that we created earlier, we make sure that all coming services that use that rule will be able to access the database.
The logging
In order to send the stdout of the container to CloudWatch, the following simple code snippet was used:
const logGroup = new logs.LogGroup(this, "WeatherScraperLogGroup", {
logGroupName: `weather-scraper-${props.environment}`,
removalPolicy: cdk.RemovalPolicy.DESTROY,
retention: 1,
});
Where:
- Most parameters are self-explanatory
retention: 1
will result in the logs being kept for 1 day.
The secrets!
In order to connect to the database, credentials was of course needed along with a valid hostname. The container used here can read these from the environment, no build arguments necessary. This ultimately makes it possible to run the same image in any environment (development / staging etc.) without rebuilding.
In order to safely provide the container with these secrets, I wanted a way of specifying them using references to secrets stored in the [[AWS Secrets Manager]]. The secrets already existed, there just had to be a way of fetching them from inside the CDK script.
Fortunately, this was very simple using CDK:
const postgresSecrets = secretsmanager.Secret.fromSecretNameV2(
this,
`PostgresSecret${props.environment}`,
`postgres-${props.environment}`
);
Using the now-defined postgresSecrets
variable, I could write the following helper function to retrieve secrets by key:
const getSecret = (key: string) =>
ecs.Secret.fromSecretsManager(postgresSecrets, key);
This will be used in the next section.
Adding the container to the task definition
With the above sections out of the way, all the necessary infrastructure was in place. All that was left to do was to add the actual container to the mix, as well as specifying the specifications of the machine that would run it.
taskDefinition.addContainer(`WeatherScraperContainer`, {
image: ecs.ContainerImage.fromEcrRepository(props.ecrRepo, props.environment),
memoryLimitMiB: 2048,
memoryReservationMiB: 1024,
cpu: 800,
logging: ecs.LogDriver.awsLogs({
logGroup,
streamPrefix: "/ecs/WeatherScraper",
}),
secrets: {
PGPASSWORD: getSecret("password"),
DB_HOST: getSecret("host"),
},
});
Here, you can see the results of some of the earlier sections being used:
- The
ecrRepo
, a reference to the repository to which new container images are pushed. - The
getSecret
helper, used to fetch an environment-specific database secret from Secrets Manager.
Tying it all together
In order for this stack to execute successfully, we need to specify some parameters. This can be done in a separate file, see [[#The app]].
The first thing to note is that the stack currently relies on being passed a reference to an ECR repository. This can also be done using the CDK! I created a new file called ecr-stack.ts
and put the following in it:
import * as cdk from "@aws-cdk/core";
import * as ecr from "@aws-cdk/aws-ecr";
export interface EcrRepoStackProps extends cdk.StackProps {
appName: string;
}
export class EcrRepoStack extends cdk.Stack {
repo: ecr.Repository;
constructor(scope: cdk.Construct, id: string, props: EcrRepoStackProps) {
super(scope, id, props);
this.repo = new ecr.Repository(this, "EcrRepo", {
repositoryName: `${props.appName}`,
});
}
}
It was then time to bring it all together in a file called cdk-app.ts
:
#!/usr/bin/env node
import "source-map-support/register";
import * as cdk from "@aws-cdk/core";
import { WeatherScraperStack } from "./cdk-stack";
import { EcrRepoStack } from "./ecr-stack";
const app = new cdk.App();
const ecrWeatherScraperRepo = new EcrRepoStack(
app,
"EcrWeatherScraperRepoStack",
{
appName: "weather-scraper",
env: {
region: "eu-west-1",
account: "282915464914",
},
description: "Scraping weather and importing into postgres DB",
tags: {
application: "weather-scraper",
realm: "charts",
},
}
);
First, a new CDK application is initialized and its value is assigned to the app
constant.
Then, a reference to a new ECR repository is created using the class we defined earlier and some new parameters:
appName
, the name of the repository shown in the ECR page in the AWS console.env
, the AWS environment (account/region) where this stack will be deployed.
Note: If either
region
oraccount
are not set, the Stack will be considered "environment-agnostic". Environment-agnostic stacks can be deployed to any environment but may not be able to take advantage of all features of the CDK. For example, they will not be able to use environmental context lookups such asec2.Vpc.fromLookup
.
Finally, we initialize the stack that we have been working on for the most part of this document using the following code:
new WeatherScraperStack(app, "WeatherScraperStackDevelopment", {
env: {
region: "eu-west-1",
account: "282915464914",
},
vpcId: "vpc-0987bc6c",
postgresSecurityGroupIds: ["sg-6af69013"],
environment: "development",
ecrRepo: ecrWeatherScraperRepo.repo,
tags: {
application: "weather-scraper",
realm: "charts",
environment: "development",
},
});
This creates an environment-specific stack, in this case for development
.
When it comes to the parameters:
env
is the same as for the section abovevpcId
andpostgresSecurityGroupIds
are explained in [[#Creating the main stack]]- The previously defined
ecrWeatherScraperRepo
variable, referencing an ECR repository, is used as value for theecrRepo
option.
Final steps
And that's it! With all of the above steps done, we now have a full infrastructure defined in code. All that's left to do is to synthesize a CloudFormation template from the code, which will catch common errors. This can be done using cdk synth --all
After confirming that the output looks OK without any errors, the final command can be run: cdk deploy --all
. With a bit of luck, your environment will now be fully configured and ready to use!
In this particular case, the next step would be to [[Schedule Tasks in ECS]]