What is the best `as Code` tool in 2023?

Welcome

Based on the success of my latest article (CHATGPT!), I’ve decided to continue in that direction by creating shorter and more user-friendly content. Today, I have some thoughts and feelings to share about a complex topic. Recently, I was asked to prepare a sample infrastructure for an OKD cluster from scratch, without pre-built templates. I first used AWS CDK, then rewrote it to Pulumi, and finally to cdktf. You may wonder why I made these changes, and the answer is simple: I really enjoy my job!

In this blog post, I will highlight the things I liked about each tool and the areas where I think improvements can be made. Towards the end, I will also touch upon winglang and evaluate its usefulness after comparing it with the main tools.

Sample Architecture

Let’s start with the AWS-based solution architecture:

Tool first: AWS CDK

Pros

• Great documentation, better than others in my opinion.
• Supports a wide range of AWS resources.
• Pleasure to write in.

Cons

• Could be faster; the use of Cfn stacks slows down the process.
• Limited to AWS resources only.

General opinion

Personally, I really enjoy working with CDK, primarily due to its clean and detailed documentation. I have no issues with Cfn stacks, so using them under the hood is not a problem for me. CDK seems to be built with developers in mind, taking a very “dev” approach. For instance:

    somesg.connections.allowFrom(
      ec2.Peer.securityGroupId(supportSecurityGroup.securityGroupId),
      ec2.Port.tcp(22),
      "Allow ssh from supportSecurityGroup"
    );

Compared to cdktf:

        ingress: [
          {
            protocol: "TCP",
            fromPort: 22,
            toPort: 22,
            securityGroups: [supportSecurityGroup.id],
            description: "Allow ssh from supportSecurityGroup",
          },
        ]

I find cdktf’s approach more clear. However, I do appreciate CDK’s way of specifying managed policies:

      managedPolicies: [
        iam.ManagedPolicy.fromAwsManagedPolicyName(
          "AmazonSSMManagedInstanceCore"
        ),
      ],

Or getting the latest AMI image:

      machineImage: new ec2.AmazonLinuxImage({
        generation: ec2.AmazonLinuxGeneration.AMAZON_LINUX_2,
      }),

For newcomers to the IaC space, especially developers without previous Cfn or Terraform experience, CDK is a great tool. However, it might be a bit of a shock for hardcore old-style cloud administrators. Personally, I would use CDK for any new AWS pet projects or exploring new services.

Tool second: Pulumi

Pros

• It has a shiny and user-friendly interface.
• Pulumi AI (ChatGPT for Pulumi) is mostly effective.
• Modules work similarly to Terraform modules.
• Pleasure to work with, but…

Cons

• CLI and getting URN can be annoying.
• Closed state file, even for small teams, requires payment.
• Some confusion around modules, with aws, awsnative, and awsx having similar components in different modules.

Pulumi’s approach to policies, supported via LSP, is great:

  managedPolicyArns: [
    awsclassic.iam.ManagedPolicy.AmazonSSMManagedInstanceCore,
    awsclassic.iam.ManagedPolicies.AmazonRoute53FullAccess,
    awsclassic.iam.ManagedPolicies.AmazonS3FullAccess,
  ],

And they have TF-style egress/ingress rules:

    ingress: [
      {
        protocol: "TCP",
        fromPort: 22,
        toPort: 22,
        securityGroups: [supportSecurityGroup.id],
        description: "allow ssh from support machines",
      },
    ],

I find Pulumi to be a fantastic product that is easy to use and has good documentation (and dedicated LLM). Its syntax is simple, but it might be too costly for smaller teams with 2-3 members.

Tool third: cdktf

Pros

• It’s stable, well-known, and considered a standard in the field.
• Easy to use, with great support for TypeScript.
• Works smoothly and has a lot of modules.
• Terraform Cloud is free for small teams!
• Great toolchain, including Infracost or tfsec.
• Supports a wide range of providers.

Cons

• Documentation for CDK is lacking, and it can be challenging to work with; relying on LSP is often more helpful than Googling.

cdktf has a very classic way of working with policies:

      managedPolicyArns: [
        AmazonS3FullAccess.arn,
        AmazonSSMManagedInstanceCore.arn,
        AmazonRoute53FullAccess.arn,
      ],
// where before we need to use
const AmazonSSMManagedInstanceCore = new DataAwsIamPolicy(
      this,
      "AmazonSSMManagedInstanceCore",
      {
        arn: "aws::arn....",
      },
    );

As you can see, cdktf is essentially Terraform in TypeScript. It offers many third-party add-ons, a wide community, and a solid tool overall. Despite its less comprehensive documentation, I highly recommend it.

Extra tool?: winglang

Wing is what we call a cloud-oriented programming language. It allows developers to build distributed systems that fully leverage the power of the cloud without having to worry about the underlying infrastructure.

Now, let’s evaluate its usefulness compared to the main tools.

At first glance, Winglang seems interesting, with a visualization console and a readable language. However, it currently lacks support for neovim, so for a fair evaluation, I switched to VScode. As always, they use bucket as a showcase object, but I wanted to explore deeper and started looking for EC2 documentation. Unfortunately, the available standard libraries at this stage are limited:

• Api
• Bucket
• Counter
• Function
• OnDeploy
• Queue
• Schedule
• Secret
• Service
• Topic
• Website

As you can see, it’s not as straightforward. I decided to try building a static website using the Website library:

bring cloud;

let testWebsite = new cloud.Website(
  path: "./public",
  // let's skip it for now
  //  domain:
);

However, it produced multiple resource creations:

tf plan | grep "will be created"
  # aws_cloudfront_distribution.cloudWebsite_Distribution_083B5AF9 will be created
  # aws_s3_bucket.cloudWebsite_WebsiteBucket_EB03D355 will be created
  # aws_s3_bucket_policy.cloudWebsite_PublicPolicy_44BB71F3 will be created
  # aws_s3_bucket_public_access_block.cloudWebsite_PublicAccessBlock_18A70311 will be created
  # aws_s3_bucket_server_side_encryption_configuration.cloudWebsite_Encryption_6A8A4E29 will be created
  # aws_s3_bucket_website_configuration.cloudWebsite_BucketWebsiteConfiguration_920E8E41 will be created
  # aws_s3_object.cloudWebsite_File--hellohtml_7DB2E316 will be created

Interesting right? Ok let’s try to build regular ec2. The code itself looks strange, we need to import cdktf module, use cdktf.Token to pass diffrent types than string.

bring cloud;
bring "cdktf" as cdktf;

let bucketModule = new cdktf.TerraformHclModule(
  source:  "terraform-aws-modules/ec2-instance/aws",
  variables: {
    "name" => "single-instance",
    "instance_type" => "t2.micro",
    "monitoring"             => cdktf.Token.asString(true),
    "tags" => cdktf.Token.asString({
      "Terraform" => "true",
      "Environment" => "dev"
    }),
  }
);

Also tf init gave me…

Initializing the backend...

Successfully configured the backend "local"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing modules...
Downloading registry.terraform.io/terraform-aws-modules/ec2-instance/aws 5.2.1 for cdktfTerraformHclModule...
- cdktfTerraformHclModule in .terraform/modules/cdktfTerraformHclModule

Initializing provider plugins...
- Finding hashicorp/aws versions matching "4.65.0, >= 4.66.0"...
╷
│ Error: Failed to query available provider packages
│ 
│ Could not retrieve the list of available versions for provider hashicorp/aws: no available releases match the given constraints 4.65.0, >= 4.66.0

Seems that modules, aren’t the best possible idea. Let’s switch do aws-cdk module.

bring "aws-cdk-lib" as awscdk;

class BasicEc2Instance{
  instance: awscdk.aws_ec2.Instance;
  init() {
    this.instance = new awscdk.aws_ec2.Instance(awscdk.aws_ec2.Instance{
    });
  }
}

new BasicEc2Instance();

And still I’m unable to compile it. As there is no type awscdk.aws_ec2.Instance, and I have no idea how to find correct information, even for LSP support. examples, docs, etc. At this level I will just skip testing…

Summary

After testing these tools, I have decided to stick with cdktf. Despite its issues with documentation, I appreciate the entire ecosystem it offers. However, if you are an old-style administrator, you may prefer using plain HCL, as it is better documented and supported. For those who prefer newer tools and are primarily in the Amazon ecosystem, CDK might be the answer. As for Pulumi, its specific pricing model makes it more suitable for personal projects or multi-cloud endeavors. As for Winglang, if you are interested in buckets and functions, you can give it a try, but personally, I find its level of abstraction too high, and I prefer having more control over created resources.

Some people say that “we like boring solutions, as they are stable,” and that holds true, especially in the area of IaC tooling. The choice of CDK, Bicep (Cfn for Azure), Terraform, or other solutions depends on fitting into the organization’s culture and the team’s ways of working.

For teams consisting mostly of developers, Pulumi might be the best option with TypeScript support, allowing them to work with a familiar language. Larger organizations may opt for Terraform due to its stability and widespread adoption, especially as a multicloud solution, which is often a requirement for corporate managers, even if they are using only one cloud provider.

Solo projects, on the other hand, provide more freedom to explore and build quickly, even at the cost of some stability. In such scenarios, fast iteration and shipping may take precedence over using a more established tool.

Ultimately, the choice of an IaC tool should align with the specific needs and priorities of the project and the team involved.