Cloud on The Cloud Optimist

Cloud Sovereignty: Getting Started with Scaleway

Sun, 17 May 2026 07:30:00 +0100

The Day I Accidentally Deleted An Api and What It Taught Me About Devops

Tue, 17 Feb 2026 07:30:00 +0100

April 4, 2024, had started off so well. A hot coffee in hand, VS Code humming with activity, and smooth electronic music in my ears—all the ingredients for a perfect day that surely couldn’t foreshadow the coming disaster.

Today, I’d like to tell you how a seemingly quiet day became a defining moment in my DevOps career!

Incident

The Context

At the time, I was working in a team tasked with providing a global API Gateway on AWS, featuring several endpoints managed by different teams. Specifically, setting up this API Gateway was the initial phase of the project. It included a DNS record pointing to the API, the API Gateway itself, and a Cognito Authorizer configured with several clients.

Once this API was ready, external teams could deploy their own endpoints onto it. To facilitate this, a CI/CD pipeline was in place, using CloudFormation to attach the endpoints directly to the existing API.

I’m keeping it as simple as possible here, but if you’re interested in the technical nitty-gritty, feel free to reach out!

On my end, besides being responsible for the global API infrastructure, I occasionally worked on or bootstrapped certain services.

The Beginning of the Drama

If you’ve ever worked with CloudFormation, you might be familiar with the dreaded UPDATE_ROLLBACK_FAILED status. This happens when you try to update a CloudFormation stack, but it encounters an error. It then tries to roll back, but if that rollback fails too, your stack gets stuck in UPDATE_ROLLBACK_FAILED.

This state is particularly annoying because you cannot trigger any new deployments until you’ve resolved the underlying issue.

That is exactly what happened on that fateful April 4, 2024. One of our services got stuck in this state, and I started investigating the “why” and the “how.”

After a few minutes, I noticed the problem stemmed from the API resource itself. Without overthinking it, and in an effort to unblock the situation quickly, I headed straight to the AWS Console, specifically to the API Gateway service. I found the resource in question and rushed to delete it.

At that exact moment, something very strange happened. An unexpected behavior that made my blood run cold. Instead of staying on the same page, AWS redirected me to the main API Gateway dashboard—the page that lists your available APIs. It now displayed a count of 0.

I realized then that I hadn’t deleted a specific resource within the API, but the entire API Gateway itself.

Anatomy of a Failure

I never thought I’d make such a rookie mistake. And I imagine as you read this, you’re thinking the same thing.

Because to get it that wrong, you really have to try!

Let me reconstruct the crime scene for you. Here is what I saw when I decided to delete a resource from the API Gateway.

Any resemblance to an actual situation is entirely coincidental.

In my shoes, which button would you have clicked? Easy! The Delete button right next to the resource!

As for me (and I still don’t know what possessed me), I chose to click the API actions button. After all, I did want to perform an “action”!

And what happens when you click that button?

Yikes! Right away, you can see it’s not what we want at all. But do you think that stopped me? Not a chance! I came here to delete something, and when I saw the word Delete, I stopped thinking. I should have kept reading, because then I would have seen the word API right next to it.

Fortunately, our friends at AWS thought of everything! When you click Delete API, they still ask if you are absolutely sure. A nice dialog box pops up, showing the name of the API and asking you to type the word confirm to validate the operation.

That’s all well and good, but the AWS engineers underestimated my impatience. In that moment, the confirmation request wasn’t a warning; it was an obstacle to my goal. In a heartbeat, I typed confirm and hit enter.

So, here is the last thing I saw before finally realizing the gravity of my mistake:

A vision of pure horror.

I wanted to walk you through this process to make one thing clear: you can put all the safeguards in the world in place, but you can’t protect a system against an impatient individual, because they won’t stop to read the warnings.

So, the next time you have to perform an “innocuous” action, take the time to read and ensure you are actually on the right path.

That being said, let’s move on to the crucial stage: resolving the incident!

Resolution

The Click

Back to the moment of impact. After realizing what had happened, I had a sudden jolt—a moment of clarity (without the aid of any white powder, I promise). I knew what I had to do: resolve the incident, but more importantly, document every single action I took.

A few weeks prior, I had looked into the 2017 GitLab incident, which took the service down for hours and resulted in data loss for some users. That’s how I discovered the concept of a Post-Mortem and why they are so vital in these situations. But I’ll save that for the next section.

In the meantime, if you’re interested in these topics, I highly recommend Kevin Fang’s YouTube channel where he, in his own words, “reads postmortems and makes low-quality videos about them.”

Rollback, Impact, and Communication

My first thought was that there might be a way to trigger a rollback directly within AWS to minimize the impact on users. If I had actually read the warning message earlier, I would have known that was impossible (but if I had read it, I wouldn’t have been in this mess anyway).

With no “undo” button in sight, I accepted that I was facing a major incident with a global impact. I first set out to map out the full extent of the damage. In my case, not only was the API down (thanks, Captain Obvious), but no new deployments could be made until the API was back online.

Finally, I made sure to notify the internal team. This kept them in the loop and let them know I was actively working on a fix.

Analysis and Troubleshooting

It was time to roll up my sleeves and find a way to redeploy the API Gateway.

First, I went to the CloudFormation service, remembering that this API was originally deployed through it. I tried updating the stack, hoping it would bring my dear API back like magic.

Obviously, it wasn’t going to be that simple. Updating the stack was impossible because the manual deletion had put the stack into a “hybrid” state that it couldn’t reconcile.

Since updating was out of the question, the logical next step was to delete the stack and redeploy it from scratch. That’s when the real trouble started. This CloudFormation stack produced several Outputs. Two of these were required by “child” stacks that had deployed their endpoints on the API. Consequently, CloudFormation blocked me from deleting the stack because it would break all the others.

After brainstorming alternatives, this heavy interdependence led me to a tough decision: I had to delete all the “child” stacks in CloudFormation—a total of 81 stacks.

To make matters worse, these child stacks didn’t have identifiable tags that would have allowed us to automate the cleanup. Fortunately, most of them used a recognizable name prefix, which allowed me to clear out the bulk of them manually.

Did I mention interdependencies? Because we’re not done! Some stacks had deployed S3 buckets. And guess what? CloudFormation won’t delete a stack if the S3 bucket isn’t empty. Naturally, 14 stacks got stuck in DELETE_FAILED. Luckily, the fix is straightforward: back up each bucket, empty it, and retry the stack deletion.

Deploying the API: Light at the End of the Tunnel?

Having finally cleared the web of interdependencies, it was time to delete the original API Gateway stack and redeploy it.

The deletion went smoothly (thank God), but—naturally—the creation did not.

First, the stack itself. A YML file existed in a GitHub repo, but it hadn’t been updated in ages. I knew I was better off using the stack definition stored within CloudFormation (and yes, I had saved a copy—I’m not that crazy).

This stack didn’t just deploy the API Gateway; it handled several AWS resources, including Lambdas. These Lambdas were still running on Python 3.7, a version that AWS no longer allows for creating new Lambdas. Fortunately, a quick upgrade to Python 3.12 was enough to satisfy the AWS gods.

And, to my surprise, the stack finally deployed without a hitch!

But to keep a long story short, there was still work to do. Several critical resources for the API were missing from the CloudFormation stack. These resources had been created manually in AWS over time, ignoring all Infrastructure as Code best practices (cries in Terraform). To restore service as quickly as possible (we’re DevOps, after all!), I recreated these resources manually once more.

Finally, several key components had the ARN of the old API hardcoded. This required a bit of “digital archaeology” to find every spot where a manual update to the new API was needed.

Ultimately, after several hours of troubleshooting, the API was back up and running, and developers could resume their deployments.

The incident began on April 4, 2024, at 3:24 PM and was resolved on April 5, 2024, at 8:46 AM.

Post-Mortem and Lessons Learned

Throughout the incident, I took intensive notes on every action taken. Having heard of the Post-Mortem principle, I knew this incident was the perfect candidate.

If you’re unfamiliar with the concept, a Post-Mortem is a document that retraces the steps of an incident, its impact, and its root cause. But most importantly—and most interestingly—it includes a Lessons Learned section. If you take this part seriously, it will be your best ally in building a more robust architecture.

In this section, you note three key points: what went well, what went wrong, and where you got lucky. And above all, be honest! Even if some points seem silly or make you look incompetent (and I’m saying this as the guy who manually deleted an API, so take it with a grain of salt), the goal isn’t to point fingers (the “blameless culture”). It’s about understanding the flaws in the system so we can fix them.

“The cost of failure is education.” — Devin Carraway (Source)

This might still feel a bit abstract, so let me share my lessons learned from this incident.

What went well

Two things went well during this incident.

First, the resolution was handled by a team member who knew the architecture inside out. This allowed for a quick understanding of what needed to be restored to get the service back online.

Second, there was excellent communication throughout. When the problem occurred, there was no attempt to hide it. Frequent updates were shared to report on progress. This is crucial—not only does it provide visibility, but communication often leads to helpful tips (like a colleague pointing you toward documentation you didn’t know existed).

What went wrong

This is the part that hurts. As I mentioned, you have to swallow your pride and highlight everything that could have been handled better.

For this incident, four things went wrong.

To start, the API infrastructure wasn’t consolidated in a single file or folder; it was scattered across multiple GitHub repos. This made it very difficult to get a bird’s-eye view of everything required for the API to function.

Next, there was a major issue with drift. This refers to the differences between your actual infrastructure and how it’s defined in your code. Ideally, no manual changes should ever occur, and everything should go through your IaC files. Had this been the case, a simple redeployment would have restored the service instantly.

Another issue was the heavy interdependence between resources. Many relied on CloudFormation stack outputs. Removing the parent stack essentially crippled the ability to manage the rest of the infrastructure.

Finally, identifying resources tied to our infrastructure was difficult. Our stack deployed resources without any associated tags, making it a scavenger hunt to find every piece of the puzzle.

Where we got lucky

This might sound positive, but it isn’t! This category covers things that went well only because of luck. Realize that at any moment, these could have been in the “What went wrong” column. Be glad this time, but don’t let your guard down!

In this case, we got lucky in three ways.

First, the incident was identified immediately (that’s the one perk of making a massive blunder yourself). But it could have been much worse! If the API had been deleted via an automated script, we had no monitoring in place to alert us.

Second, the person who deleted the API had deep knowledge of the project (yes, I’m talking about myself—I have to give myself some credit). This allowed for an immediate transition into resolution mode, but it could have been someone else who was totally lost.

Finally, this was our “Dev” API. The Production API was perfectly fine (a detail I intentionally saved for the end—you know, for the storytelling). So while the impact was minimal, the same incident could have happened in Prod with the same recovery nightmares. And that would have been much more expensive.

Preparing for the Future

Now that you’ve listed the problems encountered, as a good DevOps engineer, you must learn from them. Note everything that can be improved, but above all, set a plan! Otherwise, these are just empty words.

“A goal without a plan is just a wish.” — Antoine de Saint-Exupéry

In my case, the three key takeaways were:

Consolidation of Infrastructure as Code: Everything must be deployable in the blink of an eye. This is a project I would complete in the following months.
Improved Monitoring and Alerting: If this ever happens again, we need to be alerted immediately so we can react fast.
Clearer Documentation: Any team member should be able to handle such an incident, and that starts with reliable, understandable docs.

All these lessons were tracked as GitHub issues, and I made sure to knock them out over the following months.

Conclusion

As you’ve seen, accidents happen. The key is to make sure they benefit you and your organization. Use them as an opportunity to learn and patch vulnerabilities that were previously undetected (some companies even purposely create chaos for this very reason).

Thanks for reading to the end! I’ll leave you here—I’ve got some other infrastructures to delete!

How to migrate your State from Terraform Cloud to AWS S3?

Wed, 17 Dec 2025 07:30:00 +0200

Introduction

When I first started using Terraform (specifically for deploying my blog), I turned to Terraform Cloud (this name was later changed to HCP Terraform).

Terraform and the State

If you know a little bit about Terraform, you are likely familiar with the term State. This effectively corresponds to the current state of your infrastructure and allows Terraform to know what changes to expect if you modify your Terraform files. This State is represented by a file: terraform.tfstate.

By default, Terraform stores this file locally on your machine, but it recommends using a backend storage (read: “store this file somewhere else, for crying out loud!”).

Obviously, Terraform first suggests the HashiCorp solution: Terraform Cloud. By creating an account, you can use their backend to store your state files for free. Naturally, that is exactly what I did a few years ago.

The End of Terraform Cloud

Until this week, when I received this email from HashiCorp.

Ouch! We should have seen it coming—all good things must come to an end! The free option from back then will soon cease to exist, so I have to make a choice: upgrade my HashiCorp “plan,” or store my states elsewhere.

Looking more closely at their current offers, HashiCorp still proposes a free version, though it is limited in the number of resources that can be deployed (500 currently). It’s a fairly generous offer, but not knowing how it might evolve, I’ve decided to move my States once and for all.

Farewell, Terraform Cloud!

An alternative for storing your states

Fortunately for me, there are plenty of ways to store a Terraform state, thanks to Remote State.

To put it simply, you just need to tell Terraform where your state file will be stored. We use the backend block to do this.

There are quite a few options, but since I have an AWS account where I deploy all my resources, I naturally gravitated towards the S3 backend.

Here is an example of how to configure an S3 backend with Terraform:

terraform {
  backend "s3" {
    bucket = "mybucket"
    key    = "path/to/my/key/terraform.tfstate"
    region = "eu-west-1"
  }
}

Now that we have an alternative, it’s time to start the migration!

Migrating the State to S3

I’ll use the migration of the state I use to deploy this blog as an example.

Currently, if we look at my deployment GitHub Actions workflow, we can see that I authenticate to my Terraform Cloud account using a token. That part will no longer be necessary (at least for the credentials; the Terraform CLI will still need to be initialized).

Next, we need to create an S3 bucket to hold the future state files.

Finally, I’ll need to add a backend block to my project to tell Terraform to stop using Terraform Cloud and start using S3 when accessing the state.

Hold on, we aren’t done yet! If I leave this bucket empty, the next time I try to deploy my blog, Terraform will look for a state that… doesn’t exist! It will assume that no resources have been deployed to my AWS account yet. Consequently, Terraform will try to recreate resources that already exist.

The last step, therefore, is to retrieve the state file for my blog project and move it into my S3 bucket.

I then need to ensure that the information I provided in the backend block matches reality.

Once everything is ready, it’s time to test and deploy!

I intentionally omitted a few steps in this migration (migrating workspace variables, using GitHub Secrets for the S3 bucket name, adding AWS credentials to GitHub Actions, etc.) to keep this article concise. If you want to see more, check out the Pull Request I opened to perform this migration.

If everything went according to plan, you should see the phrase No changes. Your infrastructure matches the configuration. appear during your terraform plan!

Conclusion

It’s always a pain to perform migrations on existing infrastructure. But with a little time and some willpower, it’s the kind of task that ultimately only takes a few hours.

So, don’t wait until the last minute—dive in!

How and Why to Set Up AWS IAM Identity Center on Your AWS Account

Wed, 19 Nov 2025 07:30:00 +0200

Introduction

If you’re juggling multiple AWS accounts, or even just a single account with different users, you know that access management can quickly become a real headache. Creating individual IAM users in every account, managing their permissions, ensuring everyone is using MFA… Phew, just thinking about it is enough to give you cold sweats! But what if I told you there is an elegant, centralized, and much more secure solution to handle all of this? Ladies and Gentlemen, let me introduce you to IAM Identity Center!

Ever since I discovered and implemented IAM Identity Center (which some of you might know by its former name, AWS Single Sign-On or AWS SSO), my life as a Cloud admin has changed radically. It’s the kind of tool that, once adopted, makes you wonder how you ever lived without it.

So, what is it exactly, and why is it so great? Let’s find out together.

IAM Identity Center: What’s the Deal?

Simply put, IAM Identity Center is an AWS service that allows you to centrally manage access to all your AWS accounts and Cloud applications. Whether you have a handful of accounts or a sprawling AWS organization with dozens, or even hundreds of accounts, IAM Identity Center is there to simplify the task for you.

It offers a single point of entry (a web access portal) for your users, allowing them to access the roles and accounts they are entitled to, all with a single authentication.

The Benefits of IAM Identity Center

If I sound enthusiastic, it’s because the benefits are numerous and significant. To name the most important ones:

Single Sign-On (SSO): Your users log in once via the AWS portal (or via your existing identity provider if you have one) and then access all the accounts and roles assigned to them, without having to re-authenticate for every single account.
Centralized Management: You manage all your users, groups, and their permissions (via “Permission Sets”) from a single place, even if they need to access dozens of different AWS accounts.
Temporary Credentials: This is one of the most important points! When users access an account via IAM Identity Center, they obtain temporary credentials with a limited lifespan. Say goodbye to lingering IAM Access Keys that pose a major security risk.
MFA (Multi-Factor Authentication): You can (and should!) enforce the use of MFA directly at the IAM Identity Center login level.

If I’ve piqued your curiosity and you want to know more, check out the AWS documentation on IAM Identity Center. For those who are already sold, let’s continue!

Getting Started with IAM Identity Center

Setting up IAM Identity Center is surprisingly simple, especially if you use the built-in Identity Center directory as your identity source. Here are the key steps to get started:

Initial Configuration: Head to the AWS console and search for “IAM Identity Center”. Heads up: choose your AWS region carefully to host IAM Identity Center right from the start, as it is currently complex to change the IAM Identity Center region once configured. The initial setup is often guided and quick. You will choose your identity source (the Identity Center directory, AWS Managed Microsoft AD, or an external provider).
Creating Groups and Users: Define groups relevant to your organization (e.g., Developers, Administrators, …). Then create your users and assign them to these groups. If you are using an external IdP, this step will consist of synchronizing your existing users and groups instead.
Creating “Permission Sets”: A Permission Set is a collection of permissions (similar to an IAM policy) that you can reuse. You can start from AWS-managed policies (e.g., AdministratorAccess, ReadOnlyAccess) or create your own.
Assigning Access: This is where the magic happens. You assign a group (or a user) to one or more AWS accounts, giving them the right to use a specific Permission Set on those accounts. For example, the Developers group might have the PowerUserAccess Permission Set on the development AWS accounts.
Enforcing MFA: In the IAM Identity Center settings, configure MFA to be mandatory for all your users.
Sharing the AWS Access URL: Every IAM Identity Center configuration has a unique URL for the access portal (e.g., d-xxxxxxxxxx.awsapps.com/start). This is the URL your users will bookmark to log in.

Once logged into the portal, users will see the list of AWS accounts and roles (defined by the Permission Sets) they have access to. One click, and they are in the console of the chosen AWS account with the right permissions!

They can also get temporary credentials for the CLI. But speaking of which, how do you connect to an AWS account via CLI using IAM Identity Center? Let’s check it out!

Configuring CLI Access

The AWS documentation for configuring CLI authentication with IAM Identity Center is pretty clear, but I’ll walk you through the steps anyway. I’m assuming you already have your AWS CLI installed.

The first and only command to run will be the following:

`1`	`aws configure sso`

Next, you will need to enter some information specific to your IAM Identity Center. The most important being the “SSO start URL” (which you can find in IAM Identity Center under the name “AWS access portal URL”, a URL ending in /start). You then need to enter the region where your configuration is located; as for the “SSO registration scopes”, you can leave the default value.

SSO session name (Recommended): default
SSO start URL [None]: [https://xxxxxxxxxxxx.awsapps.com/start](https://xxxxxxxxxxxx.awsapps.com/start)
SSO region [None]: eu-west-1
SSO registration scopes [sso:account:access]:

If everything goes well, a window should open in your browser to validate your identity. Once that’s done, head back to your terminal.

In my case, I only have one AWS account, and IAM Identity Center selects it for me by default. But you might have a choice of which AWS account to select. The same goes for the role.

Finally, choose a profile name to use for your future API calls. I advise you to use default, which will allow you to avoid adding --profile my-aws-profile to the end of every command.

The only AWS account available to you is: xxxxxxxxxxxx
Using the account ID xxxxxxxxxxxx
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
Default client Region [None]: eu-west-1
CLI default output format (json if not specified) [None]: json
Profile name [AdministratorAccess-xxxxxxxxxxxx]: default
The AWS CLI is now configured to use the default profile.
Run the following command to verify your configuration:

aws sts get-caller-identity

And there you go, you’re all set!

Conclusion

As you might have gathered, I am a big fan of AWS IAM Identity Center. It brings an indispensable layer of security while considerably simplifying access management, whether for administrators or users.

It is a true pillar for a well-managed and secure AWS infrastructure. If you aren’t using it yet, I strongly encourage you to explore setting it up. It’s a minimal time investment for huge gains in security and efficiency.

Your infrastructure (and your security teams) will thank you!

On The Trail of The Lost Resource - My Investigation With Athena and CloudTrail

Wed, 09 Jul 2025 12:30:00 +0200

Introduction

Recently, I had to play detective in our AWS account.

A resource was there (a Cognito User Pool), plain as day, but nobody could remember where it came from. And of course, there were no tags to help us (if only they had read my post on why tagging your AWS resources is a must).

My mission, should I choose to accept it: find out who created it. The problem? The event was about four months old.

My first instinct was to turn to AWS CloudTrail. And there, I hit my first wall: the event history is only viewable for the last 90 days. Dead end.

Luckily, I knew our CloudTrail logs were archived in an S3 bucket. My first thought was to manually download the archives for the right month, unzip dozens of JSON files, and hit Ctrl+F while praying for a miracle. Let’s just say it was neither efficient, pleasant, nor fast.

So, I wondered if there wasn’t a simpler way to search through this pile of logs, and I finally found the perfect solution: AWS Athena.

Querying your S3 logs with Athena

For those unfamiliar, AWS Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. Basically, you can run queries on files (JSON, CSV, etc.) as if they were in a traditional database. No more downloading anything!

The idea, then, is to “map” our CloudTrail logs stored in S3 to a table in Athena. To do this, we use a single CREATE EXTERNAL TABLE query. Following the AWS documentation on the subject, I ran the following query in the Athena console.

This query creates a table and uses a very handy feature called “partition projection.” This allows Athena to infer the location of the logs based on the date, without having to manually manage partitions. This is very convenient when the structure is standardized, as is the case with AWS CloudTrail.

CREATE EXTERNAL TABLE cloudtrail_logs_pp (
    eventversion STRING,
    useridentity STRUCT<
        arn:STRING
    >,
    eventtime STRING,
    eventsource STRING,
    eventname STRING,
    awsregion STRING,
    requestparameters STRING,
    responseelements STRING,
    additionaleventdata STRING,
    requestid STRING,
    eventid STRING,
    resources ARRAY<STRUCT<
        arn:STRING,
        accountid:STRING,
        type:STRING
    >>,
    eventtype STRING,
    eventcategory STRING
)
PARTITIONED BY (
    `timestamp` string
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/CloudTrail/REGION' -- Replace with your S3 bucket path
TBLPROPERTIES (
    'projection.enabled'='true',
    'projection.timestamp.format'='yyyy/MM/dd',
    'projection.timestamp.interval'='1',
    'projection.timestamp.interval.unit'='DAYS',
    'projection.timestamp.range'='2024/01/01,NOW', -- Replace with your start date
    'projection.timestamp.type'='date',
    'storage.location.template'='s3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/CloudTrail/REGION/${timestamp}' -- Replace with your S3 bucket path
)

Warning: Don’t forget to replace the s3://... URLs with the exact path to your S3 bucket where your CloudTrail logs are stored, and to adjust the projection.timestamp.range property to the period you’re interested in.

Investigation Time: Finding the Information

Once the table is created (which only takes a few seconds), the hardest part is over! My investigation could finally begin.

I was looking for who had created a Cognito UserPoolClient on a specific date. So my SQL query looked like this:

SELECT
    eventTime,
    eventName,
    userIdentity.arn
FROM
    cloudtrail_logs_pp
WHERE
    timestamp = '2025/02/25'
    AND eventName = 'CreateUserPoolClient';

In just a few seconds, Athena scanned the logs for the requested day and returned the result.

I had the exact time, the event, and most importantly, the ARN of the user who performed the action. Mission accomplished!

The Verdict… and the Culprit

The funniest part of this story?

After setting up this solution and finding the information so easily, I discovered that the “culprit” who had created this resource four months ago… was me. I had completely forgotten!

Jokes aside, this experience confirmed one thing for me: taking a few minutes to set up Athena on your CloudTrail logs is an incredibly worthwhile investment. You’re giving yourself a long-term auditing and search capability that will save you hours of manual searching the day you really need it. Don’t be like me; don’t wait until you’re stuck to set it up!

Why Tagging your AWS Resources is a Must?

Tue, 24 Jun 2025 07:30:00 +0200

Introduction

The more services we deploy on AWS, the easier it is to get lost. At first, we know all the services we use by heart, the number of Lambdas or EC2 instances running. But little by little, it’s easy to feel overwhelmed, especially when different projects pile up. Quickly finding which resources belong to which project, or identifying those that weren’t properly cleaned up after a PoC, can quickly become a headache. And that’s without even mentioning cost visibility! Fortunately, there’s a simple yet incredibly powerful practice: tagging.

In this article, I want to show you that a good tagging strategy is crucial for your organization, the way you handle cost, and to secure your AWS account.

Ready? Tag along with me!

Why are tags useful?

Imagine labels on moving boxes. Without them, it’s impossible to know what’s inside or which room they belong to. Tags on AWS are the same! A tag is a piece of information (in key-value format) that you assign to your resources (EC2 instances, S3 buckets, RDS databases, etc.). A good tagging strategy allows you to, among other things:

Identify orphan resources: This is the classic scenario. A resource without a Project or Owner tag? There’s a good chance it has been forgotten and is consuming resources (and therefore money) for nothing. Listing untagged (or poorly tagged) resources is an essential first step in cleaning up.
Allocate costs: By tagging your resources with a project identifier, cost center, or team, you can then use AWS Cost Explorer to filter your expenses and understand precisely which projects consume the most. This is essential for internal chargebacks or simply for optimizing your budget.
Automate actions: Tags can serve as triggers for automation scripts (for example, backing up all EC2 instances with the tag Backup=Daily).
Manage access and security: The AWS IAM service can use tags to grant granular permissions.

In short, tagging is the foundation of good Cloud governance. For more details, I invite you to consult the tagging guide proposed by AWS.

Fortunately, we can associate multiple tags with the same resource. But this raises the question: how many tags are necessary?

This will depend on your company and each project, but overall, there are some tags that won’t hurt, no matter your situation:

Project: The name of the project linked to the resource. Usually, the GitHub repository name works well.
Environment: The desired environment (dev, uat, prod, …). Even if you have separate AWS accounts, this will help you identify if a resource got lost during deployment.
Owner: The owner of the resource. This could be a person, but more ideally a team (frontend, backend, security, …).

Depending on your use case, you’ll surely have other ideas for tags, but with the ones above, you’ll already have a good start! And in case of doubt, feel free to read the tagging best practices from AWS.

Let’s now look at some concrete examples of using tags in AWS.

AWS Cost Explorer: Tracking costs thanks to tags

One of the most concrete benefits of tagging is the visibility it provides over your expenses. AWS Cost Explorer is the tool of choice for this.

Once your resources are correctly tagged (for example, with the Project tag), you need to activate these tags for cost allocation in the AWS Billing and Cost Management console (Billing and Cost Management -> Cost Organization -> Cost Allocation Tags). Be aware that there might be a delay before the activated tags appear in Cost Explorer.

Once activated, you can:

Filter by tag: In Cost Explorer, you can filter your costs by the value of a specific tag.
Group by tag: You can also choose to group your expenses by tag. This will give you an overview of the cost distribution across different projects, environments, etc.
Create budgets based on tags: With AWS Budgets, you can set alert thresholds for costs associated with specific tags, helping you avoid nasty surprises.

Example of filtering by tag for my blog: the costs are minimal, am I a FinOps expert?

This ability to dissect your AWS bill by tags transforms cost management from an obscure chore into a transparent and controllable exercise. It’s a must for any organization concerned about its Cloud budget. You can now add FinOps to your LinkedIn bio!

AWS Resource Explorer: Keeping an eye on deployed resources

Now that we are convinced of the usefulness of tags, how do we list all the resources that use a specific tag? We will see this together, with one part in the AWS console, and another part in the terminal (for all the geeks reading this article!).

Visual exploration in the AWS console

If you are not yet familiar with AWS Resource Explorer, now is the time to discover it! This relatively recent service allows you to search and discover your AWS resources across all regions of your account, using a simple interface, much like a search engine.

The main advantage of Resource Explorer is its ability to give you a unified view. No more jumping from region to region. You activate indexing, and then you can search for your resources by name, ID, and of course… by tag!

It’s an excellent tool for:

Getting a quick overview.
Visually exploring resources associated with a specific tag.
Quickly identifying resources without having to code anything.

To use it, activate it in the desired regions (or all of them), let it index your resources, and then use the search bar with a syntax like tag.key:Project tag.value:Cloud*.

Example of a tag search for my blog: only three resources manage this blog!

A more technical approach with AWS CLI

For those who, like me, like to have control via the command line, or who need to automate these searches, the AWS CLI remains a key ally. More specifically, it’s the resourcegroupstaggingapi service that will interest us.

The key command is get-resources. Here is a typical example to list the ARNs of all resources having the tag Project with the value “Cloud Antoine Delia”:

1
2
3

aws resourcegroupstaggingapi get-resources \
    --tag-filters "Key=Project,Values=Cloud Antoine Delia" \
    | jq "[.ResourceTagMappingList[].ResourceARN]"

Which gives us:

1
2
3

[
  "arn:aws:s3:::antoiXXXXXXXXXXXXX"
]

Let’s break it down a bit:

aws resourcegroupstaggingapi get-resources: This is the API call; so far, so good.
--tag-filters "Key=Project,Values=Cloud Antoine Delia": This is where we specify our filter. We are looking for the tag Project with the value “Cloud Antoine Delia”. You can add multiple filters.
| jq "[.ResourceTagMappingList[].ResourceARN]": jq is a wonderful tool for manipulating JSON on the command line. Here, we use it to cleanly extract the list of ARNs of the found resources (you can do without it, but why make life difficult?).

Hey, wait a minute! In the console, you show us three resources, and now there’s only one! Where’s the catch?

Well spotted! You should know that when you make your API call, you are using a default region. However, if you have resources in various regions, you will need to specify it. Thus, if we add --region us-east-1 just before the jq pipe, we indeed get our two missing resources.

[
  "arn:aws:acm:us-east-1:6XXXXXXXXXXX0:certificate/f4ca3b13-XXXXXXXXXXXXX",
  "arn:aws:cloudfront::6XXXXXXXXXXX0:distribution/ERSXXXXXXXXXX"
]

Besides this detail that you must not forget, this command is extremely powerful because you can integrate it into scripts to:

Generate regular reports on resources by project.
Automatically detect resources that do not comply with your tagging policy.
Combine with other AWS CLI commands to perform actions on the listed resources.

For example, you could list all your resources of a certain type (e.g., all your EC2 instances) and check which ones are missing essential tags.

And if you’re wondering if AWS doesn’t already offer a service for this… It does! But we will talk about that in a future article (for the curious, I’m talking about AWS Config).

AWS IAM: Securing the use of your resources

Your resources are deployed in AWS, and now you want to give a team permission to manage all of that. But here’s the thing, in your AWS account, you also have critical resources that must absolutely not be compromised.

AWS IAM is here for you! Using a simple policy, you can specify that only resources with a certain tag can be modified by a user or group.

Let’s take the following example: you would like to allow a team to start or stop certain EC2 instances, but prevent them from accidentally stopping a critical EC2 instance! You just need to add the following policy to your users:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowStartStopEC2IfProjectCloud",
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Project": "Cloud Antoine Delia"
        }
      }
    },
    {
      "Sid": "AllowDescribeToSeeInstances",
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    }
  ]
}

Thus, your users will be autonomous in using their resources, without having the possibility to impact other resources.

Conclusion

You’ve understood it: a rigorous tagging strategy isn’t an option; it’s a necessity for operating serenely on AWS. Whether on the organizational, financial, or security management level, AWS gives you the means to take full advantage of your tags.

So, a little piece of advice: if you haven’t already, define a clear tagging policy in your organization, apply it, and use these tools to regularly check that everything is in order. You’ll thank me later!

AWS Lambda: INIT Phase To Be Billed Soon - Should We Be Alarmed?

Tue, 13 May 2025 07:30:00 +0200

Introduction

Ah, Lambdas. Probably my favorite AWS service! Almost instantly, they allow you to create a small script or even a fully hosted API backend in the Cloud, all at a low cost thanks to the “pay-as-you-go” model. But is it too good to be true?

A few days ago, AWS announced a change in how the Lambda initialization phase is billed. Starting August 1, 2025, this phase will systematically be included in the billed duration calculation, for all Lambdas.

If you weren’t aware, until now, if you used Lambda functions packaged as ZIP archives with AWS-managed runtimes (like Python, Node.js, etc.), the duration of this INIT phase wasn’t billed. It was a small “gift” from AWS (quite nice of them). But all good things must come to an end, and we will now have to pay for this initialization time for our Lambdas.

So, will this make using Lambdas too expensive? And how can we minimize this initialization part as much as possible? Today, let’s explore these questions!

Understanding the Lambda Lifecycle

Before diving into billing, it’s a good opportunity to recall the Lambda lifecycle.

For simplicity, I won’t go into the details of Lambda Extensions and Lambda SnapStart.

It consists of three main phases:

INIT: This is the “cold start” phase. When a new instance of your function needs to be created to handle a request, Lambda prepares the environment. This phase lasts a maximum of 10 seconds.
INVOKE: This is where your code (the function handler) is executed to process the request.
SHUTDOWN: When the execution environment hasn’t been used for a certain period, the Lambda shuts down to free up resources. If a new request arrives, the Lambda will have to go through the INIT phase again.

During the INIT phase, your Lambda performs several actions:

Retrieves your code (from S3 for a ZIP, or ECR for a Docker image).
Configures the environment with the allocated memory, chosen runtime, etc.
Starts the runtime (Runtime INIT).
Executes the static code of your function (everything outside the handler, like initializing global variables or boto3) (Function INIT).

The key takeaway here is that the INIT phase only occurs during a cold start. If a request arrives while an execution environment is already “warm” (ready and reused), this phase is skipped, and execution goes directly to the INVOKE phase. This is called a “warm start,” which is much faster.

AWS doesn’t disclose its logic for transitioning a Lambda from ‘warm’ to ‘cold’.

The Billing Change in Detail

Currently, Lambda billing is based on two factors:

The number of requests.
The execution duration of your code, rounded up to the nearest millisecond (the cost of this duration depends on the memory allocated to the function).

Until August 1, 2025, for functions packaged as ZIPs with managed runtimes, the INIT phase duration was not counted in the “Billed Duration”. This could be seen in CloudWatch logs:

Before August 1, 2025
Note that Billed Duration is the rounded-up value
of Duration without considering Init Duration
REPORT RequestId: xxxxx   Duration: 250.06 ms   Billed Duration: 251 ms   Init Duration: 100.77 ms

After August 1, 2025
Billed Duration is now the rounded-up value
of Duration + Init Duration
REPORT RequestId: xxxxx   Duration: 250.06 ms   Billed Duration: 351 ms   Init Duration: 100.77 ms

As you can see, AWS will now take your Init Duration into account in addition to the Duration when calculating the Billed Duration.

What Impact on the Bill?

So, this all sounds interesting, but will this change break the bank?

Let’s take an example to see more clearly. Imagine a Python Lambda configured with 1024 MB of memory, deployed in the eu-west-1 (Ireland) region.

Assume the following:

The Lambda receives 10 million invocations per month.
The cold start rate is 1% (average provided by AWS), meaning 100,000 cold starts per month.
The average invocation duration (Duration) is 3 seconds.
The average initialization duration (Init Duration) is 1 second.
Cost per request: $0.20 per million requests.
Duration cost (x86): $0.0000166667 per GB-second.

Details of the calculation

Cost Calculation BEFORE August 1, 2025:

Request Cost: 10 million req * ($0.20 / 1 million req) = $2.00
Duration Cost:
- Warm starts (9.9 million): Billed duration = 3 seconds.
- Cold starts (100,000): Billed duration = 3 seconds (INIT not billed).
- Total billed duration (seconds): (9,900,000 * 3 s) + (100,000 * 3 s) = 29,700,000 seconds + 300,000 seconds = 30,000,000 seconds.
- Total GB-seconds: 30,000,000 seconds * (1024 MB / 1024 MB) = 30,000,000 GB-s.
- Duration Cost: 30,000,000 GB-s * $0.0000166667/GB-s = $500.00
Total Monthly Cost (Before): $2.00 + $500.00 = $502.00

Cost Calculation AFTER August 1, 2025:

Request Cost: $2.00 (unchanged)
Duration Cost:
- Warm starts (9.9 million): Billed duration = 3 seconds.
- Cold starts (100,000): Billed duration = 3 seconds (Duration) + 1 second (Init Duration) = 4 seconds.
- Total billed duration (seconds): (9,900,000 * 3 s) + (100,000 * 4 s) = 29,700,000 seconds + 400,000 seconds = 30,100,000 seconds.
- Total GB-seconds: 30,100,000 seconds * (1024 MB / 1024 MB) = 30,100,000 GB-s.
- Duration Cost: 30,100,000 GB-s * $0.0000166667/GB-s = $501.67
Total Monthly Cost (After): $2.00 + $501.67 = $503.67

In this specific scenario, the increase is only $1.67 per month. This is indeed minimal and confirms AWS’s communication. But be careful, as the actual impact will strongly depend on:

The memory allocated to your Lambdas (more memory = higher cost per ms).
The actual duration of your INIT phase.
Your cold start rate (can be higher if your traffic is irregular).

Therefore, it’s wise to check with your own numbers!

Below is a small script that will allow you to quickly test this on your side.

Python Script

def calculate_lambda_costs(
    total_invocations_per_month: int,
    cold_start_rate: float,
    average_invocation_duration_sec: float,
    average_init_duration_sec: float,
    allocated_memory_mb: float, # Changed GB to MB for consistency with common AWS console values
    cost_per_million_requests: float = 0.20,
    cost_per_gb_second: float = 0.0000166667,
):
    num_cold_starts = total_invocations_per_month * cold_start_rate
    num_warm_starts = total_invocations_per_month - num_cold_starts

    request_cost = (total_invocations_per_month / 1_000_000) * cost_per_million_requests

    allocated_memory_gb = allocated_memory_mb / 1024 # Convert MB to GB for calculation

    # Duration cost BEFORE
    billed_duration_warm_starts_sec_before = num_warm_starts * average_invocation_duration_sec
    # For cold starts, only invocation duration was billed
    billed_duration_cold_starts_sec_before = num_cold_starts * average_invocation_duration_sec

    total_billed_duration_sec_before = billed_duration_warm_starts_sec_before + billed_duration_cold_starts_sec_before
    total_gb_seconds_before = total_billed_duration_sec_before * allocated_memory_gb
    duration_cost_before = total_gb_seconds_before * cost_per_gb_second

    total_monthly_cost_before = request_cost + duration_cost_before

    # Duration cost AFTER
    billed_duration_warm_starts_sec_after = num_warm_starts * average_invocation_duration_sec
     # For cold starts, invocation + init duration will be billed
    billed_duration_cold_starts_sec_after = num_cold_starts * (average_invocation_duration_sec + average_init_duration_sec)

    total_billed_duration_sec_after = billed_duration_warm_starts_sec_after + billed_duration_cold_starts_sec_after
    total_gb_seconds_after = total_billed_duration_sec_after * allocated_memory_gb
    duration_cost_after = total_gb_seconds_after * cost_per_gb_second

    total_monthly_cost_after = request_cost + duration_cost_after

    return total_monthly_cost_before, total_monthly_cost_after


def display_results(cost_before, cost_after):
    cost_difference = cost_after - cost_before
    print(f"\nEstimated monthly cost BEFORE change: ${cost_before:.2f}")
    print(f"Estimated monthly cost AFTER change: ${cost_after:.2f}")
    print(f"Estimated monthly cost increase: ${cost_difference:.2f}")
    if cost_difference > 0 and cost_before > 0:
        percentage_increase = (cost_difference / cost_before) * 100
        print(f"Percentage increase: {percentage_increase:.2f}%")
    elif cost_difference > 0:
        print("Percentage increase: Infinite (original cost was zero)")


if __name__ == "__main__":
    print("AWS Lambda Cost Calculator (Before/After Init Duration Pricing Change)")
    print("Please enter the values for your scenario or press Enter to use default values.\n")
    default_invocations = 10_000_000
    default_cold_start_rate = 0.01  # 1%
    default_invocation_duration = 3.0  # seconds
    default_init_duration = 1.0  # seconds
    default_memory_mb = 1024.0 # 1024 MB

    try:
        invocations_str = input(f"Total invocations per month (default: {default_invocations:,}): ")
        total_invocations_input = int(invocations_str) if invocations_str else default_invocations

        cold_start_rate_str = input(f"Cold start rate (e.g., 0.01 for 1%, default: {default_cold_start_rate}): ")
        cold_start_rate_input = float(cold_start_rate_str) if cold_start_rate_str else default_cold_start_rate

        invocation_duration_str = input(f"Average invocation duration in seconds (default: {default_invocation_duration}): ")
        invocation_duration_input = float(invocation_duration_str) if invocation_duration_str else default_invocation_duration

        init_duration_str = input(f"Average initialization duration in seconds (default: {default_init_duration}): ")
        init_duration_input = float(init_duration_str) if init_duration_str else default_init_duration

        memory_mb_str = input(f"Memory allocated to Lambda in MB (e.g., 512, default: {default_memory_mb}): ")
        memory_mb_input = float(memory_mb_str) if memory_mb_str else default_memory_mb

        cost_before, cost_after = calculate_lambda_costs(
            total_invocations_per_month=total_invocations_input,
            cold_start_rate=cold_start_rate_input,
            average_invocation_duration_sec=invocation_duration_input,
            average_init_duration_sec=init_duration_input,
            allocated_memory_mb=memory_mb_input,
        )

        display_results(cost_before, cost_after)

    except ValueError:
        print("\nError: Please enter valid numbers.")
    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")

How to Monitor Your INIT Phase and Estimate the Impact?

A script is nice, but if you have a bunch of Lambdas to check, you might find the task exhausting.

Fortunately, AWS provides some tools to check this efficiently. As seen earlier, each Lambda will generate a REPORT log containing its Init Duration. We can easily aggregate this information using CloudWatch Logs Insights. And the icing on the cake, AWS even provides us with the right query.

filter @type = "REPORT" and @billedDuration < ceil(@duration + @initDuration)
| stats sum((@memorySize/1024/1000) * (@billedDuration/1000)) as BilledGBs,
        sum((@memorySize/1024/1000) * ((ceil(@duration + @initDuration) - @billedDuration)/1000)) as EstimatedInitGBs,
        (EstimatedInitGBs / (EstimatedInitGBs + BilledGBs)) * 100 as EstimatedInitCostPercent

Keep in mind that CloudWatch Logs Insights is billed at $0.005 per GB of data scanned (source). Adjust the query time range to limit costs.

Just run this query on your Lambda log groups (using the /aws/lambda prefix or specific function prefixes), and you’ll get a result like this:

BilledGBs	UnbilledInitGBs	Ratio
512.8007	86.6699	0.1446

This query gives you three key pieces of information:

BilledGBs: The total GB-seconds currently billed.
EstimatedInitGBs: The total GB-seconds consumed during the INIT phase that were not previously billed (this is an estimation based on the difference between current billed duration and potential future billed duration including INIT).
EstimatedInitCostPercent: The percentage that these previously unbilled INIT GB-seconds represent compared to the total consumed GB-seconds (current billed + estimated init). This approximates the potential percentage increase in your duration costs due to the change.

In our example, the cost share of the INIT phase represents approximately 14% of the total execution cost (current + potential init cost)! Depending on your current bill, this could be significant.

Understanding and Optimizing Your Lambda

Okay, so you know you might have to pay more. But aren’t there levers we can pull to reduce this increase in your bill?

Here are some tips for making the best use of this INIT phase and thus reducing your future costs.

Strategically Use the INIT Phase

Your first instinct might be to remove all the INIT code (code outside the handler) and put it in your handler. That way, no more INIT billing! But not only would you just be shifting the problem (because executing that code will still be billed), it would be even worse, because now even your ‘warm start’ Lambda executions would have to process this code!

Yes, remember that the code in the INIT phase is executed only during cold starts. Therefore, it’s the ideal place for expensive initialization operations that can be reused by subsequent (“warm start”) invocations. This is even recommended by AWS. Thus, the INIT phase is perfect for:

Importing heavy libraries or dependencies.
Establishing connections to other AWS services (S3, DynamoDB, etc.) via SDKs (boto3 for Python).
Creating database connection pools.
Fetching parameters or secrets from Systems Manager Parameter Store or Secrets Manager.

Maximize this initialization phase to reduce the execution time of Lambdas running in ‘warm start’.

Optimize Package Size

This is often the simplest lever, because sometimes we import all sorts of things into our Lambda. So, make sure to include only strictly necessary dependencies. Also, be sure to exclude anything related to your development environment (I can’t tell you how many times I’ve found a node_modules or tests folder in Lambdas…). Finally, don’t hesitate to consult AWS articles, as some directly tackle the subject of cold starts.

Lambda SnapStart

Available for Java, .NET, and Python runtimes, SnapStart is a very interesting feature for combating cold starts. When you enable it, Lambda takes a “snapshot” of the initialized execution environment after the first INIT phase. For subsequent cold starts, Lambda restores this snapshot instead of redoing the entire INIT phase, saving you significant time.

I imagine reading about this feature, you’re thinking “Wow, this is great, I’m going to enable this feature on all my Lambdas!”. Except there are a few subtleties to be aware of.

Firstly, SnapStart is currently only compatible with specific versions of certain languages (Python 3.12 minimum, NodeJS not supported…). The same goes for regions; only a limited number support this feature at the time of writing. Finally, this feature is not free and incurs its own costs. Make sure to check the potential cost if you want to enable it on your Lambdas.

Provisioned Concurrency

If your application has predictable traffic or if cold start latency is unacceptable, you can use “Provisioned Concurrency”. You ask Lambda to keep a certain number of execution environments pre-initialized (warm) at all times.

Advantage: Requests hitting these provisioned instances never experience a cold start. The INIT phase is done upfront, before the first request.

Disadvantage: You pay for the time these environments are provisioned, whether they receive requests or not. The INIT phase is also billed during this pre-initialization.

Here too, it’s up to you to see if it’s worth enabling this on some of your key Lambdas!

Conclusion

The upcoming change in billing for Lambda’s INIT phase on August 1, 2025, is primarily a standardization. For the majority of Lambdas (specifically those using managed runtimes and ZIP packaging that previously benefited from the free init), this means the duration of this phase will now be added to the billed duration during cold starts.

Even if the financial impact will likely be small for you, it’s an excellent opportunity to:

Understand your Lambda’s lifecycle and what happens during initialization.
Measure the INIT phase duration of your critical functions using CloudWatch tools.
Optimize this phase if necessary, by reducing package sizes, using SnapStart, or considering Provisioned Concurrency for specific use cases.

So, don’t wait until August! Familiarize yourself with CloudWatch Logs Insights now. This will allow you to identify which Lambdas to optimize first and avoid any nasty surprises on your AWS bill.

How I Automated My Blog To The Cloud

Tue, 04 Mar 2025 07:30:00 +0100

How it started and why it ended

When I started blogging, I was still in engineering school. And the world of websites and blogs was still new to me. So when I wanted to create a personal blog, I turned to what I thought was the best solution at the time: WordPress.

Like it or not, WordPress is quite easy to use for beginners, even more for me who was still discovering how to host a website at the time.

So, with a brand new OVH account, I was able to quickly setup my personal blog in a few clicks!

And when the time came where I wanted to create a blog dedicated to professional topics, I naturally came back to my ol’ friend WordPress.

But after some months, the glory of WordPress started to fade into a darker view. Sure, it was convenient to add a new post by going through WordPress’ UI, but maintaining the website was always boresome. Moreover, a WordPress installation requires using a database, which incurred a higher bill at the end of each month.

All of this for a blog that I knew did not receive many visits each day.

Surely there was a better way to do this.

The start of a new plan

I will tell you here the path that led me to the rebirth of my blog on which you are today. If you are only interested in the technical details, you can move on to the next section. Otherwise, happy reading!

Discovering Hugo

Before searching for a new tool, I tried to think of what I needed.

My blog was essentially a simple webpage, a static website with no fancy JavaScript or stuff like that. Plus, there was no backend to handle API calls for user authentication for example, so I could simply not use any database at all.

So what I needed was essentially a way to focus on the content of the blog, and make sure I could attach to it a nice theme, and that was it!

But, as I’m an engineer, so in essence, a real bad designer, I wanted to avoid having to deal with CSS as much as possible. Again, I wanted to focus on the content, rather than the looks. So I searched and searched for the perfect tool. And I didn’t had to look too long to find it: Hugo.

Upon arriving on its website, I was greeted by an intriguing message message: The world’s fastest framework for building websites. Surely, I was curious, and dug into the documentation. And oh boy, was I not disappointed.

This was exactly what I wanted. A framework where I had to write in markdown format, that would be converted to HTML for me. Plus, it came with a bunch of themes, so I didn’t even had to bother about the design. I could just pick one that I liked, and move on.

So, without losing any more time, I started to convert my WordPress blog to a Hugo site. I came across a handy script called wordpress-to-hugo-exporter that converted my whole Wordpress database into compatible markdown syntax, and I was ready to go!

Now I needed to host this website somewhere.

A new challenger: AWS

In parallel, I had an opportunity working for a company that focused on the Cloud, more specifically AWS.

At the time, Cloud was brand new to me. But I was hearing more and more about it, and I wanted to check what was all the fuss around it. Would this be a true life-changer, as people would call it, or would it be yet again another buzz word, and become a dying trend?

Long story short, AWS was (and still is) amazing! Not only was it an amazing discovery, I could simply not see a world without Cloud anymore.

The more I learned about all these amazing AWS services, the more I thought: isn’t there a way for me to leverage the Cloud to host my blog?

With that in mind, I started looking for options that AWS provided. And because of my recent discovery of Hugo, my website did not needed PHP anymore, nor any database to work. It was now a simple, static website, which could perfectly fit in an S3 bucket.

Not only does AWS has a documentation about hosting a static website on an S3 bucket, but the pricing for it was just a few cents.

I had found an easy and cost-effective way to host my website.

The last point that was bothering me, is that to setup all of this, I had to create an S3 bucket, configure it, add a CDN with AWS CloudFront, take into account certificates with AWS ACM, and finally create a DNS record in AWS Route53. So many actions that, if done manually, could make it super difficult to reproduce this setup if I had to do it again.

I needed a robust way to configure this infrastructure.

Terraform to the rescue

When you start learning Cloud, you will usually hear about Infrastructure as Code.

The idea is simple: describe your Cloud infrastructure with… code! (yes, as the name stands by).

The benefits of using an IaC tool, is that you will be able to keep track of the changes done on your infrastructure (like you would do with code in GitHub).

Now, let’s say that someone wants to update the configuration of an AWS service. This person goes to the AWS console and start making their changes. But after some trial and error, they decide to give up for now, and to go back to the initial state. And here is the problem: they don’t know exactly how it was before! And unless they took note of the initial state before making their changes, they will, again, have to do a bunch of trial and errors before going back to the beginning.

All of that could have been prevented if the infrastructure was configured through an IaC tool to begin with. In such a case, a simple automated deploy could have done the trick!

There are a bunch of IaC tools out there. But how can I not talk about the most popular one, the one that made IaC a standard in the Cloud ecosystem: Terraform.

Terraform, as you could have guessed, is an Infrastructure as Code tool with a declarative approach, meaning that you have to declare the state of the infrastructure you wish to deploy. To do that, you need ton use a specific language: hcl (which stands for HashiCorp Configuration Language). Here is an example to create an S3 bucket with Terraform.

resource "aws_s3_bucket" "example" {
  bucket = "my-tf-test-bucket"

  tags = {
    Name        = "My bucket"
    Environment = "Dev"
  }
}

Let’s go back to my blog! Terraform seems to be the perfect tool to configure the infrastructure that will host my blog. All I had to do was to go through an architecture step to know which AWS services I needed, and how to interconnect them, before translating that to Terraform.

And here is the architecture diagram I came up with.

All of this was perfect, but I realized one thing. Every time I needed to add a post, or update the infrastructure, I had to clone my repository, make the appropriate changes, and do a lot of manual actions to update my blog.

Let’s say that I was not too pleased with that, and I intended to fix that!

Forget copy and paste with GitHub Actions

Like I said before, my website was ready. The last step I needed was about the deployment.

For now, when I had to create a new post, I had to manually build my Hugo website, and move the generated files to my S3 bucket. What a waste of time and energy!

So I explored the topic of CI/CD pipelines to make my life easy (if you’re interested in this topic, I suggest reading my post about setting up a CI/CD strategy). While Jenkins tried to get my attention, I decided to go with the obvious choice: GitHub Actions.

GitHub Actions are simple CI/CD pipelines that you can define directly in your GitHub repository. The main benefit, is that you do not need any additional account, nor to install anything: all is included! In addition, GitHub Actions use an easy-to-read YAML syntax (in opposition to Jenkins and its horrifying groovy!).

So I created a pipeline that was able to configure Terraform in case I made any modifications, and could also build and deploy my website automatically in my S3 bucket when I updated or added a post.

All the steps were looking good, mission accomplished!

Technically: What I Have Today

My blog is now fully automated and deploys to AWS!

If you want to dive into it, everything is publicly available on my GitHub repository: https://github.com/antoinedelia/cloud-optimist

Now, let me show you more details about it.

Project Structure

The project structure is as follow:

cloud-optimist/
├── .github/
│   └── workflows/
│       └── main.yml  # GitHub Actions
├── cloud/
│   └── ...  # My Hugo blog
├── terraform/
│   └── ...  # The Terraform configuration
└── README.md

As you can see, I split my project into three key directories:

.github : contains the main.yml file which defines the GitHub Actions CI/CD steps
cloud : contains my Hugo blog
terraform : contains all the Terraform infrastructure

The GitHub Actions

My GitHub Actions is rather simple, and can be divided into multiple steps.

The Initialization Step

Let’s have a look at the top of the file.

name: 'Build and Deploy'

on:
  push:
    branches:
    - master
  pull_request:

jobs:
  build-and-deploy:
    name: 'Build & Deploy'
    runs-on: ubuntu-latest
    environment: production

    defaults:
      run:
        shell: bash
        working-directory: cloud

Here, I ask GitHub to run the pipeline on the master branch, or if I have a Pull Request. Let me reassure you, I do not deploy anything to production with a Pull Request. You’ll see below that this only allows me to perform some “dry-run” checks and ensure no issues are found before I can safely deploy to production via the master branch.

I also ask GitHub to use bash and to use the cloud directory as the default working directory.

Let’s see what’s next:

    steps:
    - uses: actions/checkout@v3
    - uses: dorny/paths-filter@v2
      id: filter
      with:
        filters: |
          web:
            - 'cloud/**'
          terraform:
            - 'terraform/**'

This step is crucial if you wish to fasten your CI/CD and ultimately save cost.

I’m using the dorny/paths-filter actions which allows me to detect which files were modified in the last commit. In my case, I look for the cloud and terraform directories. This way, if no changes were detected on the Terraform side, no need to trigger the Terraform steps in my pipeline. In the same way, if the directory cloud is untouched, no need to try to build and deploy the contents of my blog. This will save you few cents related to the data transfer pricing to your AWS S3 bucket (no need to thanks me!).

The next steps speak for themselves.

    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v2
      
    - name: Update Git Submodules
      working-directory: ./
      run: git submodule update --init --recursive

I simply get the content of my repository and update the submodules. This last step was useful back when I was using submodules to handle my Hugo themes. I now use the Hugo Modules, which could let me remove this step altogether.

The Terraform Step

Let’s continue with Terraform:

    # Install the latest version of Terraform CLI and configure the Terraform CLI configuration file with a Terraform Cloud user API token
    - name: Setup Terraform
      if: steps.filter.outputs.terraform == 'true'
      uses: hashicorp/setup-terraform@v1
      with:
        cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

    # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
    - name: Terraform Init
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform init

    # Checks that all Terraform configuration files adhere to a canonical format
    - name: Terraform Format
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform fmt -check

    # Generates an execution plan for Terraform
    - name: Terraform Plan
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform plan

      # On push to master, build or change infrastructure according to Terraform configuration files
    - name: Terraform Apply
      working-directory: ./terraform
      if: steps.filter.outputs.terraform == 'true' && github.ref == 'refs/heads/master' && github.event_name == 'push'
      run: terraform apply -auto-approve

Now that’s a long step! Let’s break it down.

The first thing that should’ve catch your attention is this line:

`1`	`if: steps.filter.outputs.terraform == 'true'`

Do you remember? This is what allows us to tell if files were updated in the terraform directory. Here, I check if I need to run these steps or not.

Then, I am running some ordinary Terraform commands: terraform init to initialize my project, terraform fmt -check to ensure my code is fancy, and finally terraform plan to look at the changes planned.

Last but not least, terraform apply -auto-approve, which will deploy the changes to my AWS infrastructure. But if you look closely, you will notice I added two extra checks:

`1`	`github.ref == 'refs/heads/master' && github.event_name == 'push'`

This ensures that this step will only be ran if the event is a push on the master branch. This way, no need to worry about an unwanted deployment when I simply play with my infrastructure on another branch or in a Pull Request. Phew!

The Hugo and AWS Steps

Now that my infrastructure is ready, it is time to publish my blog!

    - name: Build
      uses: actions/setup-node@v2
      if: steps.filter.outputs.web == 'true'
    - run: sudo wget https://github.com/gohugoio/hugo/releases/download/v0.142.0/hugo_extended_0.142.0_linux-amd64.deb -O hugo.deb
    - run: sudo dpkg --install ./hugo.deb
    - run: hugo

    - name: Deploy to AWS
      uses: jakejarvis/s3-sync-action@master
      if: steps.filter.outputs.web == 'true' && github.ref == 'refs/heads/master' && github.event_name == 'push'
      with:
        args: --acl public-read --follow-symlinks --delete
      env:
        AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_REGION: 'eu-west-1'
        SOURCE_DIR: 'cloud/public/'

As for Terraform, we check if any modifications were done on the blog.

`1`	`if: steps.filter.outputs.web == 'true'`

Then, I’m retrieving an extended version of Hugo straight from the GitHub release (here the v0.142.0). I’m installing it and build my blog using the hugo command.

This step could be simplified using the peaceiris/actions-hugo actions

Finally, I’m using the jakejarvis/s3-sync-action actions (I realized while writing this post that this action has been archived a couple of days before, ouch! – I’ll add an edit later to mention an alternative) to move my blog’s files into my S3 bucket.

Of course, you’ll have to store some credentials in your GitHub repository so that your GitHub Actions can be authorized to use AWS, but nothing too crazy.

We are now done with the GitHub Actions part!

The Terraform part

Regarding Terraform, I’ll keep it short, as this is not too complex.

There is one personal choice here, is that I prefer to split my .tf files based on the AWS services used, rather than putting them all in a single main.tf file, which could make it hard to read. In the end, I have an s3.tf file for all resources linked to the S3 service, a cloudfront.tf file for everything related to CloudFront, and so on.

One important information, is that it is necessary to define at least the us-east-1 region, as you must deploy your ACM certificates there. For the rest, I’m deploying all my resources in the eu-west-1 region. To make this split, I’m using the Terraform alias. Here’s an example:

# The default configuration: all resources starting with `aws_` will use this provider
provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us_east_1"
}

resource "aws_s3_bucket" "site" {
  bucket        = var.bucket_name
  force_destroy = true
}

resource "aws_acm_certificate" "cert" {
  provider                  = aws.us_east_1  # Will be deployed in us-east-1
  domain_name               = var.domain_name
  subject_alternative_names = ["*.${var.domain_name}"]
  validation_method         = "DNS"
}

The Hugo part

My Hugo blog is based on the Stack theme. There is also a GitHub template available which allows you to quickly setup your own blog based on this theme.

Not a lot to say about this part, except that I encourage you to have a look at the Hugo Modules, which will save you from having to deal with submodules (and will make it easier when you need to update your theme).

Potential Improvements

While I’m extremely satisfied with the end results, I noted a few points that could be improved in the future.

First off, I saw that Hugo has a deploy command that could, well, deploy your blog directly to an S3 bucket. Perfect! So I need to learn a bit more about Hugo Deploy.

Finally, even though using AWS as my hosting platform is not expensive (less than 2$ a month), it is not free. But it would be completely possible to use GitHub Pages to make this blog accessible, while taking the advantage of GitHub as the hosting platform (on this point, I’m a bit reluctant, as then I would not be able to show-off with my beautifully made AWS diagrams…).

Conclusion

Here we are, you know everything!

From my beginnings with WordPress, which, while being useful, showed some weaknesses on the backup, maintainability and deployment part.

Of my research to find the perfect technical suite: AWS, Terraform, Hugo and GitHub Actions, all of that to enable a speedy deployment at almost no cost!

When I look back to my previous setup and compare to what I have today, I am so happy to have done the switch!

Now, it’s up to you! You have all the information to create your own blog and deploy all of this with a breeze!

Happy blogging !