DevOps on The Cloud Optimist

The Day I Accidentally Deleted An Api and What It Taught Me About Devops

Tue, 17 Feb 2026 07:30:00 +0100

April 4, 2024, had started off so well. A hot coffee in hand, VS Code humming with activity, and smooth electronic music in my ears—all the ingredients for a perfect day that surely couldn’t foreshadow the coming disaster.

Today, I’d like to tell you how a seemingly quiet day became a defining moment in my DevOps career!

Incident

The Context

At the time, I was working in a team tasked with providing a global API Gateway on AWS, featuring several endpoints managed by different teams. Specifically, setting up this API Gateway was the initial phase of the project. It included a DNS record pointing to the API, the API Gateway itself, and a Cognito Authorizer configured with several clients.

Once this API was ready, external teams could deploy their own endpoints onto it. To facilitate this, a CI/CD pipeline was in place, using CloudFormation to attach the endpoints directly to the existing API.

I’m keeping it as simple as possible here, but if you’re interested in the technical nitty-gritty, feel free to reach out!

On my end, besides being responsible for the global API infrastructure, I occasionally worked on or bootstrapped certain services.

The Beginning of the Drama

If you’ve ever worked with CloudFormation, you might be familiar with the dreaded UPDATE_ROLLBACK_FAILED status. This happens when you try to update a CloudFormation stack, but it encounters an error. It then tries to roll back, but if that rollback fails too, your stack gets stuck in UPDATE_ROLLBACK_FAILED.

This state is particularly annoying because you cannot trigger any new deployments until you’ve resolved the underlying issue.

That is exactly what happened on that fateful April 4, 2024. One of our services got stuck in this state, and I started investigating the “why” and the “how.”

After a few minutes, I noticed the problem stemmed from the API resource itself. Without overthinking it, and in an effort to unblock the situation quickly, I headed straight to the AWS Console, specifically to the API Gateway service. I found the resource in question and rushed to delete it.

At that exact moment, something very strange happened. An unexpected behavior that made my blood run cold. Instead of staying on the same page, AWS redirected me to the main API Gateway dashboard—the page that lists your available APIs. It now displayed a count of 0.

I realized then that I hadn’t deleted a specific resource within the API, but the entire API Gateway itself.

Anatomy of a Failure

I never thought I’d make such a rookie mistake. And I imagine as you read this, you’re thinking the same thing.

Because to get it that wrong, you really have to try!

Let me reconstruct the crime scene for you. Here is what I saw when I decided to delete a resource from the API Gateway.

Any resemblance to an actual situation is entirely coincidental.

In my shoes, which button would you have clicked? Easy! The Delete button right next to the resource!

As for me (and I still don’t know what possessed me), I chose to click the API actions button. After all, I did want to perform an “action”!

And what happens when you click that button?

Yikes! Right away, you can see it’s not what we want at all. But do you think that stopped me? Not a chance! I came here to delete something, and when I saw the word Delete, I stopped thinking. I should have kept reading, because then I would have seen the word API right next to it.

Fortunately, our friends at AWS thought of everything! When you click Delete API, they still ask if you are absolutely sure. A nice dialog box pops up, showing the name of the API and asking you to type the word confirm to validate the operation.

That’s all well and good, but the AWS engineers underestimated my impatience. In that moment, the confirmation request wasn’t a warning; it was an obstacle to my goal. In a heartbeat, I typed confirm and hit enter.

So, here is the last thing I saw before finally realizing the gravity of my mistake:

A vision of pure horror.

I wanted to walk you through this process to make one thing clear: you can put all the safeguards in the world in place, but you can’t protect a system against an impatient individual, because they won’t stop to read the warnings.

So, the next time you have to perform an “innocuous” action, take the time to read and ensure you are actually on the right path.

That being said, let’s move on to the crucial stage: resolving the incident!

Resolution

The Click

Back to the moment of impact. After realizing what had happened, I had a sudden jolt—a moment of clarity (without the aid of any white powder, I promise). I knew what I had to do: resolve the incident, but more importantly, document every single action I took.

A few weeks prior, I had looked into the 2017 GitLab incident, which took the service down for hours and resulted in data loss for some users. That’s how I discovered the concept of a Post-Mortem and why they are so vital in these situations. But I’ll save that for the next section.

In the meantime, if you’re interested in these topics, I highly recommend Kevin Fang’s YouTube channel where he, in his own words, “reads postmortems and makes low-quality videos about them.”

Rollback, Impact, and Communication

My first thought was that there might be a way to trigger a rollback directly within AWS to minimize the impact on users. If I had actually read the warning message earlier, I would have known that was impossible (but if I had read it, I wouldn’t have been in this mess anyway).

With no “undo” button in sight, I accepted that I was facing a major incident with a global impact. I first set out to map out the full extent of the damage. In my case, not only was the API down (thanks, Captain Obvious), but no new deployments could be made until the API was back online.

Finally, I made sure to notify the internal team. This kept them in the loop and let them know I was actively working on a fix.

Analysis and Troubleshooting

It was time to roll up my sleeves and find a way to redeploy the API Gateway.

First, I went to the CloudFormation service, remembering that this API was originally deployed through it. I tried updating the stack, hoping it would bring my dear API back like magic.

Obviously, it wasn’t going to be that simple. Updating the stack was impossible because the manual deletion had put the stack into a “hybrid” state that it couldn’t reconcile.

Since updating was out of the question, the logical next step was to delete the stack and redeploy it from scratch. That’s when the real trouble started. This CloudFormation stack produced several Outputs. Two of these were required by “child” stacks that had deployed their endpoints on the API. Consequently, CloudFormation blocked me from deleting the stack because it would break all the others.

After brainstorming alternatives, this heavy interdependence led me to a tough decision: I had to delete all the “child” stacks in CloudFormation—a total of 81 stacks.

To make matters worse, these child stacks didn’t have identifiable tags that would have allowed us to automate the cleanup. Fortunately, most of them used a recognizable name prefix, which allowed me to clear out the bulk of them manually.

Did I mention interdependencies? Because we’re not done! Some stacks had deployed S3 buckets. And guess what? CloudFormation won’t delete a stack if the S3 bucket isn’t empty. Naturally, 14 stacks got stuck in DELETE_FAILED. Luckily, the fix is straightforward: back up each bucket, empty it, and retry the stack deletion.

Deploying the API: Light at the End of the Tunnel?

Having finally cleared the web of interdependencies, it was time to delete the original API Gateway stack and redeploy it.

The deletion went smoothly (thank God), but—naturally—the creation did not.

First, the stack itself. A YML file existed in a GitHub repo, but it hadn’t been updated in ages. I knew I was better off using the stack definition stored within CloudFormation (and yes, I had saved a copy—I’m not that crazy).

This stack didn’t just deploy the API Gateway; it handled several AWS resources, including Lambdas. These Lambdas were still running on Python 3.7, a version that AWS no longer allows for creating new Lambdas. Fortunately, a quick upgrade to Python 3.12 was enough to satisfy the AWS gods.

And, to my surprise, the stack finally deployed without a hitch!

But to keep a long story short, there was still work to do. Several critical resources for the API were missing from the CloudFormation stack. These resources had been created manually in AWS over time, ignoring all Infrastructure as Code best practices (cries in Terraform). To restore service as quickly as possible (we’re DevOps, after all!), I recreated these resources manually once more.

Finally, several key components had the ARN of the old API hardcoded. This required a bit of “digital archaeology” to find every spot where a manual update to the new API was needed.

Ultimately, after several hours of troubleshooting, the API was back up and running, and developers could resume their deployments.

The incident began on April 4, 2024, at 3:24 PM and was resolved on April 5, 2024, at 8:46 AM.

Post-Mortem and Lessons Learned

Throughout the incident, I took intensive notes on every action taken. Having heard of the Post-Mortem principle, I knew this incident was the perfect candidate.

If you’re unfamiliar with the concept, a Post-Mortem is a document that retraces the steps of an incident, its impact, and its root cause. But most importantly—and most interestingly—it includes a Lessons Learned section. If you take this part seriously, it will be your best ally in building a more robust architecture.

In this section, you note three key points: what went well, what went wrong, and where you got lucky. And above all, be honest! Even if some points seem silly or make you look incompetent (and I’m saying this as the guy who manually deleted an API, so take it with a grain of salt), the goal isn’t to point fingers (the “blameless culture”). It’s about understanding the flaws in the system so we can fix them.

“The cost of failure is education.” — Devin Carraway (Source)

This might still feel a bit abstract, so let me share my lessons learned from this incident.

What went well

Two things went well during this incident.

First, the resolution was handled by a team member who knew the architecture inside out. This allowed for a quick understanding of what needed to be restored to get the service back online.

Second, there was excellent communication throughout. When the problem occurred, there was no attempt to hide it. Frequent updates were shared to report on progress. This is crucial—not only does it provide visibility, but communication often leads to helpful tips (like a colleague pointing you toward documentation you didn’t know existed).

What went wrong

This is the part that hurts. As I mentioned, you have to swallow your pride and highlight everything that could have been handled better.

For this incident, four things went wrong.

To start, the API infrastructure wasn’t consolidated in a single file or folder; it was scattered across multiple GitHub repos. This made it very difficult to get a bird’s-eye view of everything required for the API to function.

Next, there was a major issue with drift. This refers to the differences between your actual infrastructure and how it’s defined in your code. Ideally, no manual changes should ever occur, and everything should go through your IaC files. Had this been the case, a simple redeployment would have restored the service instantly.

Another issue was the heavy interdependence between resources. Many relied on CloudFormation stack outputs. Removing the parent stack essentially crippled the ability to manage the rest of the infrastructure.

Finally, identifying resources tied to our infrastructure was difficult. Our stack deployed resources without any associated tags, making it a scavenger hunt to find every piece of the puzzle.

Where we got lucky

This might sound positive, but it isn’t! This category covers things that went well only because of luck. Realize that at any moment, these could have been in the “What went wrong” column. Be glad this time, but don’t let your guard down!

In this case, we got lucky in three ways.

First, the incident was identified immediately (that’s the one perk of making a massive blunder yourself). But it could have been much worse! If the API had been deleted via an automated script, we had no monitoring in place to alert us.

Second, the person who deleted the API had deep knowledge of the project (yes, I’m talking about myself—I have to give myself some credit). This allowed for an immediate transition into resolution mode, but it could have been someone else who was totally lost.

Finally, this was our “Dev” API. The Production API was perfectly fine (a detail I intentionally saved for the end—you know, for the storytelling). So while the impact was minimal, the same incident could have happened in Prod with the same recovery nightmares. And that would have been much more expensive.

Preparing for the Future

Now that you’ve listed the problems encountered, as a good DevOps engineer, you must learn from them. Note everything that can be improved, but above all, set a plan! Otherwise, these are just empty words.

“A goal without a plan is just a wish.” — Antoine de Saint-Exupéry

In my case, the three key takeaways were:

Consolidation of Infrastructure as Code: Everything must be deployable in the blink of an eye. This is a project I would complete in the following months.
Improved Monitoring and Alerting: If this ever happens again, we need to be alerted immediately so we can react fast.
Clearer Documentation: Any team member should be able to handle such an incident, and that starts with reliable, understandable docs.

All these lessons were tracked as GitHub issues, and I made sure to knock them out over the following months.

Conclusion

As you’ve seen, accidents happen. The key is to make sure they benefit you and your organization. Use them as an opportunity to learn and patch vulnerabilities that were previously undetected (some companies even purposely create chaos for this very reason).

Thanks for reading to the end! I’ll leave you here—I’ve got some other infrastructures to delete!

Why Tagging your AWS Resources is a Must?

Tue, 24 Jun 2025 07:30:00 +0200

Introduction

The more services we deploy on AWS, the easier it is to get lost. At first, we know all the services we use by heart, the number of Lambdas or EC2 instances running. But little by little, it’s easy to feel overwhelmed, especially when different projects pile up. Quickly finding which resources belong to which project, or identifying those that weren’t properly cleaned up after a PoC, can quickly become a headache. And that’s without even mentioning cost visibility! Fortunately, there’s a simple yet incredibly powerful practice: tagging.

In this article, I want to show you that a good tagging strategy is crucial for your organization, the way you handle cost, and to secure your AWS account.

Ready? Tag along with me!

Why are tags useful?

Imagine labels on moving boxes. Without them, it’s impossible to know what’s inside or which room they belong to. Tags on AWS are the same! A tag is a piece of information (in key-value format) that you assign to your resources (EC2 instances, S3 buckets, RDS databases, etc.). A good tagging strategy allows you to, among other things:

Identify orphan resources: This is the classic scenario. A resource without a Project or Owner tag? There’s a good chance it has been forgotten and is consuming resources (and therefore money) for nothing. Listing untagged (or poorly tagged) resources is an essential first step in cleaning up.
Allocate costs: By tagging your resources with a project identifier, cost center, or team, you can then use AWS Cost Explorer to filter your expenses and understand precisely which projects consume the most. This is essential for internal chargebacks or simply for optimizing your budget.
Automate actions: Tags can serve as triggers for automation scripts (for example, backing up all EC2 instances with the tag Backup=Daily).
Manage access and security: The AWS IAM service can use tags to grant granular permissions.

In short, tagging is the foundation of good Cloud governance. For more details, I invite you to consult the tagging guide proposed by AWS.

Fortunately, we can associate multiple tags with the same resource. But this raises the question: how many tags are necessary?

This will depend on your company and each project, but overall, there are some tags that won’t hurt, no matter your situation:

Project: The name of the project linked to the resource. Usually, the GitHub repository name works well.
Environment: The desired environment (dev, uat, prod, …). Even if you have separate AWS accounts, this will help you identify if a resource got lost during deployment.
Owner: The owner of the resource. This could be a person, but more ideally a team (frontend, backend, security, …).

Depending on your use case, you’ll surely have other ideas for tags, but with the ones above, you’ll already have a good start! And in case of doubt, feel free to read the tagging best practices from AWS.

Let’s now look at some concrete examples of using tags in AWS.

AWS Cost Explorer: Tracking costs thanks to tags

One of the most concrete benefits of tagging is the visibility it provides over your expenses. AWS Cost Explorer is the tool of choice for this.

Once your resources are correctly tagged (for example, with the Project tag), you need to activate these tags for cost allocation in the AWS Billing and Cost Management console (Billing and Cost Management -> Cost Organization -> Cost Allocation Tags). Be aware that there might be a delay before the activated tags appear in Cost Explorer.

Once activated, you can:

Filter by tag: In Cost Explorer, you can filter your costs by the value of a specific tag.
Group by tag: You can also choose to group your expenses by tag. This will give you an overview of the cost distribution across different projects, environments, etc.
Create budgets based on tags: With AWS Budgets, you can set alert thresholds for costs associated with specific tags, helping you avoid nasty surprises.

Example of filtering by tag for my blog: the costs are minimal, am I a FinOps expert?

This ability to dissect your AWS bill by tags transforms cost management from an obscure chore into a transparent and controllable exercise. It’s a must for any organization concerned about its Cloud budget. You can now add FinOps to your LinkedIn bio!

AWS Resource Explorer: Keeping an eye on deployed resources

Now that we are convinced of the usefulness of tags, how do we list all the resources that use a specific tag? We will see this together, with one part in the AWS console, and another part in the terminal (for all the geeks reading this article!).

Visual exploration in the AWS console

If you are not yet familiar with AWS Resource Explorer, now is the time to discover it! This relatively recent service allows you to search and discover your AWS resources across all regions of your account, using a simple interface, much like a search engine.

The main advantage of Resource Explorer is its ability to give you a unified view. No more jumping from region to region. You activate indexing, and then you can search for your resources by name, ID, and of course… by tag!

It’s an excellent tool for:

Getting a quick overview.
Visually exploring resources associated with a specific tag.
Quickly identifying resources without having to code anything.

To use it, activate it in the desired regions (or all of them), let it index your resources, and then use the search bar with a syntax like tag.key:Project tag.value:Cloud*.

Example of a tag search for my blog: only three resources manage this blog!

A more technical approach with AWS CLI

For those who, like me, like to have control via the command line, or who need to automate these searches, the AWS CLI remains a key ally. More specifically, it’s the resourcegroupstaggingapi service that will interest us.

The key command is get-resources. Here is a typical example to list the ARNs of all resources having the tag Project with the value “Cloud Antoine Delia”:

1
2
3

aws resourcegroupstaggingapi get-resources \
    --tag-filters "Key=Project,Values=Cloud Antoine Delia" \
    | jq "[.ResourceTagMappingList[].ResourceARN]"

Which gives us:

1
2
3

[
  "arn:aws:s3:::antoiXXXXXXXXXXXXX"
]

Let’s break it down a bit:

aws resourcegroupstaggingapi get-resources: This is the API call; so far, so good.
--tag-filters "Key=Project,Values=Cloud Antoine Delia": This is where we specify our filter. We are looking for the tag Project with the value “Cloud Antoine Delia”. You can add multiple filters.
| jq "[.ResourceTagMappingList[].ResourceARN]": jq is a wonderful tool for manipulating JSON on the command line. Here, we use it to cleanly extract the list of ARNs of the found resources (you can do without it, but why make life difficult?).

Hey, wait a minute! In the console, you show us three resources, and now there’s only one! Where’s the catch?

Well spotted! You should know that when you make your API call, you are using a default region. However, if you have resources in various regions, you will need to specify it. Thus, if we add --region us-east-1 just before the jq pipe, we indeed get our two missing resources.

[
  "arn:aws:acm:us-east-1:6XXXXXXXXXXX0:certificate/f4ca3b13-XXXXXXXXXXXXX",
  "arn:aws:cloudfront::6XXXXXXXXXXX0:distribution/ERSXXXXXXXXXX"
]

Besides this detail that you must not forget, this command is extremely powerful because you can integrate it into scripts to:

Generate regular reports on resources by project.
Automatically detect resources that do not comply with your tagging policy.
Combine with other AWS CLI commands to perform actions on the listed resources.

For example, you could list all your resources of a certain type (e.g., all your EC2 instances) and check which ones are missing essential tags.

And if you’re wondering if AWS doesn’t already offer a service for this… It does! But we will talk about that in a future article (for the curious, I’m talking about AWS Config).

AWS IAM: Securing the use of your resources

Your resources are deployed in AWS, and now you want to give a team permission to manage all of that. But here’s the thing, in your AWS account, you also have critical resources that must absolutely not be compromised.

AWS IAM is here for you! Using a simple policy, you can specify that only resources with a certain tag can be modified by a user or group.

Let’s take the following example: you would like to allow a team to start or stop certain EC2 instances, but prevent them from accidentally stopping a critical EC2 instance! You just need to add the following policy to your users:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowStartStopEC2IfProjectCloud",
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Project": "Cloud Antoine Delia"
        }
      }
    },
    {
      "Sid": "AllowDescribeToSeeInstances",
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    }
  ]
}

Thus, your users will be autonomous in using their resources, without having the possibility to impact other resources.

Conclusion

You’ve understood it: a rigorous tagging strategy isn’t an option; it’s a necessity for operating serenely on AWS. Whether on the organizational, financial, or security management level, AWS gives you the means to take full advantage of your tags.

So, a little piece of advice: if you haven’t already, define a clear tagging policy in your organization, apply it, and use these tools to regularly check that everything is in order. You’ll thank me later!

How I Automated My Blog To The Cloud

Tue, 04 Mar 2025 07:30:00 +0100

How it started and why it ended

When I started blogging, I was still in engineering school. And the world of websites and blogs was still new to me. So when I wanted to create a personal blog, I turned to what I thought was the best solution at the time: WordPress.

Like it or not, WordPress is quite easy to use for beginners, even more for me who was still discovering how to host a website at the time.

So, with a brand new OVH account, I was able to quickly setup my personal blog in a few clicks!

And when the time came where I wanted to create a blog dedicated to professional topics, I naturally came back to my ol’ friend WordPress.

But after some months, the glory of WordPress started to fade into a darker view. Sure, it was convenient to add a new post by going through WordPress’ UI, but maintaining the website was always boresome. Moreover, a WordPress installation requires using a database, which incurred a higher bill at the end of each month.

All of this for a blog that I knew did not receive many visits each day.

Surely there was a better way to do this.

The start of a new plan

I will tell you here the path that led me to the rebirth of my blog on which you are today. If you are only interested in the technical details, you can move on to the next section. Otherwise, happy reading!

Discovering Hugo

Before searching for a new tool, I tried to think of what I needed.

My blog was essentially a simple webpage, a static website with no fancy JavaScript or stuff like that. Plus, there was no backend to handle API calls for user authentication for example, so I could simply not use any database at all.

So what I needed was essentially a way to focus on the content of the blog, and make sure I could attach to it a nice theme, and that was it!

But, as I’m an engineer, so in essence, a real bad designer, I wanted to avoid having to deal with CSS as much as possible. Again, I wanted to focus on the content, rather than the looks. So I searched and searched for the perfect tool. And I didn’t had to look too long to find it: Hugo.

Upon arriving on its website, I was greeted by an intriguing message message: The world’s fastest framework for building websites. Surely, I was curious, and dug into the documentation. And oh boy, was I not disappointed.

This was exactly what I wanted. A framework where I had to write in markdown format, that would be converted to HTML for me. Plus, it came with a bunch of themes, so I didn’t even had to bother about the design. I could just pick one that I liked, and move on.

So, without losing any more time, I started to convert my WordPress blog to a Hugo site. I came across a handy script called wordpress-to-hugo-exporter that converted my whole Wordpress database into compatible markdown syntax, and I was ready to go!

Now I needed to host this website somewhere.

A new challenger: AWS

In parallel, I had an opportunity working for a company that focused on the Cloud, more specifically AWS.

At the time, Cloud was brand new to me. But I was hearing more and more about it, and I wanted to check what was all the fuss around it. Would this be a true life-changer, as people would call it, or would it be yet again another buzz word, and become a dying trend?

Long story short, AWS was (and still is) amazing! Not only was it an amazing discovery, I could simply not see a world without Cloud anymore.

The more I learned about all these amazing AWS services, the more I thought: isn’t there a way for me to leverage the Cloud to host my blog?

With that in mind, I started looking for options that AWS provided. And because of my recent discovery of Hugo, my website did not needed PHP anymore, nor any database to work. It was now a simple, static website, which could perfectly fit in an S3 bucket.

Not only does AWS has a documentation about hosting a static website on an S3 bucket, but the pricing for it was just a few cents.

I had found an easy and cost-effective way to host my website.

The last point that was bothering me, is that to setup all of this, I had to create an S3 bucket, configure it, add a CDN with AWS CloudFront, take into account certificates with AWS ACM, and finally create a DNS record in AWS Route53. So many actions that, if done manually, could make it super difficult to reproduce this setup if I had to do it again.

I needed a robust way to configure this infrastructure.

Terraform to the rescue

When you start learning Cloud, you will usually hear about Infrastructure as Code.

The idea is simple: describe your Cloud infrastructure with… code! (yes, as the name stands by).

The benefits of using an IaC tool, is that you will be able to keep track of the changes done on your infrastructure (like you would do with code in GitHub).

Now, let’s say that someone wants to update the configuration of an AWS service. This person goes to the AWS console and start making their changes. But after some trial and error, they decide to give up for now, and to go back to the initial state. And here is the problem: they don’t know exactly how it was before! And unless they took note of the initial state before making their changes, they will, again, have to do a bunch of trial and errors before going back to the beginning.

All of that could have been prevented if the infrastructure was configured through an IaC tool to begin with. In such a case, a simple automated deploy could have done the trick!

There are a bunch of IaC tools out there. But how can I not talk about the most popular one, the one that made IaC a standard in the Cloud ecosystem: Terraform.

Terraform, as you could have guessed, is an Infrastructure as Code tool with a declarative approach, meaning that you have to declare the state of the infrastructure you wish to deploy. To do that, you need ton use a specific language: hcl (which stands for HashiCorp Configuration Language). Here is an example to create an S3 bucket with Terraform.

resource "aws_s3_bucket" "example" {
  bucket = "my-tf-test-bucket"

  tags = {
    Name        = "My bucket"
    Environment = "Dev"
  }
}

Let’s go back to my blog! Terraform seems to be the perfect tool to configure the infrastructure that will host my blog. All I had to do was to go through an architecture step to know which AWS services I needed, and how to interconnect them, before translating that to Terraform.

And here is the architecture diagram I came up with.

All of this was perfect, but I realized one thing. Every time I needed to add a post, or update the infrastructure, I had to clone my repository, make the appropriate changes, and do a lot of manual actions to update my blog.

Let’s say that I was not too pleased with that, and I intended to fix that!

Forget copy and paste with GitHub Actions

Like I said before, my website was ready. The last step I needed was about the deployment.

For now, when I had to create a new post, I had to manually build my Hugo website, and move the generated files to my S3 bucket. What a waste of time and energy!

So I explored the topic of CI/CD pipelines to make my life easy (if you’re interested in this topic, I suggest reading my post about setting up a CI/CD strategy). While Jenkins tried to get my attention, I decided to go with the obvious choice: GitHub Actions.

GitHub Actions are simple CI/CD pipelines that you can define directly in your GitHub repository. The main benefit, is that you do not need any additional account, nor to install anything: all is included! In addition, GitHub Actions use an easy-to-read YAML syntax (in opposition to Jenkins and its horrifying groovy!).

So I created a pipeline that was able to configure Terraform in case I made any modifications, and could also build and deploy my website automatically in my S3 bucket when I updated or added a post.

All the steps were looking good, mission accomplished!

Technically: What I Have Today

My blog is now fully automated and deploys to AWS!

If you want to dive into it, everything is publicly available on my GitHub repository: https://github.com/antoinedelia/cloud-optimist

Now, let me show you more details about it.

Project Structure

The project structure is as follow:

cloud-optimist/
├── .github/
│   └── workflows/
│       └── main.yml  # GitHub Actions
├── cloud/
│   └── ...  # My Hugo blog
├── terraform/
│   └── ...  # The Terraform configuration
└── README.md

As you can see, I split my project into three key directories:

.github : contains the main.yml file which defines the GitHub Actions CI/CD steps
cloud : contains my Hugo blog
terraform : contains all the Terraform infrastructure

The GitHub Actions

My GitHub Actions is rather simple, and can be divided into multiple steps.

The Initialization Step

Let’s have a look at the top of the file.

name: 'Build and Deploy'

on:
  push:
    branches:
    - master
  pull_request:

jobs:
  build-and-deploy:
    name: 'Build & Deploy'
    runs-on: ubuntu-latest
    environment: production

    defaults:
      run:
        shell: bash
        working-directory: cloud

Here, I ask GitHub to run the pipeline on the master branch, or if I have a Pull Request. Let me reassure you, I do not deploy anything to production with a Pull Request. You’ll see below that this only allows me to perform some “dry-run” checks and ensure no issues are found before I can safely deploy to production via the master branch.

I also ask GitHub to use bash and to use the cloud directory as the default working directory.

Let’s see what’s next:

    steps:
    - uses: actions/checkout@v3
    - uses: dorny/paths-filter@v2
      id: filter
      with:
        filters: |
          web:
            - 'cloud/**'
          terraform:
            - 'terraform/**'

This step is crucial if you wish to fasten your CI/CD and ultimately save cost.

I’m using the dorny/paths-filter actions which allows me to detect which files were modified in the last commit. In my case, I look for the cloud and terraform directories. This way, if no changes were detected on the Terraform side, no need to trigger the Terraform steps in my pipeline. In the same way, if the directory cloud is untouched, no need to try to build and deploy the contents of my blog. This will save you few cents related to the data transfer pricing to your AWS S3 bucket (no need to thanks me!).

The next steps speak for themselves.

    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v2
      
    - name: Update Git Submodules
      working-directory: ./
      run: git submodule update --init --recursive

I simply get the content of my repository and update the submodules. This last step was useful back when I was using submodules to handle my Hugo themes. I now use the Hugo Modules, which could let me remove this step altogether.

The Terraform Step

Let’s continue with Terraform:

    # Install the latest version of Terraform CLI and configure the Terraform CLI configuration file with a Terraform Cloud user API token
    - name: Setup Terraform
      if: steps.filter.outputs.terraform == 'true'
      uses: hashicorp/setup-terraform@v1
      with:
        cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

    # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
    - name: Terraform Init
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform init

    # Checks that all Terraform configuration files adhere to a canonical format
    - name: Terraform Format
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform fmt -check

    # Generates an execution plan for Terraform
    - name: Terraform Plan
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform plan

      # On push to master, build or change infrastructure according to Terraform configuration files
    - name: Terraform Apply
      working-directory: ./terraform
      if: steps.filter.outputs.terraform == 'true' && github.ref == 'refs/heads/master' && github.event_name == 'push'
      run: terraform apply -auto-approve

Now that’s a long step! Let’s break it down.

The first thing that should’ve catch your attention is this line:

`1`	`if: steps.filter.outputs.terraform == 'true'`

Do you remember? This is what allows us to tell if files were updated in the terraform directory. Here, I check if I need to run these steps or not.

Then, I am running some ordinary Terraform commands: terraform init to initialize my project, terraform fmt -check to ensure my code is fancy, and finally terraform plan to look at the changes planned.

Last but not least, terraform apply -auto-approve, which will deploy the changes to my AWS infrastructure. But if you look closely, you will notice I added two extra checks:

`1`	`github.ref == 'refs/heads/master' && github.event_name == 'push'`

This ensures that this step will only be ran if the event is a push on the master branch. This way, no need to worry about an unwanted deployment when I simply play with my infrastructure on another branch or in a Pull Request. Phew!

The Hugo and AWS Steps

Now that my infrastructure is ready, it is time to publish my blog!

    - name: Build
      uses: actions/setup-node@v2
      if: steps.filter.outputs.web == 'true'
    - run: sudo wget https://github.com/gohugoio/hugo/releases/download/v0.142.0/hugo_extended_0.142.0_linux-amd64.deb -O hugo.deb
    - run: sudo dpkg --install ./hugo.deb
    - run: hugo

    - name: Deploy to AWS
      uses: jakejarvis/s3-sync-action@master
      if: steps.filter.outputs.web == 'true' && github.ref == 'refs/heads/master' && github.event_name == 'push'
      with:
        args: --acl public-read --follow-symlinks --delete
      env:
        AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_REGION: 'eu-west-1'
        SOURCE_DIR: 'cloud/public/'

As for Terraform, we check if any modifications were done on the blog.

`1`	`if: steps.filter.outputs.web == 'true'`

Then, I’m retrieving an extended version of Hugo straight from the GitHub release (here the v0.142.0). I’m installing it and build my blog using the hugo command.

This step could be simplified using the peaceiris/actions-hugo actions

Finally, I’m using the jakejarvis/s3-sync-action actions (I realized while writing this post that this action has been archived a couple of days before, ouch! – I’ll add an edit later to mention an alternative) to move my blog’s files into my S3 bucket.

Of course, you’ll have to store some credentials in your GitHub repository so that your GitHub Actions can be authorized to use AWS, but nothing too crazy.

We are now done with the GitHub Actions part!

The Terraform part

Regarding Terraform, I’ll keep it short, as this is not too complex.

There is one personal choice here, is that I prefer to split my .tf files based on the AWS services used, rather than putting them all in a single main.tf file, which could make it hard to read. In the end, I have an s3.tf file for all resources linked to the S3 service, a cloudfront.tf file for everything related to CloudFront, and so on.

One important information, is that it is necessary to define at least the us-east-1 region, as you must deploy your ACM certificates there. For the rest, I’m deploying all my resources in the eu-west-1 region. To make this split, I’m using the Terraform alias. Here’s an example:

# The default configuration: all resources starting with `aws_` will use this provider
provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us_east_1"
}

resource "aws_s3_bucket" "site" {
  bucket        = var.bucket_name
  force_destroy = true
}

resource "aws_acm_certificate" "cert" {
  provider                  = aws.us_east_1  # Will be deployed in us-east-1
  domain_name               = var.domain_name
  subject_alternative_names = ["*.${var.domain_name}"]
  validation_method         = "DNS"
}

The Hugo part

My Hugo blog is based on the Stack theme. There is also a GitHub template available which allows you to quickly setup your own blog based on this theme.

Not a lot to say about this part, except that I encourage you to have a look at the Hugo Modules, which will save you from having to deal with submodules (and will make it easier when you need to update your theme).

Potential Improvements

While I’m extremely satisfied with the end results, I noted a few points that could be improved in the future.

First off, I saw that Hugo has a deploy command that could, well, deploy your blog directly to an S3 bucket. Perfect! So I need to learn a bit more about Hugo Deploy.

Finally, even though using AWS as my hosting platform is not expensive (less than 2$ a month), it is not free. But it would be completely possible to use GitHub Pages to make this blog accessible, while taking the advantage of GitHub as the hosting platform (on this point, I’m a bit reluctant, as then I would not be able to show-off with my beautifully made AWS diagrams…).

Conclusion

Here we are, you know everything!

From my beginnings with WordPress, which, while being useful, showed some weaknesses on the backup, maintainability and deployment part.

Of my research to find the perfect technical suite: AWS, Terraform, Hugo and GitHub Actions, all of that to enable a speedy deployment at almost no cost!

When I look back to my previous setup and compare to what I have today, I am so happy to have done the switch!

Now, it’s up to you! You have all the information to create your own blog and deploy all of this with a breeze!

Happy blogging !

How We Built a CI/CD Strategy That Onboards 100+ Python Projects in Under a Minute

Sun, 19 Jan 2025 16:45:00 +0200

We all know CI/CD is important. In fact, it seems impossible to imagine a world where we would ship a project without checking the quality of its code, and having a detailed test suite. Moreover, to enable developers to focus on development, all of this should be automated.

Well, things are not always so easy.

Today, I’ll share how we transitioned from having no CI/CD strategy to onboarding 100+ Python projects in under a minute.

The Real World

When I joined my current company, I noticed we were managing a large number of Python projects. But when I tried to check for the CI/CD of these, well it was a bit of a mess. Some projects had CI/CD in place, but not too many. And the ones with one were not using the same rules to verify code compliance. It was clear that putting in place a CI/CD for any given project was treated as best effort. And, to be honest, I cannot blame this on anyone.

Indeed, while CI/CD is widely recognized as a core component of any project, implementing it in the real world often proves challenging.

But I knew we could change that somehow. So before jumping into a strategy to put in place, I wanted to observe what had prevented people to implement a CI/CD to begin with.

Lack of Permissions

The first thing that I realized, is that not all developers had the same level of access to our Jenkins instance. So while some were able to create new pipelines for their projects, some couldn’t. In large organizations, it is not uncommon to have this kind of scenario.

Lesson learned: a lack of permissions should not be a drawback to use a CI/CD pipeline.

Lack of Knowledge

When people had the ability to create a pipeline, some did not, as they simply did not have the proper knowledge to do so. Here, I need to mention that our company uses a Jenkins instance, so developers familiar with other CI/CD tools (e.g., GitHub Actions, CircleCI) could not fully transfer these knowledge to Jenkins. Moreover, we did not have a clear documentation on the process to follow to create a new pipeline, so beginners, fearing to break something, would simply do not take the risk to mess with it.

Lesson learned: we should ensure people without proper Jenkins knowledge can use a CI/CD pipeline.

Lack of Time

It takes effort to put in place a CI/CD at the beginning of a project, something that might be overlooked by managers that want to ship a product as fast as possible. Moreover, it is sometimes difficult to quantify the return on investment of putting in place a CI/CD pipeline. And if it is difficult to prove this can bring business value, it will end up in the “we’ll deal with this later” box. And we know all too well that the tasks that end up in this box will never see the light of day again.

Lesson learned: setting up a CI/CD pipeline for a new or existing project should be easy and straightforward.

Lack of Clear Guidelines

Finally, I had a look at the projects that did have a CI/CD pipeline. They were working fine, but I could clearly see that they lacked a common vision. Some of them used the same formatter (black), but they not always used the same line-length. Some included a testing stage, some didn’t. Not only did this led to projects not having the same code quality and compliance, I also thought that this could potentially led to confusion for a newcomer, not knowing which standard to use.

Lesson learned: people should use the same CI/CD pipeline to ensure everyone follows the same guidelines (and we should ensure these guidelines are documented somewhere).

The Global Vision

After reviewing what could go wrong, it is now important to think of a solution that could address all of these, while following the company’s best practices, and using the tools at our disposal.

Formatting

The first stage should use a formatter to ensure every line of code in our codebase looks the same. This makes sure we are not ending up with different coding standards across our projects.

Previously, we used black as our formatter. But after hearing all the good news and testing the new cool kid in the block, we decided to switch to ruff, as it has the same benefits as black, but with a faster execution.

Linting

The next stage should use a linter to find potential issues with our code. This makes sure we avoid complexity in our code, as well as identifying code smells or security issues.

In the past, I used flake8 a lot. But given that we were already using ruff, and as it can also act as a linter, it was a no-brainer to keep it for this task.

There are many rules that ruff can apply. We decided to use some of them by default, while letting developers the choice to update the ones their project would follow.

Unit Tests

Tests are a critical part of the development of any project. It ensures we ship quality code to production, while also being able to trust that our code would run just fine.

We decided to use pytest to run these tests. The default code directory would be called src, and all tests should be in a tests folder, with files being prefixed by test_.

Code Coverage

Closely related to unit tests, code coverage ensures we are able to know how much of our code has been tested. This could quickly tell us if we sufficiently tested our code, as well as pointing out the remaining lines to cover.

As we were using pytest, we decided to use pytest-cov to generate a coverage report, as it integrates nicely with pytest.

We set the minimum coverage threshold at 50%. Anything lower would risk overlooking significant portions of code, while setting it higher might discourage developers from writing the necessary tests.

Organization Folders for Jenkins

We had our different stages ready. Now all we needed to do, was find a way to globally apply said pipeline to our Python repositories. So I tried looking for a way to easily do that in Jenkins.

That’s when I stumbled upon Organization Folders.

Organization Folders are designed for scenarios like ours: automatically scan an organization (as in, a GitHub organization), filter the repositories you want, and apply a Jenkins pipeline to them.

In our example, we are able to look for all repositories with the “python” topic, and identify them as Python projects. They will then be automatically built. If a new repository is created with this topic, it will also get picked up by Jenkins.

So, in less than 5 seconds, your project could be onboarded, without having to create it in Jenkins. All is done automatically so you can focus on your code.

Examples and Documentation

All of this was great, but I was fearing of one last obstacle. What would happen if developers adopted the CI/CD pipeline only to find stages failing, with no clear documentation to resolve issues? They would probably give up or try to fix it later, which would destroy the initial goal.

So I knew that if I wanted to onboard people in this, we needed to deliver clear documentation with direct examples, so they would be able to understand why these errors might appear, and how to fix them.

Especially on the unit tests stage, as I know this is always a daunting task to start with. So I prepared a project in advance with some unit tests that I knew they could have a look at to take some inspiration, or that directly covered some tricky parts (mocking boto3 API calls, etc.).

The last step was to make a presentation on all of the above. This was key to give meaning to people, so they could really understand the point of doing all this, while making sure they had all the keys to be autonomous.

What We Have Today

From a developer’s view, all he has to do to get his Python project onboarded, is to add the python topic in his repository, and ensures a pyproject.toml file is created at the root of the repository.

These two requirements are here to tell Jenkins which project it should take into account. Moreover, the pyproject.toml file is mandatory for the ruff stages in the pipeline.

With that done, his Python project will now check for formatting issues, linting errors, validation of unit tests, and code coverage.

All in all, we are now able to setup CI/CD pipelines in less than a minute, whether you have a new or existing project!

Some Flaws

While this simplifies things a lot, there is still room for improvements.

The pipeline cannot be enforced

Currently, this CI/CD pipeline cannot be enforced because developers can simply remove the python topic to bypass it. And while this is fine at first, as we do not want to block developers in their work, at some point, the goal is still to make sure we are applying the same best practices in all Python projects.

This might be resolved in the future by the use of GitHub Rulesets.

They essentially act quite the same as branch protection rules, expect you define these rules at the organization level.

This way, we could be able to protect our main branches for all our repositories that matches a specific custom properties, and require them to successfully pass the CI/CD pipeline before they are able to merge.

The pipeline’s stages can be ignored

The pyproject.toml is used to say to ruff which format it should apply, or which rules to follow for the linting part.

And because we are currently using the pyproject.toml inside each repository, a developer could just update the rules on his own, bypassing all the guidelines we were trying to apply in the first place.

Again, while we are allowing this for now to account for the number of fixes to resolve at first, in the end, we might want to prevent this from happening.

We could either use a common and fixed pyproject.toml file, or add it in the GitHub’s CODEOWNERS file to ensure it cannot be modified without strict approval.

What’s Next?

With these in place, we can now think about the evolution of this pipeline.

For example, our company has a SonarQube instance. We would be interested to add a stage that could scan the repository for code smells.

We are also exploring the use of mkdocs, so that projects can share a common style for documentation.

And one of my personal favorites, we might want to explore the use of uv to install requirements, as it is significantly faster than the old pip guy!

You might have also noticed that while I talked about CI/CD throughout this post, at no point do we have a step that, well, deploys anything (which basically leaves us with a CI pipeline). We are already thinking about a way to build and deploy Python packages to our Artifactory repository manager, which would finally make this a CI/CD pipeline!

Finally, we only covered Python in this post, but the same logic could apply to other types of projects. To give you an example, we are also working on a CI/CD pipeline for Terraform.

Conclusion

By tackling permissions, knowledge gaps, and inconsistent guidelines, we built a unified CI/CD strategy that now supports over 100 Python projects. It’s proof that with the right approach, automation is achievable for any organization

It’s been a long but rewarding journey!

I hope this post proved the value of CI/CD, helped you understand what could prevent it from being applied, and gave you some ideas on how to implement a similar strategy in your organization!