The Cloud Optimist

Cloud Sovereignty: Getting Started with Scaleway

Sun, 17 May 2026 07:30:00 +0100

The Day I Accidentally Deleted An Api and What It Taught Me About Devops

Tue, 17 Feb 2026 07:30:00 +0100

April 4, 2024, had started off so well. A hot coffee in hand, VS Code humming with activity, and smooth electronic music in my ears—all the ingredients for a perfect day that surely couldn’t foreshadow the coming disaster.

Today, I’d like to tell you how a seemingly quiet day became a defining moment in my DevOps career!

Incident

The Context

At the time, I was working in a team tasked with providing a global API Gateway on AWS, featuring several endpoints managed by different teams. Specifically, setting up this API Gateway was the initial phase of the project. It included a DNS record pointing to the API, the API Gateway itself, and a Cognito Authorizer configured with several clients.

Once this API was ready, external teams could deploy their own endpoints onto it. To facilitate this, a CI/CD pipeline was in place, using CloudFormation to attach the endpoints directly to the existing API.

I’m keeping it as simple as possible here, but if you’re interested in the technical nitty-gritty, feel free to reach out!

On my end, besides being responsible for the global API infrastructure, I occasionally worked on or bootstrapped certain services.

The Beginning of the Drama

If you’ve ever worked with CloudFormation, you might be familiar with the dreaded UPDATE_ROLLBACK_FAILED status. This happens when you try to update a CloudFormation stack, but it encounters an error. It then tries to roll back, but if that rollback fails too, your stack gets stuck in UPDATE_ROLLBACK_FAILED.

This state is particularly annoying because you cannot trigger any new deployments until you’ve resolved the underlying issue.

That is exactly what happened on that fateful April 4, 2024. One of our services got stuck in this state, and I started investigating the “why” and the “how.”

After a few minutes, I noticed the problem stemmed from the API resource itself. Without overthinking it, and in an effort to unblock the situation quickly, I headed straight to the AWS Console, specifically to the API Gateway service. I found the resource in question and rushed to delete it.

At that exact moment, something very strange happened. An unexpected behavior that made my blood run cold. Instead of staying on the same page, AWS redirected me to the main API Gateway dashboard—the page that lists your available APIs. It now displayed a count of 0.

I realized then that I hadn’t deleted a specific resource within the API, but the entire API Gateway itself.

Anatomy of a Failure

I never thought I’d make such a rookie mistake. And I imagine as you read this, you’re thinking the same thing.

Because to get it that wrong, you really have to try!

Let me reconstruct the crime scene for you. Here is what I saw when I decided to delete a resource from the API Gateway.

Any resemblance to an actual situation is entirely coincidental.

In my shoes, which button would you have clicked? Easy! The Delete button right next to the resource!

As for me (and I still don’t know what possessed me), I chose to click the API actions button. After all, I did want to perform an “action”!

And what happens when you click that button?

Yikes! Right away, you can see it’s not what we want at all. But do you think that stopped me? Not a chance! I came here to delete something, and when I saw the word Delete, I stopped thinking. I should have kept reading, because then I would have seen the word API right next to it.

Fortunately, our friends at AWS thought of everything! When you click Delete API, they still ask if you are absolutely sure. A nice dialog box pops up, showing the name of the API and asking you to type the word confirm to validate the operation.

That’s all well and good, but the AWS engineers underestimated my impatience. In that moment, the confirmation request wasn’t a warning; it was an obstacle to my goal. In a heartbeat, I typed confirm and hit enter.

So, here is the last thing I saw before finally realizing the gravity of my mistake:

A vision of pure horror.

I wanted to walk you through this process to make one thing clear: you can put all the safeguards in the world in place, but you can’t protect a system against an impatient individual, because they won’t stop to read the warnings.

So, the next time you have to perform an “innocuous” action, take the time to read and ensure you are actually on the right path.

That being said, let’s move on to the crucial stage: resolving the incident!

Resolution

The Click

Back to the moment of impact. After realizing what had happened, I had a sudden jolt—a moment of clarity (without the aid of any white powder, I promise). I knew what I had to do: resolve the incident, but more importantly, document every single action I took.

A few weeks prior, I had looked into the 2017 GitLab incident, which took the service down for hours and resulted in data loss for some users. That’s how I discovered the concept of a Post-Mortem and why they are so vital in these situations. But I’ll save that for the next section.

In the meantime, if you’re interested in these topics, I highly recommend Kevin Fang’s YouTube channel where he, in his own words, “reads postmortems and makes low-quality videos about them.”

Rollback, Impact, and Communication

My first thought was that there might be a way to trigger a rollback directly within AWS to minimize the impact on users. If I had actually read the warning message earlier, I would have known that was impossible (but if I had read it, I wouldn’t have been in this mess anyway).

With no “undo” button in sight, I accepted that I was facing a major incident with a global impact. I first set out to map out the full extent of the damage. In my case, not only was the API down (thanks, Captain Obvious), but no new deployments could be made until the API was back online.

Finally, I made sure to notify the internal team. This kept them in the loop and let them know I was actively working on a fix.

Analysis and Troubleshooting

It was time to roll up my sleeves and find a way to redeploy the API Gateway.

First, I went to the CloudFormation service, remembering that this API was originally deployed through it. I tried updating the stack, hoping it would bring my dear API back like magic.

Obviously, it wasn’t going to be that simple. Updating the stack was impossible because the manual deletion had put the stack into a “hybrid” state that it couldn’t reconcile.

Since updating was out of the question, the logical next step was to delete the stack and redeploy it from scratch. That’s when the real trouble started. This CloudFormation stack produced several Outputs. Two of these were required by “child” stacks that had deployed their endpoints on the API. Consequently, CloudFormation blocked me from deleting the stack because it would break all the others.

After brainstorming alternatives, this heavy interdependence led me to a tough decision: I had to delete all the “child” stacks in CloudFormation—a total of 81 stacks.

To make matters worse, these child stacks didn’t have identifiable tags that would have allowed us to automate the cleanup. Fortunately, most of them used a recognizable name prefix, which allowed me to clear out the bulk of them manually.

Did I mention interdependencies? Because we’re not done! Some stacks had deployed S3 buckets. And guess what? CloudFormation won’t delete a stack if the S3 bucket isn’t empty. Naturally, 14 stacks got stuck in DELETE_FAILED. Luckily, the fix is straightforward: back up each bucket, empty it, and retry the stack deletion.

Deploying the API: Light at the End of the Tunnel?

Having finally cleared the web of interdependencies, it was time to delete the original API Gateway stack and redeploy it.

The deletion went smoothly (thank God), but—naturally—the creation did not.

First, the stack itself. A YML file existed in a GitHub repo, but it hadn’t been updated in ages. I knew I was better off using the stack definition stored within CloudFormation (and yes, I had saved a copy—I’m not that crazy).

This stack didn’t just deploy the API Gateway; it handled several AWS resources, including Lambdas. These Lambdas were still running on Python 3.7, a version that AWS no longer allows for creating new Lambdas. Fortunately, a quick upgrade to Python 3.12 was enough to satisfy the AWS gods.

And, to my surprise, the stack finally deployed without a hitch!

But to keep a long story short, there was still work to do. Several critical resources for the API were missing from the CloudFormation stack. These resources had been created manually in AWS over time, ignoring all Infrastructure as Code best practices (cries in Terraform). To restore service as quickly as possible (we’re DevOps, after all!), I recreated these resources manually once more.

Finally, several key components had the ARN of the old API hardcoded. This required a bit of “digital archaeology” to find every spot where a manual update to the new API was needed.

Ultimately, after several hours of troubleshooting, the API was back up and running, and developers could resume their deployments.

The incident began on April 4, 2024, at 3:24 PM and was resolved on April 5, 2024, at 8:46 AM.

Post-Mortem and Lessons Learned

Throughout the incident, I took intensive notes on every action taken. Having heard of the Post-Mortem principle, I knew this incident was the perfect candidate.

If you’re unfamiliar with the concept, a Post-Mortem is a document that retraces the steps of an incident, its impact, and its root cause. But most importantly—and most interestingly—it includes a Lessons Learned section. If you take this part seriously, it will be your best ally in building a more robust architecture.

In this section, you note three key points: what went well, what went wrong, and where you got lucky. And above all, be honest! Even if some points seem silly or make you look incompetent (and I’m saying this as the guy who manually deleted an API, so take it with a grain of salt), the goal isn’t to point fingers (the “blameless culture”). It’s about understanding the flaws in the system so we can fix them.

“The cost of failure is education.” — Devin Carraway (Source)

This might still feel a bit abstract, so let me share my lessons learned from this incident.

What went well

Two things went well during this incident.

First, the resolution was handled by a team member who knew the architecture inside out. This allowed for a quick understanding of what needed to be restored to get the service back online.

Second, there was excellent communication throughout. When the problem occurred, there was no attempt to hide it. Frequent updates were shared to report on progress. This is crucial—not only does it provide visibility, but communication often leads to helpful tips (like a colleague pointing you toward documentation you didn’t know existed).

What went wrong

This is the part that hurts. As I mentioned, you have to swallow your pride and highlight everything that could have been handled better.

For this incident, four things went wrong.

To start, the API infrastructure wasn’t consolidated in a single file or folder; it was scattered across multiple GitHub repos. This made it very difficult to get a bird’s-eye view of everything required for the API to function.

Next, there was a major issue with drift. This refers to the differences between your actual infrastructure and how it’s defined in your code. Ideally, no manual changes should ever occur, and everything should go through your IaC files. Had this been the case, a simple redeployment would have restored the service instantly.

Another issue was the heavy interdependence between resources. Many relied on CloudFormation stack outputs. Removing the parent stack essentially crippled the ability to manage the rest of the infrastructure.

Finally, identifying resources tied to our infrastructure was difficult. Our stack deployed resources without any associated tags, making it a scavenger hunt to find every piece of the puzzle.

Where we got lucky

This might sound positive, but it isn’t! This category covers things that went well only because of luck. Realize that at any moment, these could have been in the “What went wrong” column. Be glad this time, but don’t let your guard down!

In this case, we got lucky in three ways.

First, the incident was identified immediately (that’s the one perk of making a massive blunder yourself). But it could have been much worse! If the API had been deleted via an automated script, we had no monitoring in place to alert us.

Second, the person who deleted the API had deep knowledge of the project (yes, I’m talking about myself—I have to give myself some credit). This allowed for an immediate transition into resolution mode, but it could have been someone else who was totally lost.

Finally, this was our “Dev” API. The Production API was perfectly fine (a detail I intentionally saved for the end—you know, for the storytelling). So while the impact was minimal, the same incident could have happened in Prod with the same recovery nightmares. And that would have been much more expensive.

Preparing for the Future

Now that you’ve listed the problems encountered, as a good DevOps engineer, you must learn from them. Note everything that can be improved, but above all, set a plan! Otherwise, these are just empty words.

“A goal without a plan is just a wish.” — Antoine de Saint-Exupéry

In my case, the three key takeaways were:

Consolidation of Infrastructure as Code: Everything must be deployable in the blink of an eye. This is a project I would complete in the following months.
Improved Monitoring and Alerting: If this ever happens again, we need to be alerted immediately so we can react fast.
Clearer Documentation: Any team member should be able to handle such an incident, and that starts with reliable, understandable docs.

All these lessons were tracked as GitHub issues, and I made sure to knock them out over the following months.

Conclusion

As you’ve seen, accidents happen. The key is to make sure they benefit you and your organization. Use them as an opportunity to learn and patch vulnerabilities that were previously undetected (some companies even purposely create chaos for this very reason).

Thanks for reading to the end! I’ll leave you here—I’ve got some other infrastructures to delete!

How to migrate your State from Terraform Cloud to AWS S3?

Wed, 17 Dec 2025 07:30:00 +0200

Introduction

When I first started using Terraform (specifically for deploying my blog), I turned to Terraform Cloud (this name was later changed to HCP Terraform).

Terraform and the State

If you know a little bit about Terraform, you are likely familiar with the term State. This effectively corresponds to the current state of your infrastructure and allows Terraform to know what changes to expect if you modify your Terraform files. This State is represented by a file: terraform.tfstate.

By default, Terraform stores this file locally on your machine, but it recommends using a backend storage (read: “store this file somewhere else, for crying out loud!”).

Obviously, Terraform first suggests the HashiCorp solution: Terraform Cloud. By creating an account, you can use their backend to store your state files for free. Naturally, that is exactly what I did a few years ago.

The End of Terraform Cloud

Until this week, when I received this email from HashiCorp.

Ouch! We should have seen it coming—all good things must come to an end! The free option from back then will soon cease to exist, so I have to make a choice: upgrade my HashiCorp “plan,” or store my states elsewhere.

Looking more closely at their current offers, HashiCorp still proposes a free version, though it is limited in the number of resources that can be deployed (500 currently). It’s a fairly generous offer, but not knowing how it might evolve, I’ve decided to move my States once and for all.

Farewell, Terraform Cloud!

An alternative for storing your states

Fortunately for me, there are plenty of ways to store a Terraform state, thanks to Remote State.

To put it simply, you just need to tell Terraform where your state file will be stored. We use the backend block to do this.

There are quite a few options, but since I have an AWS account where I deploy all my resources, I naturally gravitated towards the S3 backend.

Here is an example of how to configure an S3 backend with Terraform:

terraform {
  backend "s3" {
    bucket = "mybucket"
    key    = "path/to/my/key/terraform.tfstate"
    region = "eu-west-1"
  }
}

Now that we have an alternative, it’s time to start the migration!

Migrating the State to S3

I’ll use the migration of the state I use to deploy this blog as an example.

Currently, if we look at my deployment GitHub Actions workflow, we can see that I authenticate to my Terraform Cloud account using a token. That part will no longer be necessary (at least for the credentials; the Terraform CLI will still need to be initialized).

Next, we need to create an S3 bucket to hold the future state files.

Finally, I’ll need to add a backend block to my project to tell Terraform to stop using Terraform Cloud and start using S3 when accessing the state.

Hold on, we aren’t done yet! If I leave this bucket empty, the next time I try to deploy my blog, Terraform will look for a state that… doesn’t exist! It will assume that no resources have been deployed to my AWS account yet. Consequently, Terraform will try to recreate resources that already exist.

The last step, therefore, is to retrieve the state file for my blog project and move it into my S3 bucket.

I then need to ensure that the information I provided in the backend block matches reality.

Once everything is ready, it’s time to test and deploy!

I intentionally omitted a few steps in this migration (migrating workspace variables, using GitHub Secrets for the S3 bucket name, adding AWS credentials to GitHub Actions, etc.) to keep this article concise. If you want to see more, check out the Pull Request I opened to perform this migration.

If everything went according to plan, you should see the phrase No changes. Your infrastructure matches the configuration. appear during your terraform plan!

Conclusion

It’s always a pain to perform migrations on existing infrastructure. But with a little time and some willpower, it’s the kind of task that ultimately only takes a few hours.

So, don’t wait until the last minute—dive in!

How and Why to Set Up AWS IAM Identity Center on Your AWS Account

Wed, 19 Nov 2025 07:30:00 +0200

Introduction

If you’re juggling multiple AWS accounts, or even just a single account with different users, you know that access management can quickly become a real headache. Creating individual IAM users in every account, managing their permissions, ensuring everyone is using MFA… Phew, just thinking about it is enough to give you cold sweats! But what if I told you there is an elegant, centralized, and much more secure solution to handle all of this? Ladies and Gentlemen, let me introduce you to IAM Identity Center!

Ever since I discovered and implemented IAM Identity Center (which some of you might know by its former name, AWS Single Sign-On or AWS SSO), my life as a Cloud admin has changed radically. It’s the kind of tool that, once adopted, makes you wonder how you ever lived without it.

So, what is it exactly, and why is it so great? Let’s find out together.

IAM Identity Center: What’s the Deal?

Simply put, IAM Identity Center is an AWS service that allows you to centrally manage access to all your AWS accounts and Cloud applications. Whether you have a handful of accounts or a sprawling AWS organization with dozens, or even hundreds of accounts, IAM Identity Center is there to simplify the task for you.

It offers a single point of entry (a web access portal) for your users, allowing them to access the roles and accounts they are entitled to, all with a single authentication.

The Benefits of IAM Identity Center

If I sound enthusiastic, it’s because the benefits are numerous and significant. To name the most important ones:

Single Sign-On (SSO): Your users log in once via the AWS portal (or via your existing identity provider if you have one) and then access all the accounts and roles assigned to them, without having to re-authenticate for every single account.
Centralized Management: You manage all your users, groups, and their permissions (via “Permission Sets”) from a single place, even if they need to access dozens of different AWS accounts.
Temporary Credentials: This is one of the most important points! When users access an account via IAM Identity Center, they obtain temporary credentials with a limited lifespan. Say goodbye to lingering IAM Access Keys that pose a major security risk.
MFA (Multi-Factor Authentication): You can (and should!) enforce the use of MFA directly at the IAM Identity Center login level.

If I’ve piqued your curiosity and you want to know more, check out the AWS documentation on IAM Identity Center. For those who are already sold, let’s continue!

Getting Started with IAM Identity Center

Setting up IAM Identity Center is surprisingly simple, especially if you use the built-in Identity Center directory as your identity source. Here are the key steps to get started:

Initial Configuration: Head to the AWS console and search for “IAM Identity Center”. Heads up: choose your AWS region carefully to host IAM Identity Center right from the start, as it is currently complex to change the IAM Identity Center region once configured. The initial setup is often guided and quick. You will choose your identity source (the Identity Center directory, AWS Managed Microsoft AD, or an external provider).
Creating Groups and Users: Define groups relevant to your organization (e.g., Developers, Administrators, …). Then create your users and assign them to these groups. If you are using an external IdP, this step will consist of synchronizing your existing users and groups instead.
Creating “Permission Sets”: A Permission Set is a collection of permissions (similar to an IAM policy) that you can reuse. You can start from AWS-managed policies (e.g., AdministratorAccess, ReadOnlyAccess) or create your own.
Assigning Access: This is where the magic happens. You assign a group (or a user) to one or more AWS accounts, giving them the right to use a specific Permission Set on those accounts. For example, the Developers group might have the PowerUserAccess Permission Set on the development AWS accounts.
Enforcing MFA: In the IAM Identity Center settings, configure MFA to be mandatory for all your users.
Sharing the AWS Access URL: Every IAM Identity Center configuration has a unique URL for the access portal (e.g., d-xxxxxxxxxx.awsapps.com/start). This is the URL your users will bookmark to log in.

Once logged into the portal, users will see the list of AWS accounts and roles (defined by the Permission Sets) they have access to. One click, and they are in the console of the chosen AWS account with the right permissions!

They can also get temporary credentials for the CLI. But speaking of which, how do you connect to an AWS account via CLI using IAM Identity Center? Let’s check it out!

Configuring CLI Access

The AWS documentation for configuring CLI authentication with IAM Identity Center is pretty clear, but I’ll walk you through the steps anyway. I’m assuming you already have your AWS CLI installed.

The first and only command to run will be the following:

`1`	`aws configure sso`

Next, you will need to enter some information specific to your IAM Identity Center. The most important being the “SSO start URL” (which you can find in IAM Identity Center under the name “AWS access portal URL”, a URL ending in /start). You then need to enter the region where your configuration is located; as for the “SSO registration scopes”, you can leave the default value.

SSO session name (Recommended): default
SSO start URL [None]: [https://xxxxxxxxxxxx.awsapps.com/start](https://xxxxxxxxxxxx.awsapps.com/start)
SSO region [None]: eu-west-1
SSO registration scopes [sso:account:access]:

If everything goes well, a window should open in your browser to validate your identity. Once that’s done, head back to your terminal.

In my case, I only have one AWS account, and IAM Identity Center selects it for me by default. But you might have a choice of which AWS account to select. The same goes for the role.

Finally, choose a profile name to use for your future API calls. I advise you to use default, which will allow you to avoid adding --profile my-aws-profile to the end of every command.

The only AWS account available to you is: xxxxxxxxxxxx
Using the account ID xxxxxxxxxxxx
The only role available to you is: AdministratorAccess
Using the role name "AdministratorAccess"
Default client Region [None]: eu-west-1
CLI default output format (json if not specified) [None]: json
Profile name [AdministratorAccess-xxxxxxxxxxxx]: default
The AWS CLI is now configured to use the default profile.
Run the following command to verify your configuration:

aws sts get-caller-identity

And there you go, you’re all set!

Conclusion

As you might have gathered, I am a big fan of AWS IAM Identity Center. It brings an indispensable layer of security while considerably simplifying access management, whether for administrators or users.

It is a true pillar for a well-managed and secure AWS infrastructure. If you aren’t using it yet, I strongly encourage you to explore setting it up. It’s a minimal time investment for huge gains in security and efficiency.

Your infrastructure (and your security teams) will thank you!

On The Trail of The Lost Resource - My Investigation With Athena and CloudTrail

Wed, 09 Jul 2025 12:30:00 +0200

Introduction

Recently, I had to play detective in our AWS account.

A resource was there (a Cognito User Pool), plain as day, but nobody could remember where it came from. And of course, there were no tags to help us (if only they had read my post on why tagging your AWS resources is a must).

My mission, should I choose to accept it: find out who created it. The problem? The event was about four months old.

My first instinct was to turn to AWS CloudTrail. And there, I hit my first wall: the event history is only viewable for the last 90 days. Dead end.

Luckily, I knew our CloudTrail logs were archived in an S3 bucket. My first thought was to manually download the archives for the right month, unzip dozens of JSON files, and hit Ctrl+F while praying for a miracle. Let’s just say it was neither efficient, pleasant, nor fast.

So, I wondered if there wasn’t a simpler way to search through this pile of logs, and I finally found the perfect solution: AWS Athena.

Querying your S3 logs with Athena

For those unfamiliar, AWS Athena is an interactive query service that makes it easy to analyze data directly in Amazon S3 using standard SQL. Basically, you can run queries on files (JSON, CSV, etc.) as if they were in a traditional database. No more downloading anything!

The idea, then, is to “map” our CloudTrail logs stored in S3 to a table in Athena. To do this, we use a single CREATE EXTERNAL TABLE query. Following the AWS documentation on the subject, I ran the following query in the Athena console.

This query creates a table and uses a very handy feature called “partition projection.” This allows Athena to infer the location of the logs based on the date, without having to manually manage partitions. This is very convenient when the structure is standardized, as is the case with AWS CloudTrail.

CREATE EXTERNAL TABLE cloudtrail_logs_pp (
    eventversion STRING,
    useridentity STRUCT<
        arn:STRING
    >,
    eventtime STRING,
    eventsource STRING,
    eventname STRING,
    awsregion STRING,
    requestparameters STRING,
    responseelements STRING,
    additionaleventdata STRING,
    requestid STRING,
    eventid STRING,
    resources ARRAY<STRUCT<
        arn:STRING,
        accountid:STRING,
        type:STRING
    >>,
    eventtype STRING,
    eventcategory STRING
)
PARTITIONED BY (
    `timestamp` string
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/CloudTrail/REGION' -- Replace with your S3 bucket path
TBLPROPERTIES (
    'projection.enabled'='true',
    'projection.timestamp.format'='yyyy/MM/dd',
    'projection.timestamp.interval'='1',
    'projection.timestamp.interval.unit'='DAYS',
    'projection.timestamp.range'='2024/01/01,NOW', -- Replace with your start date
    'projection.timestamp.type'='date',
    'storage.location.template'='s3://BUCKET-NAME/AWSLogs/ACCOUNT-ID/CloudTrail/REGION/${timestamp}' -- Replace with your S3 bucket path
)

Warning: Don’t forget to replace the s3://... URLs with the exact path to your S3 bucket where your CloudTrail logs are stored, and to adjust the projection.timestamp.range property to the period you’re interested in.

Investigation Time: Finding the Information

Once the table is created (which only takes a few seconds), the hardest part is over! My investigation could finally begin.

I was looking for who had created a Cognito UserPoolClient on a specific date. So my SQL query looked like this:

SELECT
    eventTime,
    eventName,
    userIdentity.arn
FROM
    cloudtrail_logs_pp
WHERE
    timestamp = '2025/02/25'
    AND eventName = 'CreateUserPoolClient';

In just a few seconds, Athena scanned the logs for the requested day and returned the result.

I had the exact time, the event, and most importantly, the ARN of the user who performed the action. Mission accomplished!

The Verdict… and the Culprit

The funniest part of this story?

After setting up this solution and finding the information so easily, I discovered that the “culprit” who had created this resource four months ago… was me. I had completely forgotten!

Jokes aside, this experience confirmed one thing for me: taking a few minutes to set up Athena on your CloudTrail logs is an incredibly worthwhile investment. You’re giving yourself a long-term auditing and search capability that will save you hours of manual searching the day you really need it. Don’t be like me; don’t wait until you’re stuck to set it up!

Why Tagging your AWS Resources is a Must?

Tue, 24 Jun 2025 07:30:00 +0200

Introduction

The more services we deploy on AWS, the easier it is to get lost. At first, we know all the services we use by heart, the number of Lambdas or EC2 instances running. But little by little, it’s easy to feel overwhelmed, especially when different projects pile up. Quickly finding which resources belong to which project, or identifying those that weren’t properly cleaned up after a PoC, can quickly become a headache. And that’s without even mentioning cost visibility! Fortunately, there’s a simple yet incredibly powerful practice: tagging.

In this article, I want to show you that a good tagging strategy is crucial for your organization, the way you handle cost, and to secure your AWS account.

Ready? Tag along with me!

Why are tags useful?

Imagine labels on moving boxes. Without them, it’s impossible to know what’s inside or which room they belong to. Tags on AWS are the same! A tag is a piece of information (in key-value format) that you assign to your resources (EC2 instances, S3 buckets, RDS databases, etc.). A good tagging strategy allows you to, among other things:

Identify orphan resources: This is the classic scenario. A resource without a Project or Owner tag? There’s a good chance it has been forgotten and is consuming resources (and therefore money) for nothing. Listing untagged (or poorly tagged) resources is an essential first step in cleaning up.
Allocate costs: By tagging your resources with a project identifier, cost center, or team, you can then use AWS Cost Explorer to filter your expenses and understand precisely which projects consume the most. This is essential for internal chargebacks or simply for optimizing your budget.
Automate actions: Tags can serve as triggers for automation scripts (for example, backing up all EC2 instances with the tag Backup=Daily).
Manage access and security: The AWS IAM service can use tags to grant granular permissions.

In short, tagging is the foundation of good Cloud governance. For more details, I invite you to consult the tagging guide proposed by AWS.

Fortunately, we can associate multiple tags with the same resource. But this raises the question: how many tags are necessary?

This will depend on your company and each project, but overall, there are some tags that won’t hurt, no matter your situation:

Project: The name of the project linked to the resource. Usually, the GitHub repository name works well.
Environment: The desired environment (dev, uat, prod, …). Even if you have separate AWS accounts, this will help you identify if a resource got lost during deployment.
Owner: The owner of the resource. This could be a person, but more ideally a team (frontend, backend, security, …).

Depending on your use case, you’ll surely have other ideas for tags, but with the ones above, you’ll already have a good start! And in case of doubt, feel free to read the tagging best practices from AWS.

Let’s now look at some concrete examples of using tags in AWS.

AWS Cost Explorer: Tracking costs thanks to tags

One of the most concrete benefits of tagging is the visibility it provides over your expenses. AWS Cost Explorer is the tool of choice for this.

Once your resources are correctly tagged (for example, with the Project tag), you need to activate these tags for cost allocation in the AWS Billing and Cost Management console (Billing and Cost Management -> Cost Organization -> Cost Allocation Tags). Be aware that there might be a delay before the activated tags appear in Cost Explorer.

Once activated, you can:

Filter by tag: In Cost Explorer, you can filter your costs by the value of a specific tag.
Group by tag: You can also choose to group your expenses by tag. This will give you an overview of the cost distribution across different projects, environments, etc.
Create budgets based on tags: With AWS Budgets, you can set alert thresholds for costs associated with specific tags, helping you avoid nasty surprises.

Example of filtering by tag for my blog: the costs are minimal, am I a FinOps expert?

This ability to dissect your AWS bill by tags transforms cost management from an obscure chore into a transparent and controllable exercise. It’s a must for any organization concerned about its Cloud budget. You can now add FinOps to your LinkedIn bio!

AWS Resource Explorer: Keeping an eye on deployed resources

Now that we are convinced of the usefulness of tags, how do we list all the resources that use a specific tag? We will see this together, with one part in the AWS console, and another part in the terminal (for all the geeks reading this article!).

Visual exploration in the AWS console

If you are not yet familiar with AWS Resource Explorer, now is the time to discover it! This relatively recent service allows you to search and discover your AWS resources across all regions of your account, using a simple interface, much like a search engine.

The main advantage of Resource Explorer is its ability to give you a unified view. No more jumping from region to region. You activate indexing, and then you can search for your resources by name, ID, and of course… by tag!

It’s an excellent tool for:

Getting a quick overview.
Visually exploring resources associated with a specific tag.
Quickly identifying resources without having to code anything.

To use it, activate it in the desired regions (or all of them), let it index your resources, and then use the search bar with a syntax like tag.key:Project tag.value:Cloud*.

Example of a tag search for my blog: only three resources manage this blog!

A more technical approach with AWS CLI

For those who, like me, like to have control via the command line, or who need to automate these searches, the AWS CLI remains a key ally. More specifically, it’s the resourcegroupstaggingapi service that will interest us.

The key command is get-resources. Here is a typical example to list the ARNs of all resources having the tag Project with the value “Cloud Antoine Delia”:

1
2
3

aws resourcegroupstaggingapi get-resources \
    --tag-filters "Key=Project,Values=Cloud Antoine Delia" \
    | jq "[.ResourceTagMappingList[].ResourceARN]"

Which gives us:

1
2
3

[
  "arn:aws:s3:::antoiXXXXXXXXXXXXX"
]

Let’s break it down a bit:

aws resourcegroupstaggingapi get-resources: This is the API call; so far, so good.
--tag-filters "Key=Project,Values=Cloud Antoine Delia": This is where we specify our filter. We are looking for the tag Project with the value “Cloud Antoine Delia”. You can add multiple filters.
| jq "[.ResourceTagMappingList[].ResourceARN]": jq is a wonderful tool for manipulating JSON on the command line. Here, we use it to cleanly extract the list of ARNs of the found resources (you can do without it, but why make life difficult?).

Hey, wait a minute! In the console, you show us three resources, and now there’s only one! Where’s the catch?

Well spotted! You should know that when you make your API call, you are using a default region. However, if you have resources in various regions, you will need to specify it. Thus, if we add --region us-east-1 just before the jq pipe, we indeed get our two missing resources.

[
  "arn:aws:acm:us-east-1:6XXXXXXXXXXX0:certificate/f4ca3b13-XXXXXXXXXXXXX",
  "arn:aws:cloudfront::6XXXXXXXXXXX0:distribution/ERSXXXXXXXXXX"
]

Besides this detail that you must not forget, this command is extremely powerful because you can integrate it into scripts to:

Generate regular reports on resources by project.
Automatically detect resources that do not comply with your tagging policy.
Combine with other AWS CLI commands to perform actions on the listed resources.

For example, you could list all your resources of a certain type (e.g., all your EC2 instances) and check which ones are missing essential tags.

And if you’re wondering if AWS doesn’t already offer a service for this… It does! But we will talk about that in a future article (for the curious, I’m talking about AWS Config).

AWS IAM: Securing the use of your resources

Your resources are deployed in AWS, and now you want to give a team permission to manage all of that. But here’s the thing, in your AWS account, you also have critical resources that must absolutely not be compromised.

AWS IAM is here for you! Using a simple policy, you can specify that only resources with a certain tag can be modified by a user or group.

Let’s take the following example: you would like to allow a team to start or stop certain EC2 instances, but prevent them from accidentally stopping a critical EC2 instance! You just need to add the following policy to your users:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowStartStopEC2IfProjectCloud",
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "arn:aws:ec2:*:*:instance/*",
      "Condition": {
        "StringEquals": {
          "aws:ResourceTag/Project": "Cloud Antoine Delia"
        }
      }
    },
    {
      "Sid": "AllowDescribeToSeeInstances",
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    }
  ]
}

Thus, your users will be autonomous in using their resources, without having the possibility to impact other resources.

Conclusion

You’ve understood it: a rigorous tagging strategy isn’t an option; it’s a necessity for operating serenely on AWS. Whether on the organizational, financial, or security management level, AWS gives you the means to take full advantage of your tags.

So, a little piece of advice: if you haven’t already, define a clear tagging policy in your organization, apply it, and use these tools to regularly check that everything is in order. You’ll thank me later!

uv: The Best Thing to Happen to Python - and Why You Should Use It

Mon, 26 May 2025 07:30:00 +0200

Introduction

Python is surely my favorite programming language. Its simplicity allows you to develop scripts, backends, or even websites in no time at all, without racking your brains.

But there has always been one thing I disliked about Python: its packaging tools. For a long time, I stayed away from all these tools like Poetry, because frankly, I didn’t understand a thing!

And then, I made a discovery that not only showed me that packaging Python code is a very simple task, but also completely changed all my habits.

I’m talking about uv.

Created by the same talented team at Astral (who had already delighted us with their tool ruff), uv is presented as an “extremely fast” Python package installer and resolver. This tool is what you might call a game-changer: once you’ve tried it, there’s no going back!

So, what’s so special about uv? Why all the excitement? That’s what we’re going to break down together. Hold on tight, you might just fall in love!

This guide is based on my experience with uv, and the examples were tested with version 0.7.3. Depending on your configuration, especially behind certain corporate proxies, you might need to add the --native-tls option to some uv commands if you encounter SSL connection issues.

uv, what is it exactly?

In a nutshell, uv is a command-line tool that aims to replace pip, pip-tools, venv, pyenv, and even parts of virtualenv and pipx, while being much, much faster. Yes, all of that!

Like many trendy new tools, it’s written in Rust. Sorry, I should say, it’s written in ✨ Rust ✨. This largely explains its lightning-fast performance.

Preamble

Before we dive headfirst into uv and how it works, I’d first like to explain how uv handles dependencies, virtual environments, and Python version installation. This was rather confusing for me the first time I encountered it, so I think it’s important to mention it upfront.

Dependency Management

uv bases its entire configuration on your pyproject.toml file. If you were used to using a requirements.txt file, for example, be aware that uv will completely ignore it.

Since I’m nice, here’s a gist of a pyproject.toml template that I use every time I start a new Python project.

pyproject.toml

[project]
name = "project-name"
version = "0.1.0"
description = "Project description"
readme = "README.md"
requires-python = ">=3.12"

[[tool.uv.index]]
name = "your-index"
url = "[https://my-artifactory.corp/api/pypi/my-virtual-repo](https://my-artifactory.corp/api/pypi/my-virtual-repo)"
publish-url = "[https://my-artifactory.corp/api/pypi/my-local-repo](https://my-artifactory.corp/api/pypi/my-local-repo)"
default = true

[tool.ruff]
# Exclude common directories that are typically not part of the source code or are generated by tools.
exclude = [
    ".bzr",
    ".cache",
    ".direnv",
    ".eggs",
    ".git",
    ".git-rewrite",
    ".hg",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pytype",
    ".ruff_cache",
    ".svn",
    ".tox",
    ".venv",
    "__pypackages__",
    "_build",
    "buck-out",
    "build",
    "dist",
    "node_modules",
    "venv",
]
 
# Set the maximum line length to 127 characters.
line-length = 127
 
# Define the number of spaces used for indentation, aligning with Black's style.
indent-width = 4
 
# The minimum Python version to target, e.g., when considering automatic code upgrades,
# like rewriting type annotations
target-version = "py312"
 
[tool.ruff.lint]
# Enable Pyflakes (F) and a subset of the pycodestyle (E) codes by default.
# pycodestyle warnings (W)
# Activate Security Rules (S) to replace bandit
# Enable the isort rules (I) to replace isort
# flake8-bugbear (B)
# flake8-simplify (SIM)
select = ["F", "E4", "E7", "E9", "W", "S", "I", "B", "SIM", "PGH004"]
ignore = [] # List any rules to be ignored, currently empty.
 
# Allow auto-fixing of all enabled rules when using the `--fix` option.
fixable = ["ALL"]
unfixable = [] # Specify rules that cannot be auto-fixed, if any.
 
# Define a regex pattern for allowed unused variables (typically underscore-prefixed).
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"
 
[tool.ruff.format]
# Enforce double quotes for strings, following Black's style.
quote-style = "double"
 
# Use spaces for indentation, in line with Black's formatting style.
indent-style = "space"
 
# Keep magic trailing commas, a feature of Black's formatting.
skip-magic-trailing-comma = false
 
# Automatically detect and use the appropriate line ending style.
line-ending = "auto"
 
# Update this with your package name or directory to be scanned by pytest-cov
[tool.pytest.ini_options]
addopts = "-s --no-header --cov=src --cov-fail-under=50" # force cmd flags
testpaths = [ # what directories contain tests
    "tests",
]
pythonpath = [ # what to add to the python path
    "."
]

Instead of adding your dependencies manually, you can now use uv add <package_name>, and it will be added to your pyproject.toml.

Be careful though, as some external tools still rely on a requirements.txt file. You might therefore need to run the command uv pip compile -o requirements.txt in your CI/CD pipeline.

Virtual Environment

Like me, you might have been used to using the command source .venv/bin/activate to activate your virtual environment. With uv, it’s a bit different. Your .venv will still be created, but by using uv, it will default to your virtual environment. Let’s take an example.

Where previously you would have had to do these kinds of commands to run your script:

python -m venv .venv
source .venv/bin/activate
pip install requests
python main.py

Now, uv removes virtual environment management from your workflow. Thus, the commands above will be replaced by these:

1
2

uv add requests
uv run main.py

When you add a new package to your project (with the uv add command), uv automatically takes care of creating a virtual environment if one doesn’t already exist.

Then, by running your script via uv run, uv will automatically place itself in your virtual environment.

Python Installation

You might have used pyenv to install your different Python versions. With uv, all that is a thing of the past!

When you want to start a project with a specific Python version, you have several ways to do it:

Use the .python-version file
Create a virtual environment: uv venv --python 3.11.6
Launch a specific Python version: uvx python@3.12

What you need to remember is that uv doesn’t rely on a globally installed Python version but will always try to use the version specific to your project.

Alright, I think you know enough to begin your initiation. Let’s go!

Installation

Installing uv is child’s play. You have several options, choose the one that suits you best (feel free to consult the official installation documentation if needed):

# On macOS and Linux - I recommend using this if possible
curl -LsSf [https://astral.sh/uv/install.sh](https://astral.sh/uv/install.sh) | sh

# On Windows (with PowerShell)
powershell -c "irm [https://astral.sh/uv/install.ps1](https://astral.sh/uv/install.ps1) | iex"

# With pip (if you already have a Python environment)
pip install uv

Once installed, you can update it very easily:

`1`	`uv self update`

And there you have it, uv is ready to use!

Managing Python Versions with uv

One of uv’s nice features is its ability to install specific Python versions. No more need to go through the official Python website, juggle with pyenv, or other tools.

To install the latest stable version of Python:

`1`	`uv python install`

Need a particular version? No problem:

1
2

# Install Python 3.9
uv python install 3.9

You can then use this version to run a script:

`1`	`uv run --python 3.9 python my_script.py`

As I explained a little earlier, Python versions installed by uv are not “globally” available on your system via the simple python command. To use them, you must go through uv run --python <version>, or activate them in a virtual environment created with that specific version.

uvx for Running Packages on the Fly

You might be familiar with npx in the JavaScript world? uvx (the shortcut for uv tool run) is the equivalent offered by uv. It allows you to run a Python CLI command (like ruff, black, ipython, etc.) in a temporary environment with the specified dependencies, without polluting your project or your system.

For example, to run a specific version of ruff:

`1`	`uvx --python 3.12 ruff@0.9.6 check main.py`

Or to run ipython with requests temporarily available:

`1`	`uvx --with requests -p 3.13 ipython`

Extremely practical for one-shots or for testing tools!

Project Management with uv

Alright, all these commands are very nice, but I imagine you’re wondering how to integrate uv into a new project, or an existing one.

So let me show you the full power of uv for creating and managing a Python project.

Starting a New Project

To initialize a new project with uv:

uv init

This command will create a few files for you:

.python-version: Specifies the Python version to use for this project (e.g., 3.11).
main.py: An example Python file.
pyproject.toml: The central file for your project’s configuration, including its dependencies. This is the modern standard in Python.

Now, let’s see how to manage our dependencies.

With uv, pyproject.toml becomes your source of truth for dependencies.

To add a new dependency to your project:

`1`	`uv add requests`

To add a development dependency (only for dev, like ruff or pytest):

1
2

uv add ruff --dev
uv add pytest --dev

⚠️ Very important: The first time you add a dependency, uv will generate a uv.lock file. This file contains the exact versions of all your dependencies (direct and indirect) that were resolved. This uv.lock file is crucial and MUST be committed to your GitHub repository. It ensures that the versions of your packages are the same for everyone, according to your working environment.

Once your dependencies (especially dev dependencies) are added, you can use them with uv run:

1
2

uv run ruff format .
uv run pytest

To remove a package:

# For a normal dependency
uv remove requests

# For a development dependency (note the --group dev)
uv remove ruff --group dev

In case you are using uv in an enterprise with a private index, you can configure that too!

By default, uv uses PyPI. If you work in a corporate environment (with an Artifactory or another private index), you can configure uv to use it via the pyproject.toml file.

[[tool.uv.index]]
name = "your-index"
url = "[https://my-artifactory.corp/api/pypi/my-virtual-repo](https://my-artifactory.corp/api/pypi/my-virtual-repo)"
publish-url = "[https://my-artifactory.corp/api/pypi/my-local-repo](https://my-artifactory.corp/api/pypi/my-local-repo)"
default = true

Then, you can, for example, export environment variables to authenticate.

# Environment variables must follow this format: UV_INDEX_{name}_USERNAME
# where {name} is the index name you defined in your pyproject.toml file
export UV_INDEX_YOUR_INDEX_USERNAME="delia, antoine"
export UV_INDEX_YOUR_INDEX_PASSWORD="cmVmdG**************************FJUjNw"

Feel free to consult the uv documentation on indexes for more details.

Migrating an Existing Project to uv

Do you have a project with a good old requirements.txt? Migration is quite simple:

If you don’t have a pyproject.toml, create one (feel free to use the one provided above).
Run the following commands to add your requirements to your pyproject.toml: uv add -r requirements.txt and uv add --dev -r requirements-dev.txt.
Once all dependencies are migrated, you can delete your old requirements files.

If you were using other tools like Poetry, migration tools exist (such as: uvx migrate-to-uv).

And there you have it, your project is powered by uv!

Joining a Project That Already Uses uv

If you clone a project that is already managed by uv (so it should have a pyproject.toml and a uv.lock), setting it up is incredibly simple. You just need to run:

uv sync

This magic command will read the uv.lock file and install exactly the same versions of all the packages listed there. Isn’t that beautiful?

Build and Publish Your Project

uv doesn’t stop there and also offers commands for building and publishing.

To build your package:

`1`	`uv build`

To publish to PyPI (or a private index):

1
2
3

uv publish
# For a private index, you might need to specify the URL (unless already specified in your pyproject.toml)
# uv publish --repository-url [https://my-artifactory.corp/api/pypi/my-local-repo](https://my-artifactory.corp/api/pypi/my-local-repo)

And to quickly test if your freshly built package installs and imports correctly:

1
2

# Replace <MY_PACKAGE> with your package name
uv run --with <MY_PACKAGE>-<VERSION>.whl --no-project --python -c "import <MY_PACKAGE>"

And there you go! You’ve published your package in the blink of an eye!

Cleaning Your Cache

Over time, uv (just like pip) accumulates a cache of downloaded packages. While this allows for quick installation of your dependencies across multiple projects, in the long run, it could take up quite a bit of space on your computer. To clean it, simply run the following command:

`1`	`uv cache clean`

Conclusion

As you’ve understood, uv isn’t just “another package manager.” It’s a real breath of fresh air in the Python ecosystem.

Speed: This is the first thing that shocks you when you use it. Installations, resolutions, everything is incredibly faster than pip. On large projects, the time savings are phenomenal.
Unification: uv brings together functionalities that previously required multiple tools (pip, venv, pip-tools, or even pyenv for basic needs). Having a single, coherent interface greatly simplifies the workflow.
Compliance: The uv developers base all their choices on Python guidelines (those famous PEPs). You can therefore be sure that your pyproject.toml respects modern Python packaging standards.
Reliability: The lockfile system (uv.lock) ensures reproducible builds, an essential point for teamwork and continuous integration.

Since I started using uv, my interactions with Python dependency management have become faster, simpler, and more enjoyable. It’s the kind of tool that, once adopted, makes you wonder how you ever managed before.

So, if you’re looking to modernize your Python usage and gain productivity (and peace of mind!), I can only encourage you to give uv a try. Trying it is very often adopting it!

References

Official uv documentation: https://docs.astral.sh/uv
uv GitHub repo: https://github.com/astral-sh/uv
A year of uv: pros, cons, and should you switch? (bitecode.dev)
uv tricks (bitecode.dev)

Bonus: uv cheatsheet

Here’s a list of super handy uv commands. A cheatsheet, if you will. Feel free to come back and consult it as needed!

# Update uv
uv self update

# Run a script with a specific Python version
uv run --python 3.12.3 main.py

# Quickly run a tool with a specific version
uvx --python 3.12 ruff@0.9.6 check

# Quickly run ipython with requests installed
uvx --with requests -p 3.13 ipython

# Update your project's version
uv version --bump [major/minor/patch]

# Clean the cache
uv cache clean

AWS Lambda: INIT Phase To Be Billed Soon - Should We Be Alarmed?

Tue, 13 May 2025 07:30:00 +0200

Introduction

Ah, Lambdas. Probably my favorite AWS service! Almost instantly, they allow you to create a small script or even a fully hosted API backend in the Cloud, all at a low cost thanks to the “pay-as-you-go” model. But is it too good to be true?

A few days ago, AWS announced a change in how the Lambda initialization phase is billed. Starting August 1, 2025, this phase will systematically be included in the billed duration calculation, for all Lambdas.

If you weren’t aware, until now, if you used Lambda functions packaged as ZIP archives with AWS-managed runtimes (like Python, Node.js, etc.), the duration of this INIT phase wasn’t billed. It was a small “gift” from AWS (quite nice of them). But all good things must come to an end, and we will now have to pay for this initialization time for our Lambdas.

So, will this make using Lambdas too expensive? And how can we minimize this initialization part as much as possible? Today, let’s explore these questions!

Understanding the Lambda Lifecycle

Before diving into billing, it’s a good opportunity to recall the Lambda lifecycle.

For simplicity, I won’t go into the details of Lambda Extensions and Lambda SnapStart.

It consists of three main phases:

INIT: This is the “cold start” phase. When a new instance of your function needs to be created to handle a request, Lambda prepares the environment. This phase lasts a maximum of 10 seconds.
INVOKE: This is where your code (the function handler) is executed to process the request.
SHUTDOWN: When the execution environment hasn’t been used for a certain period, the Lambda shuts down to free up resources. If a new request arrives, the Lambda will have to go through the INIT phase again.

During the INIT phase, your Lambda performs several actions:

Retrieves your code (from S3 for a ZIP, or ECR for a Docker image).
Configures the environment with the allocated memory, chosen runtime, etc.
Starts the runtime (Runtime INIT).
Executes the static code of your function (everything outside the handler, like initializing global variables or boto3) (Function INIT).

The key takeaway here is that the INIT phase only occurs during a cold start. If a request arrives while an execution environment is already “warm” (ready and reused), this phase is skipped, and execution goes directly to the INVOKE phase. This is called a “warm start,” which is much faster.

AWS doesn’t disclose its logic for transitioning a Lambda from ‘warm’ to ‘cold’.

The Billing Change in Detail

Currently, Lambda billing is based on two factors:

The number of requests.
The execution duration of your code, rounded up to the nearest millisecond (the cost of this duration depends on the memory allocated to the function).

Until August 1, 2025, for functions packaged as ZIPs with managed runtimes, the INIT phase duration was not counted in the “Billed Duration”. This could be seen in CloudWatch logs:

Before August 1, 2025
Note that Billed Duration is the rounded-up value
of Duration without considering Init Duration
REPORT RequestId: xxxxx   Duration: 250.06 ms   Billed Duration: 251 ms   Init Duration: 100.77 ms

After August 1, 2025
Billed Duration is now the rounded-up value
of Duration + Init Duration
REPORT RequestId: xxxxx   Duration: 250.06 ms   Billed Duration: 351 ms   Init Duration: 100.77 ms

As you can see, AWS will now take your Init Duration into account in addition to the Duration when calculating the Billed Duration.

What Impact on the Bill?

So, this all sounds interesting, but will this change break the bank?

Let’s take an example to see more clearly. Imagine a Python Lambda configured with 1024 MB of memory, deployed in the eu-west-1 (Ireland) region.

Assume the following:

The Lambda receives 10 million invocations per month.
The cold start rate is 1% (average provided by AWS), meaning 100,000 cold starts per month.
The average invocation duration (Duration) is 3 seconds.
The average initialization duration (Init Duration) is 1 second.
Cost per request: $0.20 per million requests.
Duration cost (x86): $0.0000166667 per GB-second.

Details of the calculation

Cost Calculation BEFORE August 1, 2025:

Request Cost: 10 million req * ($0.20 / 1 million req) = $2.00
Duration Cost:
- Warm starts (9.9 million): Billed duration = 3 seconds.
- Cold starts (100,000): Billed duration = 3 seconds (INIT not billed).
- Total billed duration (seconds): (9,900,000 * 3 s) + (100,000 * 3 s) = 29,700,000 seconds + 300,000 seconds = 30,000,000 seconds.
- Total GB-seconds: 30,000,000 seconds * (1024 MB / 1024 MB) = 30,000,000 GB-s.
- Duration Cost: 30,000,000 GB-s * $0.0000166667/GB-s = $500.00
Total Monthly Cost (Before): $2.00 + $500.00 = $502.00

Cost Calculation AFTER August 1, 2025:

Request Cost: $2.00 (unchanged)
Duration Cost:
- Warm starts (9.9 million): Billed duration = 3 seconds.
- Cold starts (100,000): Billed duration = 3 seconds (Duration) + 1 second (Init Duration) = 4 seconds.
- Total billed duration (seconds): (9,900,000 * 3 s) + (100,000 * 4 s) = 29,700,000 seconds + 400,000 seconds = 30,100,000 seconds.
- Total GB-seconds: 30,100,000 seconds * (1024 MB / 1024 MB) = 30,100,000 GB-s.
- Duration Cost: 30,100,000 GB-s * $0.0000166667/GB-s = $501.67
Total Monthly Cost (After): $2.00 + $501.67 = $503.67

In this specific scenario, the increase is only $1.67 per month. This is indeed minimal and confirms AWS’s communication. But be careful, as the actual impact will strongly depend on:

The memory allocated to your Lambdas (more memory = higher cost per ms).
The actual duration of your INIT phase.
Your cold start rate (can be higher if your traffic is irregular).

Therefore, it’s wise to check with your own numbers!

Below is a small script that will allow you to quickly test this on your side.

Python Script

def calculate_lambda_costs(
    total_invocations_per_month: int,
    cold_start_rate: float,
    average_invocation_duration_sec: float,
    average_init_duration_sec: float,
    allocated_memory_mb: float, # Changed GB to MB for consistency with common AWS console values
    cost_per_million_requests: float = 0.20,
    cost_per_gb_second: float = 0.0000166667,
):
    num_cold_starts = total_invocations_per_month * cold_start_rate
    num_warm_starts = total_invocations_per_month - num_cold_starts

    request_cost = (total_invocations_per_month / 1_000_000) * cost_per_million_requests

    allocated_memory_gb = allocated_memory_mb / 1024 # Convert MB to GB for calculation

    # Duration cost BEFORE
    billed_duration_warm_starts_sec_before = num_warm_starts * average_invocation_duration_sec
    # For cold starts, only invocation duration was billed
    billed_duration_cold_starts_sec_before = num_cold_starts * average_invocation_duration_sec

    total_billed_duration_sec_before = billed_duration_warm_starts_sec_before + billed_duration_cold_starts_sec_before
    total_gb_seconds_before = total_billed_duration_sec_before * allocated_memory_gb
    duration_cost_before = total_gb_seconds_before * cost_per_gb_second

    total_monthly_cost_before = request_cost + duration_cost_before

    # Duration cost AFTER
    billed_duration_warm_starts_sec_after = num_warm_starts * average_invocation_duration_sec
     # For cold starts, invocation + init duration will be billed
    billed_duration_cold_starts_sec_after = num_cold_starts * (average_invocation_duration_sec + average_init_duration_sec)

    total_billed_duration_sec_after = billed_duration_warm_starts_sec_after + billed_duration_cold_starts_sec_after
    total_gb_seconds_after = total_billed_duration_sec_after * allocated_memory_gb
    duration_cost_after = total_gb_seconds_after * cost_per_gb_second

    total_monthly_cost_after = request_cost + duration_cost_after

    return total_monthly_cost_before, total_monthly_cost_after


def display_results(cost_before, cost_after):
    cost_difference = cost_after - cost_before
    print(f"\nEstimated monthly cost BEFORE change: ${cost_before:.2f}")
    print(f"Estimated monthly cost AFTER change: ${cost_after:.2f}")
    print(f"Estimated monthly cost increase: ${cost_difference:.2f}")
    if cost_difference > 0 and cost_before > 0:
        percentage_increase = (cost_difference / cost_before) * 100
        print(f"Percentage increase: {percentage_increase:.2f}%")
    elif cost_difference > 0:
        print("Percentage increase: Infinite (original cost was zero)")


if __name__ == "__main__":
    print("AWS Lambda Cost Calculator (Before/After Init Duration Pricing Change)")
    print("Please enter the values for your scenario or press Enter to use default values.\n")
    default_invocations = 10_000_000
    default_cold_start_rate = 0.01  # 1%
    default_invocation_duration = 3.0  # seconds
    default_init_duration = 1.0  # seconds
    default_memory_mb = 1024.0 # 1024 MB

    try:
        invocations_str = input(f"Total invocations per month (default: {default_invocations:,}): ")
        total_invocations_input = int(invocations_str) if invocations_str else default_invocations

        cold_start_rate_str = input(f"Cold start rate (e.g., 0.01 for 1%, default: {default_cold_start_rate}): ")
        cold_start_rate_input = float(cold_start_rate_str) if cold_start_rate_str else default_cold_start_rate

        invocation_duration_str = input(f"Average invocation duration in seconds (default: {default_invocation_duration}): ")
        invocation_duration_input = float(invocation_duration_str) if invocation_duration_str else default_invocation_duration

        init_duration_str = input(f"Average initialization duration in seconds (default: {default_init_duration}): ")
        init_duration_input = float(init_duration_str) if init_duration_str else default_init_duration

        memory_mb_str = input(f"Memory allocated to Lambda in MB (e.g., 512, default: {default_memory_mb}): ")
        memory_mb_input = float(memory_mb_str) if memory_mb_str else default_memory_mb

        cost_before, cost_after = calculate_lambda_costs(
            total_invocations_per_month=total_invocations_input,
            cold_start_rate=cold_start_rate_input,
            average_invocation_duration_sec=invocation_duration_input,
            average_init_duration_sec=init_duration_input,
            allocated_memory_mb=memory_mb_input,
        )

        display_results(cost_before, cost_after)

    except ValueError:
        print("\nError: Please enter valid numbers.")
    except Exception as e:
        print(f"\nAn unexpected error occurred: {e}")

How to Monitor Your INIT Phase and Estimate the Impact?

A script is nice, but if you have a bunch of Lambdas to check, you might find the task exhausting.

Fortunately, AWS provides some tools to check this efficiently. As seen earlier, each Lambda will generate a REPORT log containing its Init Duration. We can easily aggregate this information using CloudWatch Logs Insights. And the icing on the cake, AWS even provides us with the right query.

filter @type = "REPORT" and @billedDuration < ceil(@duration + @initDuration)
| stats sum((@memorySize/1024/1000) * (@billedDuration/1000)) as BilledGBs,
        sum((@memorySize/1024/1000) * ((ceil(@duration + @initDuration) - @billedDuration)/1000)) as EstimatedInitGBs,
        (EstimatedInitGBs / (EstimatedInitGBs + BilledGBs)) * 100 as EstimatedInitCostPercent

Keep in mind that CloudWatch Logs Insights is billed at $0.005 per GB of data scanned (source). Adjust the query time range to limit costs.

Just run this query on your Lambda log groups (using the /aws/lambda prefix or specific function prefixes), and you’ll get a result like this:

BilledGBs	UnbilledInitGBs	Ratio
512.8007	86.6699	0.1446

This query gives you three key pieces of information:

BilledGBs: The total GB-seconds currently billed.
EstimatedInitGBs: The total GB-seconds consumed during the INIT phase that were not previously billed (this is an estimation based on the difference between current billed duration and potential future billed duration including INIT).
EstimatedInitCostPercent: The percentage that these previously unbilled INIT GB-seconds represent compared to the total consumed GB-seconds (current billed + estimated init). This approximates the potential percentage increase in your duration costs due to the change.

In our example, the cost share of the INIT phase represents approximately 14% of the total execution cost (current + potential init cost)! Depending on your current bill, this could be significant.

Understanding and Optimizing Your Lambda

Okay, so you know you might have to pay more. But aren’t there levers we can pull to reduce this increase in your bill?

Here are some tips for making the best use of this INIT phase and thus reducing your future costs.

Strategically Use the INIT Phase

Your first instinct might be to remove all the INIT code (code outside the handler) and put it in your handler. That way, no more INIT billing! But not only would you just be shifting the problem (because executing that code will still be billed), it would be even worse, because now even your ‘warm start’ Lambda executions would have to process this code!

Yes, remember that the code in the INIT phase is executed only during cold starts. Therefore, it’s the ideal place for expensive initialization operations that can be reused by subsequent (“warm start”) invocations. This is even recommended by AWS. Thus, the INIT phase is perfect for:

Importing heavy libraries or dependencies.
Establishing connections to other AWS services (S3, DynamoDB, etc.) via SDKs (boto3 for Python).
Creating database connection pools.
Fetching parameters or secrets from Systems Manager Parameter Store or Secrets Manager.

Maximize this initialization phase to reduce the execution time of Lambdas running in ‘warm start’.

Optimize Package Size

This is often the simplest lever, because sometimes we import all sorts of things into our Lambda. So, make sure to include only strictly necessary dependencies. Also, be sure to exclude anything related to your development environment (I can’t tell you how many times I’ve found a node_modules or tests folder in Lambdas…). Finally, don’t hesitate to consult AWS articles, as some directly tackle the subject of cold starts.

Lambda SnapStart

Available for Java, .NET, and Python runtimes, SnapStart is a very interesting feature for combating cold starts. When you enable it, Lambda takes a “snapshot” of the initialized execution environment after the first INIT phase. For subsequent cold starts, Lambda restores this snapshot instead of redoing the entire INIT phase, saving you significant time.

I imagine reading about this feature, you’re thinking “Wow, this is great, I’m going to enable this feature on all my Lambdas!”. Except there are a few subtleties to be aware of.

Firstly, SnapStart is currently only compatible with specific versions of certain languages (Python 3.12 minimum, NodeJS not supported…). The same goes for regions; only a limited number support this feature at the time of writing. Finally, this feature is not free and incurs its own costs. Make sure to check the potential cost if you want to enable it on your Lambdas.

Provisioned Concurrency

If your application has predictable traffic or if cold start latency is unacceptable, you can use “Provisioned Concurrency”. You ask Lambda to keep a certain number of execution environments pre-initialized (warm) at all times.

Advantage: Requests hitting these provisioned instances never experience a cold start. The INIT phase is done upfront, before the first request.

Disadvantage: You pay for the time these environments are provisioned, whether they receive requests or not. The INIT phase is also billed during this pre-initialization.

Here too, it’s up to you to see if it’s worth enabling this on some of your key Lambdas!

Conclusion

The upcoming change in billing for Lambda’s INIT phase on August 1, 2025, is primarily a standardization. For the majority of Lambdas (specifically those using managed runtimes and ZIP packaging that previously benefited from the free init), this means the duration of this phase will now be added to the billed duration during cold starts.

Even if the financial impact will likely be small for you, it’s an excellent opportunity to:

Understand your Lambda’s lifecycle and what happens during initialization.
Measure the INIT phase duration of your critical functions using CloudWatch tools.
Optimize this phase if necessary, by reducing package sizes, using SnapStart, or considering Provisioned Concurrency for specific use cases.

So, don’t wait until August! Familiarize yourself with CloudWatch Logs Insights now. This will allow you to identify which Lambdas to optimize first and avoid any nasty surprises on your AWS bill.

How I Automated My Blog To The Cloud

Tue, 04 Mar 2025 07:30:00 +0100

How it started and why it ended

When I started blogging, I was still in engineering school. And the world of websites and blogs was still new to me. So when I wanted to create a personal blog, I turned to what I thought was the best solution at the time: WordPress.

Like it or not, WordPress is quite easy to use for beginners, even more for me who was still discovering how to host a website at the time.

So, with a brand new OVH account, I was able to quickly setup my personal blog in a few clicks!

And when the time came where I wanted to create a blog dedicated to professional topics, I naturally came back to my ol’ friend WordPress.

But after some months, the glory of WordPress started to fade into a darker view. Sure, it was convenient to add a new post by going through WordPress’ UI, but maintaining the website was always boresome. Moreover, a WordPress installation requires using a database, which incurred a higher bill at the end of each month.

All of this for a blog that I knew did not receive many visits each day.

Surely there was a better way to do this.

The start of a new plan

I will tell you here the path that led me to the rebirth of my blog on which you are today. If you are only interested in the technical details, you can move on to the next section. Otherwise, happy reading!

Discovering Hugo

Before searching for a new tool, I tried to think of what I needed.

My blog was essentially a simple webpage, a static website with no fancy JavaScript or stuff like that. Plus, there was no backend to handle API calls for user authentication for example, so I could simply not use any database at all.

So what I needed was essentially a way to focus on the content of the blog, and make sure I could attach to it a nice theme, and that was it!

But, as I’m an engineer, so in essence, a real bad designer, I wanted to avoid having to deal with CSS as much as possible. Again, I wanted to focus on the content, rather than the looks. So I searched and searched for the perfect tool. And I didn’t had to look too long to find it: Hugo.

Upon arriving on its website, I was greeted by an intriguing message message: The world’s fastest framework for building websites. Surely, I was curious, and dug into the documentation. And oh boy, was I not disappointed.

This was exactly what I wanted. A framework where I had to write in markdown format, that would be converted to HTML for me. Plus, it came with a bunch of themes, so I didn’t even had to bother about the design. I could just pick one that I liked, and move on.

So, without losing any more time, I started to convert my WordPress blog to a Hugo site. I came across a handy script called wordpress-to-hugo-exporter that converted my whole Wordpress database into compatible markdown syntax, and I was ready to go!

Now I needed to host this website somewhere.

A new challenger: AWS

In parallel, I had an opportunity working for a company that focused on the Cloud, more specifically AWS.

At the time, Cloud was brand new to me. But I was hearing more and more about it, and I wanted to check what was all the fuss around it. Would this be a true life-changer, as people would call it, or would it be yet again another buzz word, and become a dying trend?

Long story short, AWS was (and still is) amazing! Not only was it an amazing discovery, I could simply not see a world without Cloud anymore.

The more I learned about all these amazing AWS services, the more I thought: isn’t there a way for me to leverage the Cloud to host my blog?

With that in mind, I started looking for options that AWS provided. And because of my recent discovery of Hugo, my website did not needed PHP anymore, nor any database to work. It was now a simple, static website, which could perfectly fit in an S3 bucket.

Not only does AWS has a documentation about hosting a static website on an S3 bucket, but the pricing for it was just a few cents.

I had found an easy and cost-effective way to host my website.

The last point that was bothering me, is that to setup all of this, I had to create an S3 bucket, configure it, add a CDN with AWS CloudFront, take into account certificates with AWS ACM, and finally create a DNS record in AWS Route53. So many actions that, if done manually, could make it super difficult to reproduce this setup if I had to do it again.

I needed a robust way to configure this infrastructure.

Terraform to the rescue

When you start learning Cloud, you will usually hear about Infrastructure as Code.

The idea is simple: describe your Cloud infrastructure with… code! (yes, as the name stands by).

The benefits of using an IaC tool, is that you will be able to keep track of the changes done on your infrastructure (like you would do with code in GitHub).

Now, let’s say that someone wants to update the configuration of an AWS service. This person goes to the AWS console and start making their changes. But after some trial and error, they decide to give up for now, and to go back to the initial state. And here is the problem: they don’t know exactly how it was before! And unless they took note of the initial state before making their changes, they will, again, have to do a bunch of trial and errors before going back to the beginning.

All of that could have been prevented if the infrastructure was configured through an IaC tool to begin with. In such a case, a simple automated deploy could have done the trick!

There are a bunch of IaC tools out there. But how can I not talk about the most popular one, the one that made IaC a standard in the Cloud ecosystem: Terraform.

Terraform, as you could have guessed, is an Infrastructure as Code tool with a declarative approach, meaning that you have to declare the state of the infrastructure you wish to deploy. To do that, you need ton use a specific language: hcl (which stands for HashiCorp Configuration Language). Here is an example to create an S3 bucket with Terraform.

resource "aws_s3_bucket" "example" {
  bucket = "my-tf-test-bucket"

  tags = {
    Name        = "My bucket"
    Environment = "Dev"
  }
}

Let’s go back to my blog! Terraform seems to be the perfect tool to configure the infrastructure that will host my blog. All I had to do was to go through an architecture step to know which AWS services I needed, and how to interconnect them, before translating that to Terraform.

And here is the architecture diagram I came up with.

All of this was perfect, but I realized one thing. Every time I needed to add a post, or update the infrastructure, I had to clone my repository, make the appropriate changes, and do a lot of manual actions to update my blog.

Let’s say that I was not too pleased with that, and I intended to fix that!

Forget copy and paste with GitHub Actions

Like I said before, my website was ready. The last step I needed was about the deployment.

For now, when I had to create a new post, I had to manually build my Hugo website, and move the generated files to my S3 bucket. What a waste of time and energy!

So I explored the topic of CI/CD pipelines to make my life easy (if you’re interested in this topic, I suggest reading my post about setting up a CI/CD strategy). While Jenkins tried to get my attention, I decided to go with the obvious choice: GitHub Actions.

GitHub Actions are simple CI/CD pipelines that you can define directly in your GitHub repository. The main benefit, is that you do not need any additional account, nor to install anything: all is included! In addition, GitHub Actions use an easy-to-read YAML syntax (in opposition to Jenkins and its horrifying groovy!).

So I created a pipeline that was able to configure Terraform in case I made any modifications, and could also build and deploy my website automatically in my S3 bucket when I updated or added a post.

All the steps were looking good, mission accomplished!

Technically: What I Have Today

My blog is now fully automated and deploys to AWS!

If you want to dive into it, everything is publicly available on my GitHub repository: https://github.com/antoinedelia/cloud-optimist

Now, let me show you more details about it.

Project Structure

The project structure is as follow:

cloud-optimist/
├── .github/
│   └── workflows/
│       └── main.yml  # GitHub Actions
├── cloud/
│   └── ...  # My Hugo blog
├── terraform/
│   └── ...  # The Terraform configuration
└── README.md

As you can see, I split my project into three key directories:

.github : contains the main.yml file which defines the GitHub Actions CI/CD steps
cloud : contains my Hugo blog
terraform : contains all the Terraform infrastructure

The GitHub Actions

My GitHub Actions is rather simple, and can be divided into multiple steps.

The Initialization Step

Let’s have a look at the top of the file.

name: 'Build and Deploy'

on:
  push:
    branches:
    - master
  pull_request:

jobs:
  build-and-deploy:
    name: 'Build & Deploy'
    runs-on: ubuntu-latest
    environment: production

    defaults:
      run:
        shell: bash
        working-directory: cloud

Here, I ask GitHub to run the pipeline on the master branch, or if I have a Pull Request. Let me reassure you, I do not deploy anything to production with a Pull Request. You’ll see below that this only allows me to perform some “dry-run” checks and ensure no issues are found before I can safely deploy to production via the master branch.

I also ask GitHub to use bash and to use the cloud directory as the default working directory.

Let’s see what’s next:

    steps:
    - uses: actions/checkout@v3
    - uses: dorny/paths-filter@v2
      id: filter
      with:
        filters: |
          web:
            - 'cloud/**'
          terraform:
            - 'terraform/**'

This step is crucial if you wish to fasten your CI/CD and ultimately save cost.

I’m using the dorny/paths-filter actions which allows me to detect which files were modified in the last commit. In my case, I look for the cloud and terraform directories. This way, if no changes were detected on the Terraform side, no need to trigger the Terraform steps in my pipeline. In the same way, if the directory cloud is untouched, no need to try to build and deploy the contents of my blog. This will save you few cents related to the data transfer pricing to your AWS S3 bucket (no need to thanks me!).

The next steps speak for themselves.

    # Checkout the repository to the GitHub Actions runner
    - name: Checkout
      uses: actions/checkout@v2
      
    - name: Update Git Submodules
      working-directory: ./
      run: git submodule update --init --recursive

I simply get the content of my repository and update the submodules. This last step was useful back when I was using submodules to handle my Hugo themes. I now use the Hugo Modules, which could let me remove this step altogether.

The Terraform Step

Let’s continue with Terraform:

    # Install the latest version of Terraform CLI and configure the Terraform CLI configuration file with a Terraform Cloud user API token
    - name: Setup Terraform
      if: steps.filter.outputs.terraform == 'true'
      uses: hashicorp/setup-terraform@v1
      with:
        cli_config_credentials_token: ${{ secrets.TF_API_TOKEN }}

    # Initialize a new or existing Terraform working directory by creating initial files, loading any remote state, downloading modules, etc.
    - name: Terraform Init
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform init

    # Checks that all Terraform configuration files adhere to a canonical format
    - name: Terraform Format
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform fmt -check

    # Generates an execution plan for Terraform
    - name: Terraform Plan
      if: steps.filter.outputs.terraform == 'true'
      working-directory: ./terraform
      run: terraform plan

      # On push to master, build or change infrastructure according to Terraform configuration files
    - name: Terraform Apply
      working-directory: ./terraform
      if: steps.filter.outputs.terraform == 'true' && github.ref == 'refs/heads/master' && github.event_name == 'push'
      run: terraform apply -auto-approve

Now that’s a long step! Let’s break it down.

The first thing that should’ve catch your attention is this line:

`1`	`if: steps.filter.outputs.terraform == 'true'`

Do you remember? This is what allows us to tell if files were updated in the terraform directory. Here, I check if I need to run these steps or not.

Then, I am running some ordinary Terraform commands: terraform init to initialize my project, terraform fmt -check to ensure my code is fancy, and finally terraform plan to look at the changes planned.

Last but not least, terraform apply -auto-approve, which will deploy the changes to my AWS infrastructure. But if you look closely, you will notice I added two extra checks:

`1`	`github.ref == 'refs/heads/master' && github.event_name == 'push'`

This ensures that this step will only be ran if the event is a push on the master branch. This way, no need to worry about an unwanted deployment when I simply play with my infrastructure on another branch or in a Pull Request. Phew!

The Hugo and AWS Steps

Now that my infrastructure is ready, it is time to publish my blog!

    - name: Build
      uses: actions/setup-node@v2
      if: steps.filter.outputs.web == 'true'
    - run: sudo wget https://github.com/gohugoio/hugo/releases/download/v0.142.0/hugo_extended_0.142.0_linux-amd64.deb -O hugo.deb
    - run: sudo dpkg --install ./hugo.deb
    - run: hugo

    - name: Deploy to AWS
      uses: jakejarvis/s3-sync-action@master
      if: steps.filter.outputs.web == 'true' && github.ref == 'refs/heads/master' && github.event_name == 'push'
      with:
        args: --acl public-read --follow-symlinks --delete
      env:
        AWS_S3_BUCKET: ${{ secrets.AWS_S3_BUCKET }}
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        AWS_REGION: 'eu-west-1'
        SOURCE_DIR: 'cloud/public/'

As for Terraform, we check if any modifications were done on the blog.

`1`	`if: steps.filter.outputs.web == 'true'`

Then, I’m retrieving an extended version of Hugo straight from the GitHub release (here the v0.142.0). I’m installing it and build my blog using the hugo command.

This step could be simplified using the peaceiris/actions-hugo actions

Finally, I’m using the jakejarvis/s3-sync-action actions (I realized while writing this post that this action has been archived a couple of days before, ouch! – I’ll add an edit later to mention an alternative) to move my blog’s files into my S3 bucket.

Of course, you’ll have to store some credentials in your GitHub repository so that your GitHub Actions can be authorized to use AWS, but nothing too crazy.

We are now done with the GitHub Actions part!

The Terraform part

Regarding Terraform, I’ll keep it short, as this is not too complex.

There is one personal choice here, is that I prefer to split my .tf files based on the AWS services used, rather than putting them all in a single main.tf file, which could make it hard to read. In the end, I have an s3.tf file for all resources linked to the S3 service, a cloudfront.tf file for everything related to CloudFront, and so on.

One important information, is that it is necessary to define at least the us-east-1 region, as you must deploy your ACM certificates there. For the rest, I’m deploying all my resources in the eu-west-1 region. To make this split, I’m using the Terraform alias. Here’s an example:

# The default configuration: all resources starting with `aws_` will use this provider
provider "aws" {
  region = "eu-west-1"
}

provider "aws" {
  region = "us-east-1"
  alias  = "us_east_1"
}

resource "aws_s3_bucket" "site" {
  bucket        = var.bucket_name
  force_destroy = true
}

resource "aws_acm_certificate" "cert" {
  provider                  = aws.us_east_1  # Will be deployed in us-east-1
  domain_name               = var.domain_name
  subject_alternative_names = ["*.${var.domain_name}"]
  validation_method         = "DNS"
}

The Hugo part

My Hugo blog is based on the Stack theme. There is also a GitHub template available which allows you to quickly setup your own blog based on this theme.

Not a lot to say about this part, except that I encourage you to have a look at the Hugo Modules, which will save you from having to deal with submodules (and will make it easier when you need to update your theme).

Potential Improvements

While I’m extremely satisfied with the end results, I noted a few points that could be improved in the future.

First off, I saw that Hugo has a deploy command that could, well, deploy your blog directly to an S3 bucket. Perfect! So I need to learn a bit more about Hugo Deploy.

Finally, even though using AWS as my hosting platform is not expensive (less than 2$ a month), it is not free. But it would be completely possible to use GitHub Pages to make this blog accessible, while taking the advantage of GitHub as the hosting platform (on this point, I’m a bit reluctant, as then I would not be able to show-off with my beautifully made AWS diagrams…).

Conclusion

Here we are, you know everything!

From my beginnings with WordPress, which, while being useful, showed some weaknesses on the backup, maintainability and deployment part.

Of my research to find the perfect technical suite: AWS, Terraform, Hugo and GitHub Actions, all of that to enable a speedy deployment at almost no cost!

When I look back to my previous setup and compare to what I have today, I am so happy to have done the switch!

Now, it’s up to you! You have all the information to create your own blog and deploy all of this with a breeze!

Happy blogging !

How We Built a CI/CD Strategy That Onboards 100+ Python Projects in Under a Minute

Sun, 19 Jan 2025 16:45:00 +0200

We all know CI/CD is important. In fact, it seems impossible to imagine a world where we would ship a project without checking the quality of its code, and having a detailed test suite. Moreover, to enable developers to focus on development, all of this should be automated.

Well, things are not always so easy.

Today, I’ll share how we transitioned from having no CI/CD strategy to onboarding 100+ Python projects in under a minute.

The Real World

When I joined my current company, I noticed we were managing a large number of Python projects. But when I tried to check for the CI/CD of these, well it was a bit of a mess. Some projects had CI/CD in place, but not too many. And the ones with one were not using the same rules to verify code compliance. It was clear that putting in place a CI/CD for any given project was treated as best effort. And, to be honest, I cannot blame this on anyone.

Indeed, while CI/CD is widely recognized as a core component of any project, implementing it in the real world often proves challenging.

But I knew we could change that somehow. So before jumping into a strategy to put in place, I wanted to observe what had prevented people to implement a CI/CD to begin with.

Lack of Permissions

The first thing that I realized, is that not all developers had the same level of access to our Jenkins instance. So while some were able to create new pipelines for their projects, some couldn’t. In large organizations, it is not uncommon to have this kind of scenario.

Lesson learned: a lack of permissions should not be a drawback to use a CI/CD pipeline.

Lack of Knowledge

When people had the ability to create a pipeline, some did not, as they simply did not have the proper knowledge to do so. Here, I need to mention that our company uses a Jenkins instance, so developers familiar with other CI/CD tools (e.g., GitHub Actions, CircleCI) could not fully transfer these knowledge to Jenkins. Moreover, we did not have a clear documentation on the process to follow to create a new pipeline, so beginners, fearing to break something, would simply do not take the risk to mess with it.

Lesson learned: we should ensure people without proper Jenkins knowledge can use a CI/CD pipeline.

Lack of Time

It takes effort to put in place a CI/CD at the beginning of a project, something that might be overlooked by managers that want to ship a product as fast as possible. Moreover, it is sometimes difficult to quantify the return on investment of putting in place a CI/CD pipeline. And if it is difficult to prove this can bring business value, it will end up in the “we’ll deal with this later” box. And we know all too well that the tasks that end up in this box will never see the light of day again.

Lesson learned: setting up a CI/CD pipeline for a new or existing project should be easy and straightforward.

Lack of Clear Guidelines

Finally, I had a look at the projects that did have a CI/CD pipeline. They were working fine, but I could clearly see that they lacked a common vision. Some of them used the same formatter (black), but they not always used the same line-length. Some included a testing stage, some didn’t. Not only did this led to projects not having the same code quality and compliance, I also thought that this could potentially led to confusion for a newcomer, not knowing which standard to use.

Lesson learned: people should use the same CI/CD pipeline to ensure everyone follows the same guidelines (and we should ensure these guidelines are documented somewhere).

The Global Vision

After reviewing what could go wrong, it is now important to think of a solution that could address all of these, while following the company’s best practices, and using the tools at our disposal.

Formatting

The first stage should use a formatter to ensure every line of code in our codebase looks the same. This makes sure we are not ending up with different coding standards across our projects.

Previously, we used black as our formatter. But after hearing all the good news and testing the new cool kid in the block, we decided to switch to ruff, as it has the same benefits as black, but with a faster execution.

Linting

The next stage should use a linter to find potential issues with our code. This makes sure we avoid complexity in our code, as well as identifying code smells or security issues.

In the past, I used flake8 a lot. But given that we were already using ruff, and as it can also act as a linter, it was a no-brainer to keep it for this task.

There are many rules that ruff can apply. We decided to use some of them by default, while letting developers the choice to update the ones their project would follow.

Unit Tests

Tests are a critical part of the development of any project. It ensures we ship quality code to production, while also being able to trust that our code would run just fine.

We decided to use pytest to run these tests. The default code directory would be called src, and all tests should be in a tests folder, with files being prefixed by test_.

Code Coverage

Closely related to unit tests, code coverage ensures we are able to know how much of our code has been tested. This could quickly tell us if we sufficiently tested our code, as well as pointing out the remaining lines to cover.

As we were using pytest, we decided to use pytest-cov to generate a coverage report, as it integrates nicely with pytest.

We set the minimum coverage threshold at 50%. Anything lower would risk overlooking significant portions of code, while setting it higher might discourage developers from writing the necessary tests.

Organization Folders for Jenkins

We had our different stages ready. Now all we needed to do, was find a way to globally apply said pipeline to our Python repositories. So I tried looking for a way to easily do that in Jenkins.

That’s when I stumbled upon Organization Folders.

Organization Folders are designed for scenarios like ours: automatically scan an organization (as in, a GitHub organization), filter the repositories you want, and apply a Jenkins pipeline to them.

In our example, we are able to look for all repositories with the “python” topic, and identify them as Python projects. They will then be automatically built. If a new repository is created with this topic, it will also get picked up by Jenkins.

So, in less than 5 seconds, your project could be onboarded, without having to create it in Jenkins. All is done automatically so you can focus on your code.

Examples and Documentation

All of this was great, but I was fearing of one last obstacle. What would happen if developers adopted the CI/CD pipeline only to find stages failing, with no clear documentation to resolve issues? They would probably give up or try to fix it later, which would destroy the initial goal.

So I knew that if I wanted to onboard people in this, we needed to deliver clear documentation with direct examples, so they would be able to understand why these errors might appear, and how to fix them.

Especially on the unit tests stage, as I know this is always a daunting task to start with. So I prepared a project in advance with some unit tests that I knew they could have a look at to take some inspiration, or that directly covered some tricky parts (mocking boto3 API calls, etc.).

The last step was to make a presentation on all of the above. This was key to give meaning to people, so they could really understand the point of doing all this, while making sure they had all the keys to be autonomous.

What We Have Today

From a developer’s view, all he has to do to get his Python project onboarded, is to add the python topic in his repository, and ensures a pyproject.toml file is created at the root of the repository.

These two requirements are here to tell Jenkins which project it should take into account. Moreover, the pyproject.toml file is mandatory for the ruff stages in the pipeline.

With that done, his Python project will now check for formatting issues, linting errors, validation of unit tests, and code coverage.

All in all, we are now able to setup CI/CD pipelines in less than a minute, whether you have a new or existing project!

Some Flaws

While this simplifies things a lot, there is still room for improvements.

The pipeline cannot be enforced

Currently, this CI/CD pipeline cannot be enforced because developers can simply remove the python topic to bypass it. And while this is fine at first, as we do not want to block developers in their work, at some point, the goal is still to make sure we are applying the same best practices in all Python projects.

This might be resolved in the future by the use of GitHub Rulesets.

They essentially act quite the same as branch protection rules, expect you define these rules at the organization level.

This way, we could be able to protect our main branches for all our repositories that matches a specific custom properties, and require them to successfully pass the CI/CD pipeline before they are able to merge.

The pipeline’s stages can be ignored

The pyproject.toml is used to say to ruff which format it should apply, or which rules to follow for the linting part.

And because we are currently using the pyproject.toml inside each repository, a developer could just update the rules on his own, bypassing all the guidelines we were trying to apply in the first place.

Again, while we are allowing this for now to account for the number of fixes to resolve at first, in the end, we might want to prevent this from happening.

We could either use a common and fixed pyproject.toml file, or add it in the GitHub’s CODEOWNERS file to ensure it cannot be modified without strict approval.

What’s Next?

With these in place, we can now think about the evolution of this pipeline.

For example, our company has a SonarQube instance. We would be interested to add a stage that could scan the repository for code smells.

We are also exploring the use of mkdocs, so that projects can share a common style for documentation.

And one of my personal favorites, we might want to explore the use of uv to install requirements, as it is significantly faster than the old pip guy!

You might have also noticed that while I talked about CI/CD throughout this post, at no point do we have a step that, well, deploys anything (which basically leaves us with a CI pipeline). We are already thinking about a way to build and deploy Python packages to our Artifactory repository manager, which would finally make this a CI/CD pipeline!

Finally, we only covered Python in this post, but the same logic could apply to other types of projects. To give you an example, we are also working on a CI/CD pipeline for Terraform.

Conclusion

By tackling permissions, knowledge gaps, and inconsistent guidelines, we built a unified CI/CD strategy that now supports over 100 Python projects. It’s proof that with the right approach, automation is achievable for any organization

It’s been a long but rewarding journey!

I hope this post proved the value of CI/CD, helped you understand what could prevent it from being applied, and gave you some ideas on how to implement a similar strategy in your organization!

Postman pre-request script, never worry about refreshing your access token

Mon, 16 May 2022 22:49:13 +0200

One of the most common tasks when you work with APIs is trying to ping them to see if they work as intended (also known as: testing).

One of the simplest, more useful tool on the market is Postman, an API client that lets you make requests to any URL you like, using all the methods you could dream of. You can even specify some HTTP headers, which are useful if you must pass an access token in the request to authenticate your API call.

And that’s where I began encountering a problem. Not a problem, per se, but something even worse for a Cloud Engineer: repetition.

You see, in order to pass this access token to the request, you must first generate it (usually from another API call). The thing is, this access token is short-lived, and will expire after 1 hour of its generation.

So, what does that imply? Well, every hour, before making another request to your API, you will have to make a first request to get a brand new access token, copy the result, and paste it in the appropriate header of your initial request. In total, this will make you lost 10 seconds.

Ugh! Can you imagine? 10 seconds?! All the things that could be done if we managed to get this precious time back!

But in a product like Postman, I knew this could be resolved somehow. And indeed, after literally one single web search, I stumbled across the perfect solution: Pre-request script.

The idea is pretty simple: execute a script before any request. That’s it.

How is this going to help us, you ask? Well, we can now write a small piece of code, that will get a new access token and store it it the variables of the collection we’re in. Then, our requests will always reference this variable to be sure they have an up-to-date token.

Let me show you an example I used in one of my projects where I needed to get an AWS Cognito access token before each request.

pm.sendRequest({
    url: "https://YOUR_COGNITO_DOMAIN_NAME_HERE.auth.eu-west-1.amazoncognito.com/oauth2/token",
    method: 'POST',
    header: {
        'Content-Type': 'application/x-www-form-urlencoded'
    },
    body: {
        mode: 'urlencoded',
        urlencoded: [
            {
                key: "grant_type",
                value: "client_credentials"
            },
            {
                key: "client_id",
                value: "YOUR_CLIENT_ID_HERE"
            },
            {
                key: "client_secret",
                value: "YOUR_CLIENT_SECRET_HERE"
            },
            {
                key: "scope",
                value: "YOUR_SCOPE_HERE"
            }
        ]
    }
}, (err, res) => {
    pm.collectionVariables.set("access_token", res.json().access_token)
});

As you can see, this is pretty straightforward, and can probably be applied to any of your projects that require OAuth authentication.

I hope this has been helpful and don’t forget: if you can save 10 seconds, do it!

About

Sun, 06 Mar 2022 00:00:00 +0000

My name is Antoine Delia and I’m a Cloud & DevOps Engineer.

On this blog, I want to talk about topics related to Cloud and DevOps. These are topics that I’m deeply folded with, et I like to share mu knowledge, adding a bit of fun in the mix.

I hope you’ll enjoy it!

Skills:

Cloud (AWS)
Infrastructure as Code (Terraform / Serverless Framework / CloudFormation)
Scripting (Python, Bash)
CI/CD (GitHub Actions, Jenkins)

Projects and achievements:

The Cloud Optimist (the blog you’re on right now): Hugo blog deployed with GitHub Actions on AWS, infrastructure managed by Terraform
Introduction to Serverless: webinar (https://www.youtube.com/watch?v=nCGTWbUsfN4)
3 AWS certifications (Architect Associate / Developer Associate / Cloud Practitioner)
1800+ points on StackOverflow
Marathon in Toulouse 2025 achieved in 03:58:44
TOEIC (965/990)

The Rubber Duck Debugging - why does it work?

Wed, 26 Jun 2019 22:49:13 +0200

Programming is a painful art. You are a real architect and you mission is to design, build, decorate and maintain an application, much like how you would do for a house.

One difficulty emerges though: you have to communicate information with a machine through programming languages to make everything work. Surely you don’t talk to your furniture to place them into your house, right? Right?

This little difference is crucial, as you sometimes will encounter an issue that you have no idea why it is there. Truly you developed this thing so perfectly that it should not fail. But here we are, a bug is there and you don’t know why.

The first thing to do is of course to blame it on the computer. I mean, you know what you’re doing, there is no way you wrote something the wrong way. It’s probably this damn laptop which is making you look like a fool again. But the truth is hard to swallow.

The machine does exactly what you tell it to do.

– Bill Gates, probably

Like it or not, your computer is dumb. It will mindlessly retrieve the information you provided and execute them. And if it fails, it is, without a doubt, your fault. I know the pain, trust me. But don’t worry too much, it happened to every one of us, and at the end of the day, everything will be fine.

Now, it is all up to you. You know that you did something wrong, yet are unable to find what.So what’s next? Should look up answers on StackOverflow? Resign of your position as you clearly are not good enough for this job? The answer is simple.

The first step when you are stuck, is to ask the wisdom of the duck.

I know what you’re thinking right now: What are you smoking and where can I get some? But trust me, this whole duck thing is not an hallucination, it is THE best way to help you.

Don’t worry though, you won’t need to drive to the nearest farm and steal a duck to apply this advice. In our case, a rubber duck is what we need.

Once you get your hands on one and place it on your desk, the only thing left is to talk to him. Go ahead, don’t be shy and explain to him what’s happening with your code.

I’d rather tell about it right now so you don’t end up shocked when confronted with the truth, but the duck will not respond to you. Hearth-breaking, I know. However, he is the perfect listener and his patience is way above any other human being unlike your co-workers. Moreover, the duck will never judge your mistakes. He is kind and comprehensive.

And the more you will explain to him your troubles, the more you’ll realize what was wrong all along in your code. Indeed, trying to explain properly to the duck what you are trying to do will expose a tiny inconsistency between what you did and what you are trying to achieve. Slowly but surely, you will soon enough be able to pinpoint the exact line of code that was causing all your troubles and fix it like the pro you are.

And that’s why the duck is so useful. Not only did you let your coworkers work in peace, this exercise forces you to take a step back at your code so that you can explain it to someone who has no clue of what’s going on.

This effort to put words on the expected behavior and what you were able to achieve for now will highlight any potential mistakes you might have made during the development, things that can’t be seen when you spent eight hours looking at your code non-stop.

In describing what the code is supposed to do and observing what it actually does, any incongruity between these two becomes apparent.

– Stephen J. Baker

So next time you’re stuck on something, don’t bother your co-workers, save your internet data and simply ask the duck!

If you wish to learn more on this method, I strongly suggest having a look at the website behind this post’s inspiration. https://rubberduckdebugging.com/

FizzBuzz, the simplest, more efficient way to test a developer

Mon, 27 May 2019 13:35:55 +0200

When I got out of my engineering school, I passed a lot of interviews in the hope of getting a developer job. My experiences being limited to having done a few internships, I was fearing the part of the interview where I would have had to get up and resolve some complex algorithm exercise on a white board, or even just answer a few technical questions. I was wrong.

Not a single interview tested me on my development skills.

I could’ve lied my way through the whole interview that I would still have gotten the job.

To be honest, I’m not part of the people who praise technical questions like it is some kind of magic spell to detect good programmers. I’d rather think than someone with a good temperament, passionate and humanely pleasant is much more appreciated than a know-it-all who always brings his science to the table without wanting to hear anyone else opinions. You can improve a technical gap, but you can’t improve a shitty character.

I do believe however that it is mandatory to at least check the way of thinking of someone when being confronted to a problem. Because in the end, that’s what he is gonna do most of his job: resolve problems. As such, it is crucial to see how someone will behave when being presented a problem. Will he run away? Will he immediately ask for help?

That’s when the FizzBuzz test comes into play.

This post was inspired by Tom Scott’s video that I strongly suggest you to have a look at if this topic interests you.

The idea behind FizzBuzz is straight-forward. You need to count out loud numbers starting at 1 until you reach 100. But, if the number is a multiple of 3, you have to say the word “Fizz” instead. In the same way, if the number is a multiple of 5, you have to say the word “Buzz”. Now you might be thinking: “Then what am I supposed to do when it is a multiple of both 3 and 5?”. Well, you would simply have to say “FizzBuzz”. Roll credits.

By reading this simple test, you might wonder why this should be used, as it appears that even an intern could solve it quite easily. Well, that’s what I thought too, until I ran across Jeff Atwood’s post on this subject, stating that he was surprised how this problem actually gave a lot of troubles to candidates, even senior ones.

So let’s try to solve it together!

In most technical interviews and as such for this exercise, you are free to use any programming language you feel most comfortable with, or even write using pseudocode. For this post, I’m gonna use Python.

First things first, let’s create a simple loop that prints all numbers from 1 to 100.

1
2

for i in range(1, 101):
    print(i)

Off to a great start! Now let’s review the original statement: if the number is a multiple of 3, show “Fizz”, if the number is a multiple of 5, show “Buzz”. No Let’s add these conditions.

for i in range(1, 101):
    if i % 3 == 0:
        print('Fizz')
    if i % 5 == 0:
        print('Buzz')
    if i % 3 != 0 and i % 5 != 0:
        print(i)

Let’s take a minute to see what we did here. In order to check whether a number is a multiple of 3, we are using the modulo with the symbol %.We are doing the same thing for multiples of 5. Finally, if the number is neither a multiple of 3 nor 5, we simply display it.

This way we get exactly what we want !… Except when a number is a multiple of both 3 and 5. When that happens, the words Fizz and Buzz are not displayed on the same line ! We shall then take into account this case by adding a new condition, and prevent the first two if to trigger.

for i in range(1, 101):
    if i % 3 == 0 and i % 5 != 0:
        print('Fizz')
    if i % 5 == 0 and i % 3 != 0:
        print('Buzz')
    if i % 3 == 0 and i % 5 == 0:
        print('FizzBuzz')
    if i % 3 != 0 and i % 5 != 0:
        print(i)

This finally works! But at what cost? This whole code is a mess. And it is by no means a piece of code that can easily be maintained. Take this issue as an example: what would happen if we now want to display the word Fizz for multiples of 2 instead of 3? We will have to change this value at 4 different places through the code to make it work.

This might look trivial with such a minimalistic code, but imagine what would happen if this happened on a much bigger project. This code is prone to errors, complicates its maintainability and makes it almost impossible to add new features easily.

Let’s try another approach to make this code more elegant! We’ll first declare a variable which will be the value we want to print at the end. We’ll make it an empty string for now. Each time we successfully match a condition, we’ll simply append the corresponding word to the result. At the end, checking if the variable is empty will let us know if we should print it or rather print the number.

This is what we get.

for i in range(1, 101):
    result = ''
    if i % 3 == 0:
        result += 'Fizz'
    if i % 5 == 0:
        result += 'Buzz'
    print(result or i)

It’s already much clearer! Not only did we shrink the number of lines by half, our if checks are now crystal clear and don’t let any rooms for misinterpretation. Plus, we can easily change the value of any of the numbers if we need to without having to look for all its occurrences. Last but not least, if we ever need to add yet another condition for multiples of 7, we’ll simply have to add two new lines without changing the rest of the code!

We can finally see how this simple FizzBuzz test is so interesting. We were able to test one’s ability to use simple algorithmic principles, but also to check his way of thinking when being confronted to a problem, and if he is able to produce a code that takes care of its future.

To make it short, FizzBuzz is a good way to differentiate a developer from a good developer.

Links

Mon, 01 Jan 0001 00:00:00 +0000

Search

Mon, 01 Jan 0001 00:00:00 +0000