Automate Image Optimization (WebP) upon Upload to S3 Bucket

Hey there, Darian Vance here. Let’s talk about a real time-saver. For the longest time, our workflow involved a designer uploading JPEGs and PNGs, and then someone from my team would have to manually pull them, run them through an optimization script, and re-upload WebP versions. It was a tedious, error-prone step that burned a few hours every week. Automating this directly in AWS with a Lambda function triggered by an S3 upload was a complete game-changer for us. It’s one of those “set it and forget it” solutions that pays dividends in both performance and sanity. I want to show you exactly how we did it.

Prerequisites

Before we dive in, make sure you have the following ready:

• An AWS account with permissions to manage IAM, S3, and Lambda.
• A basic understanding of Python.
• The AWS CLI configured on your machine. It just makes testing and deploying easier.
• Two S3 bucket names in mind: one for raw uploads and one for the optimized WebP images.

The Step-by-Step Guide

Step 1: The IAM Role for Our Lambda

First things first, our Lambda function needs permission to interact with other AWS services. We’ll create an IAM role that grants it just enough access to do its job—nothing more. This is the principle of least privilege in action.

In the IAM console, create a new role. Select “AWS service” as the trusted entity and “Lambda” as the use case. For permissions, you’ll create a new inline policy. Here’s a JSON policy I often use as a baseline. Remember to replace your-source-bucket-name and your-destination-bucket-name with your actual bucket names.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-source-bucket-name/*"
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::your-destination-bucket-name/*"
        }
    ]
}

The first statement gives the function permission to write logs to CloudWatch, which is essential for debugging. The next two grant it read access to the source bucket and write access to the destination. Give the role a descriptive name like S3-WebP-Converter-Role and save it.

Step 2: The Python Logic for the Lambda Function

Now for the main event. This Python script will be the engine of our operation. I’ll skip the standard project setup steps like creating a directory or virtual environment since you probably have your own workflow for that. The key is that you’ll need two Python libraries: boto3 (the AWS SDK for Python) and Pillow (for image processing). You can typically get these by running pip install boto3 Pillow in your project environment.

Here is the code for a file you can call lambda_function.py:

import boto3
from PIL import Image
import os
import io

s3 = boto3.client('s3')
DESTINATION_BUCKET = os.environ['DEST_BUCKET']

def lambda_handler(event, context):
    source_bucket = event['Records'][0]['s3']['bucket']['name']
    object_key = event['Records'][0]['s3']['object']['key']

    # Avoid processing non-image files if needed
    if not object_key.lower().endswith(('.png', '.jpg', '.jpeg')):
        print(f"Skipping non-image file: {object_key}")
        return {
            'statusCode': 200,
            'body': 'File is not a supported image type.'
        }

    try:
        # Download the file to the Lambda's temporary storage
        # Note: Lambda provides a writable /tmp directory
        download_path = f'/tmp/{os.path.basename(object_key)}'
        s3.download_file(source_bucket, object_key, download_path)

        # Use an in-memory buffer for the converted image
        buffer = io.BytesIO()
        
        with Image.open(download_path) as image:
            # Handle PNG transparency by ensuring RGBA mode before save
            if image.mode in ('RGBA', 'LA') or (image.mode == 'P' and 'transparency' in image.info):
                image.save(buffer, 'WEBP', quality=85, method=6)
            else:
                image.convert('RGB').save(buffer, 'WEBP', quality=85, method=6)
        
        buffer.seek(0) # Rewind the buffer to the beginning

        # Construct the new key and upload to the destination bucket
        new_key = os.path.splitext(object_key)[0] + '.webp'
        s3.put_object(Bucket=DESTINATION_BUCKET, Key=new_key, Body=buffer, ContentType='image/webp')

        print(f"Successfully converted {object_key} to {new_key} in {DESTINATION_BUCKET}")
        return {
            'statusCode': 200,
            'body': f'Successfully processed {object_key}'
        }

    except Exception as e:
        print(f"Error processing {object_key}: {str(e)}")
        # In a real-world scenario, you might want to send a notification here
        return {
            'statusCode': 500,
            'body': f'Error processing {object_key}: {str(e)}'
        }

The logic is straightforward: the function gets triggered by an S3 event, downloads the source image, uses Pillow to convert it to WebP in memory, and then uploads the result to our destination bucket.

Pro Tip: Notice the DEST_BUCKET environment variable. Using environment variables in your Lambda configuration is much cleaner than hardcoding values like bucket names directly into your code. It makes the function reusable across different environments (dev, staging, prod).

Step 3: Package and Deploy the Lambda

Because our code uses a library (Pillow) that isn’t included in the standard Lambda runtime, we have to package it up. The simplest way is to create a ZIP file containing your `lambda_function.py` and all the installed package folders.

Once you have your ZIP file:

1. Go to the AWS Lambda console and create a new function from scratch.
2. Give it a name, select a recent Python runtime (e.g., Python 3.9).
3. Under “Permissions,” choose “Use an existing role” and select the IAM role we created in Step 1.
4. Once created, upload your ZIP file under the “Code source” section.
5. Go to the “Configuration” tab, then “Environment variables.” Add a new variable with the key DEST_BUCKET and the value as your destination bucket’s name (e.g., my-app-assets-optimized).
6. In “General configuration,” you might want to increase the timeout from the default 3 seconds to 15 or 30 seconds, just in case you upload a large image that takes longer to process.

Step 4: Connect S3 to Lambda with an Event Notification

The final piece is telling S3 to trigger our function. In the Lambda console, click “Add trigger.” Select S3 as the source. Choose your source bucket (e.g., `my-app-uploads-raw`). For the event type, select “All object create events.” This ensures the function runs whenever a new file is uploaded.

And that’s it! Now, any time you upload a JPG or PNG to your source bucket, the Lambda will automatically fire, convert it, and place a shiny new WebP version in your destination bucket.

Common Pitfalls (Where I Usually Mess Up)

• The Infinite Loop: This is the classic mistake. If you configure your Lambda to listen to a bucket and then write the output *back to the same bucket*, you will trigger an infinite loop. The newly created WebP file is an “object create event,” which triggers the Lambda again. Always use separate source and destination buckets or, if you must use one, configure very specific prefix/suffix filters on your S3 trigger.
• IAM Permissions: If the function fails with an “Access Denied” error in the CloudWatch logs, it’s almost always the IAM policy. Double-check that the role allows `s3:GetObject` on the source bucket ARN and `s3:PutObject` on the destination.
• Forgetting PNG Transparency: A standard `image.convert(‘RGB’)` call will strip transparency from PNGs, leaving you with an ugly black background. The code I provided includes a check to preserve that transparency during the WebP conversion, which is crucial for things like logos or icons.

Conclusion

You’ve just built a scalable, serverless pipeline that automates a critical part of web performance optimization. This setup saves development time, eliminates human error, and ensures your applications are always serving fast, modern image formats. In my production setups, this pattern is a workhorse. It’s a small infrastructure investment for a massive payoff in performance and team productivity. Happy building!

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions