Detecting Unencrypted S3 Objects and Auto-Remediating

Hey there, Darian Vance here. As a Senior DevOps Engineer at TechResolve, I’ve seen my fair share of late-night alerts for compliance issues. One of the most common (and tedious) ones was tracking down S3 objects uploaded without server-side encryption. I used to spend hours sifting through CloudTrail logs to find the culprits. It was a time sink and, frankly, a perfect task for automation. That’s why I built this event-driven workflow. It automatically detects and encrypts non-compliant objects within seconds of their upload. This guide will walk you through setting it up so you can reclaim your time and ensure your data is always protected.

Prerequisites

Before we dive in, make sure you have the following ready. I’m assuming you’re comfortable navigating the AWS console.

• An AWS account with permissions to manage S3, CloudTrail, EventBridge, and Lambda.
• Python 3 and the AWS Boto3 library installed on your local machine for packaging the Lambda function. I’ll skip the standard `virtualenv` setup since you likely have your own workflow for that. Just be sure to package Boto3 with your function if you’re using libraries not included in the standard Lambda runtime.
• An S3 bucket that you want to monitor. For this guide, let’s call it my-company-reports.

The Step-by-Step Guide

Our architecture is simple and effective: CloudTrail logs the S3 upload event, EventBridge filters for that specific event, and Lambda executes our remediation logic. Let’s build it piece by piece.

Step 1: Enable CloudTrail Data Events

This is our foundation. By default, CloudTrail tracks management events (like creating a bucket), but not data events (like uploading an object). We need to explicitly turn this on for the bucket we want to monitor.

1. Navigate to the CloudTrail service in your AWS console.
2. Go to “Trails” and either create a new trail or select an existing one.
3. Find the “Data events” section and click “Edit”.
4. Select “S3” as the data event source.
5. Configure it to log “Write” events for “Current and future S3 buckets” or, for better cost control, specify the exact bucket ARN (e.g., arn:aws:s3:::my-company-reports/).
6. Save your changes. Now, every PutObject API call to our target bucket will be logged.

Step 2: Create the EventBridge Rule

Next, we need EventBridge to act as our watchdog. It will monitor the stream of CloudTrail events and pick out the exact ones we care about, ignoring all the noise.

1. In the AWS console, go to Amazon EventBridge and click “Create rule”.
2. Give it a descriptive name like S3-Unencrypted-Object-Detector.
3. In the “Event pattern” section, select “Custom pattern”.
4. Paste the following JSON into the editor. Remember to replace my-company-reports with your actual bucket name.

{
  "source": ["aws.s3"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "eventSource": ["s3.amazonaws.com"],
  "eventName": ["PutObject"],
  "detail": {
    "requestParameters": {
      "bucketName": ["my-company-reports"]
    }
  }
}

This pattern tells EventBridge to only fire when a PutObject API call occurs in our specific bucket. It’s precise and efficient.

Pro Tip: In my production setups, I often create a more generic rule that listens for PutObject in *any* bucket and then have the Lambda function decide if the bucket is in scope. This scales better if you have dozens of buckets to monitor, but for a single bucket, the pattern above is perfect.

Step 3: Write the Lambda Remediation Function

This is the core of our automation. The Lambda function will receive the event from EventBridge, inspect the newly uploaded object, and if it’s unencrypted, it will fix it.

Here’s the Python code for the function. The logic is straightforward:

• It receives the event and extracts the bucket and object key.
• It uses head_object to check the object’s metadata.
• Crucially, it checks for a custom metadata tag, remediated-by-lambda. If this tag exists, it stops immediately. This prevents an infinite loop where our fix triggers the rule again.
• If the object has no encryption and no remediation tag, it calls copy_object. This is a server-side operation that copies the object over itself, but this time adding AES256 encryption and our custom metadata tag.

Create a new Lambda function with a Python 3.x runtime and use the following code:

import boto3
import logging
import urllib.parse

logger = logging.getLogger()
logger.setLevel(logging.INFO)

s3_client = boto3.client('s3')

REMEDIATION_METADATA_KEY = 'remediated-by-lambda'

def lambda_handler(event, context):
    try:
        # 1. Extract bucket and key from the event
        bucket_name = event['detail']['requestParameters']['bucketName']
        # The key can have special characters, so we unquote it
        object_key = urllib.parse.unquote_plus(event['detail']['requestParameters']['key'])
        
        logger.info(f"Processing object '{object_key}' in bucket '{bucket_name}'")

        # 2. Check the object's metadata and encryption status
        head_response = s3_client.head_object(Bucket=bucket_name, Key=object_key)
        metadata = head_response.get('Metadata', {})
        
        # 3. Prevent infinite loops: check if we already remediated it
        if metadata.get(REMEDIATION_METADATA_KEY) == 'true':
            logger.info(f"Object '{object_key}' was already remediated. Ignoring.")
            return {'status': 'ignored', 'reason': 'already remediated'}

        # 4. Check for encryption
        encryption_status = head_response.get('ServerSideEncryption')
        if encryption_status:
            logger.info(f"Object '{object_key}' is already encrypted with {encryption_status}. No action needed.")
            return {'status': 'compliant', 'reason': 'already encrypted'}

        # 5. Remediate: copy the object in place with encryption
        logger.warning(f"Object '{object_key}' is NOT encrypted. Applying remediation...")
        
        copy_source = {'Bucket': bucket_name, 'Key': object_key}
        
        # Preserve existing metadata and add our remediation tag
        new_metadata = metadata
        new_metadata[REMEDIATION_METADATA_KEY] = 'true'

        s3_client.copy_object(
            CopySource=copy_source,
            Bucket=bucket_name,
            Key=object_key,
            ServerSideEncryption='AES256',
            MetadataDirective='REPLACE',
            Metadata=new_metadata
        )
        
        logger.info(f"Successfully remediated and encrypted '{object_key}'.")
        return {'status': 'remediated'}

    except KeyError as e:
        logger.error(f"Could not parse event. Missing key: {e}")
        return {'status': 'error', 'reason': 'malformed event'}
    except Exception as e:
        logger.error(f"An unexpected error occurred: {e}")
        raise e

Step 4: Connect EventBridge to Lambda

Finally, we wire everything together. In the EventBridge rule you created in Step 2, select your Lambda function as the “Target”. EventBridge will automatically handle invoking the function and passing the event payload.

You’ll also need to configure the Lambda function’s IAM Execution Role. It needs permissions to perform S3 actions on the target bucket. A minimal policy would look like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:HeadObject"
            ],
            "Resource": "arn:aws:s3:::my-company-reports/*"
        }
    ]
}

And that’s it! Now, try uploading a file to your S3 bucket without specifying any encryption. Within a few moments, you should see the object’s properties update in the S3 console to show AES256 encryption and our custom metadata tag. Check the Lambda’s CloudWatch logs to see the play-by-play.

Common Pitfalls (Where I Usually Mess Up)

Even a simple workflow has places to trip up. Here are the two biggest ones I’ve run into:

• IAM Permissions: By far the most common issue. If the Lambda function’s role doesn’t have s3:HeadObject and s3:PutObject (for the copy operation), it will fail silently. Always check the CloudWatch logs for timeout or “Access Denied” errors first.
• The Infinite Loop: Before I added the metadata check, my early versions of this script would get stuck in a loop. The remediation (a PutObject via copy) would trigger the rule, which would invoke the Lambda, which would remediate it again… and again. The metadata check is a simple and foolproof way to break the cycle.

Conclusion

You’ve just built a powerful, serverless, and cost-effective security control. This “detect and remediate” pattern is incredibly useful and can be adapted for all sorts of compliance automation. It frees you up from manual checks, tightens your security posture, and gives you peace of mind that your data is protected by default. Happy building!

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions