Real-time DDoS Attack Detection using Nginx Access Logs

Hey there, Darian Vance here. As a Senior DevOps Engineer at TechResolve, I’ve seen my fair share of late-night alerts. I used to spend my mornings combing through gigabytes of Nginx logs after a traffic spike, trying to figure out if it was a legitimate flash mob or something more malicious. I was wasting hours a week on reactive analysis. That all changed when I built a simple, real-time detection script. It’s not a full-blown WAF, but for catching sudden, brute-force attacks, it’s been an absolute game-changer. It gives me a heads-up the moment things start looking suspicious, letting me act before the server grinds to a halt. This is how I built it.

Prerequisites

Before we dive in, make sure you have the following ready. This whole setup is pretty lightweight.

• Access to a server running Nginx.
• Read permissions for the Nginx access.log file.
• Python 3 installed on the server.
• The ability to install Python packages. We’ll need one called tailer.

The Step-by-Step Guide

Alright, let’s get this thing built. The core idea is simple: we’re going to watch the Nginx access log file as new lines are written, keep track of how many requests each IP address makes within a short time window, and flag any IP that gets a little too aggressive.

Step 1: Setting Up Your Environment

First things first, you’ll want to get your Python environment ready. I’ll skip the standard virtualenv setup since you likely have your own workflow for that. The key dependency you’ll need is `tailer`, which is great for following log files, much like the `tail -f` command. You can get it by running a pip install command for `tailer` in your activated environment. Let’s jump straight to the Python logic.

Step 2: The Detection Script

Create a Python file, let’s call it log_monitor.py. This script is where the magic happens. We’re going to read the log line-by-line in real-time and do our analysis.

Here’s the logic behind the code we’re about to write:

1. Follow the Log: Use the `tailer` library to continuously read new lines from `access.log`.
2. Track IPs: We’ll use a Python dictionary to store IP addresses as keys. The value for each IP will be a list of timestamps for its recent requests.
3. Use a Sliding Window: For each new request from an IP, we’ll add the current timestamp to its list. Then, we’ll remove any timestamps from that list that are older than our defined time window (e.g., 60 seconds).
4. Check the Threshold: After cleaning up old timestamps, the number of timestamps left in the list is the IP’s request count within our window. If this count exceeds a set threshold, we trigger an alert.

Here is the full script. I’ve added comments to explain each part.

import time
import re
from collections import deque
import tailer

# --- Configuration ---
# Point this to your Nginx access log file.
LOG_FILE_PATH = 'your/path/to/nginx/access.log'

# The time window in seconds to track requests.
TIME_WINDOW_SECONDS = 60

# The number of requests from a single IP to trigger an alert.
REQUEST_THRESHOLD = 100

# --- Main Logic ---

def monitor_log_file():
    """
    Tails the Nginx log file and detects potential DDoS attacks.
    """
    print(f"Starting log monitor for {LOG_FILE_PATH}...")
    
    # This dictionary will store deque objects for each IP.
    # A deque is a double-ended queue, perfect for our sliding window.
    ip_requests = {}
    
    # A simple regex to extract the client IP address from a standard Nginx log line.
    ip_pattern = re.compile(r'(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})')

    try:
        # tailer.follow is a generator that yields new lines as they are added.
        for line in tailer.follow(open(LOG_FILE_PATH)):
            match = ip_pattern.match(line)
            if not match:
                continue

            client_ip = match.group(1)
            current_time = time.time()

            # Get the request history for this IP, or create it if it's new.
            if client_ip not in ip_requests:
                ip_requests[client_ip] = deque()
            
            # Add the current request's timestamp to this IP's history.
            ip_requests[client_ip].append(current_time)

            # --- The Sliding Window Logic ---
            # Remove timestamps from the left of the deque that are outside our time window.
            # This is much more efficient than rebuilding a list every time.
            while ip_requests[client_ip] and ip_requests[client_ip][0] <= current_time - TIME_WINDOW_SECONDS:
                ip_requests[client_ip].popleft()

            # --- Check Against Threshold ---
            request_count = len(ip_requests[client_ip])
            if request_count > REQUEST_THRESHOLD:
                print(f"ALERT! High traffic from IP: {client_ip}. "
                      f"Requests in last {TIME_WINDOW_SECONDS}s: {request_count}")
                # In a real setup, you'd trigger a firewall block or a notification here.

    except FileNotFoundError:
        print(f"Error: Log file not found at {LOG_FILE_PATH}")
    except Exception as e:
        print(f"An unexpected error occurred: {e}")

if __name__ == "__main__":
    monitor_log_file()

Pro Tip: Tuning Your Threshold

The `REQUEST_THRESHOLD` value of 100 is just a starting point. In my production setups, I first run the script in a “silent” mode (just printing counts, not alerting) to establish a baseline for normal traffic. You might find that your API servers have a much lower normal threshold than your front-end web servers. Tailor it to your specific traffic patterns to avoid false positives.

Step 3: Running the Monitor

To run this script, you just execute it from your terminal. Since you want it to run continuously, you should run it as a background process. On a Linux system, you can use `nohup` to keep it running even after you log out, or better yet, set it up as a `systemd` service for proper process management. For simplicity, `nohup python3 log_monitor.py &` gets the job done for a quick start.

While this script is designed for real-time monitoring, you could adapt it to run periodically via cron if you prefer summary reports. A cron job like * * * * * python3 your_script.py would run it every minute, but you’d lose the real-time benefit of tailing the file.

Here’s Where I Usually Mess Up (Common Pitfalls)

• Log Rotation: The first time I deployed a script like this, it broke at midnight. Why? Log rotation. Nginx moved the active log file and created a new one. The `tailer` library is pretty good at handling this, but it’s something to be aware of. Always double-check that your script can survive a log rotation event.
• Threshold Too Low: Being overzealous with a low threshold is a classic mistake. You’ll end up getting alerts for legitimate traffic spikes or even search engine crawlers. Always baseline your traffic first.
• Ignoring IPv6: My example regex only captures IPv4 addresses. If your server is on a dual-stack network, you absolutely need to update the regex to handle IPv6 addresses as well, or you’ll miss a huge chunk of traffic.
• The Alert Is Just the Beginning: Just printing an alert to the console is a good start, but it’s not an automated solution. The real power comes from integrating this script with your firewall. When an IP is flagged, the script should automatically add a rule to `iptables` or your cloud provider’s firewall API to temporarily block that IP.

Conclusion

And that’s it. With a relatively simple Python script, you’ve moved from being a reactive log-diver to a proactive threat-hunter. This setup gives you immediate visibility into suspicious traffic patterns and forms a solid foundation for building more complex, automated defense systems. It won’t stop every sophisticated attack, but it’s an incredibly effective and low-cost first line of defense that has saved my team countless hours. Happy monitoring!

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions