Quick Summary: Stuck dealing with a legacy server that refuses to honor HTTP Range headers for a massive file? Here is how to force a partial download by terminating the stream client-side, saving your local disk space and sanity.

Partial Downloads: How to Win When the Server Says “All or Nothing”

I still remember the night I almost crashed my laptop because of a misconfigured Apache server from the early 2000s. It was 2 AM, and I was troubleshooting an incident on prod-legacy-billing-03. I needed to see the header of a backup archive—just the first few kilobytes to verify a timestamp. I casually ran a wget, assuming I could CTRL+C it after a few seconds.

The server, however, had other plans. It ignored my attempts to pause, flooded my network buffer, and because the file was a 200GB tarball served with zero compression and no range support, my terminal froze solid trying to buffer the output. I wasn’t fighting the code; I was fighting the laws of TCP physics. If you’ve ever tried to pull “just a little bit” of a file from a server that responds to a Range request with a hearty “200 OK” and the entire file, you know my pain. Here is how we handle that at TechResolve without pulling our hair out.

The “Why”: It’s Not You, It’s the Header

Before we fix it, you need to understand why your server is acting like a stubborn toddler. Modern, well-behaved servers (Nginx, AWS S3) support Range Requests. When you ask for bytes 0-100, they reply with HTTP/1.1 206 Partial Content.

However, when you deal with dynamically generated reports, ancient IIS installs, or weird middleware, the server often lacks the ability to seek to a specific byte offset. It sends Accept-Ranges: none (or just omits the header entirely). When your client asks for a slice, the server ignores the request and starts pouring the whole bucket—sending a standard 200 OK.

Since the server won’t stop sending, the client must stop listening. Here are three ways to do exactly that.

Solution 1: The Quick Fix (The “Unix Pipe” Method)

If you are in a terminal and just need the top of a file (like checking a !DOCTYPE or a log header), don’t overthink it. We can abuse the behavior of Unix pipes.

When you pipe curl into head, curl starts writing data to the pipe. Once head receives the amount of data it requested (say, 10MB), it terminates successfully and closes the read end of the pipe. When curl tries to write to that closed pipe again, the OS sends a SIGPIPE signal, killing the download process instantly.

This is hacky, but effective for grabbing the start of a file.

# Download the first 10MB of a webpage or file
curl -sN https://internal-docs-01.techresolve.io/massive-report.html | head -c 10M > report_partial.html

Pro Tip: The -N flag in curl is crucial here. It disables buffering. Without it, curl might buffer more data than you want before it realizes the pipe is broken, wasting bandwidth.

Solution 2: The Permanent Fix (The “Python Surgeon”)

If you are building an automation script or an internal tool, you can’t rely on head and SIGPIPE errors. You need control. This is where Python’s requests library shines, provided you use the stream=True parameter.

This method opens the socket but doesn’t download the body until you iterate over it. You count the bytes as they come in, and the moment you hit your limit, you close the connection. This is how I built our internal “Log Sniffer” tool.

import requests

url = "https://prod-db-04.techresolve.internal/slow-query.log"
output_file = "partial_download.log"
max_bytes = 1024 * 1024 * 5  # 5 MB limit

# NOTE: stream=True is mandatory here!
with requests.get(url, stream=True) as r:
    # Check if the server actually ignored us (200 instead of 206)
    if r.status_code == 200:
        print("Server ignores ranges. Manual stream termination engaged.")
    
    with open(output_file, 'wb') as f:
        downloaded = 0
        # Read in small chunks (1KB)
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
                f.write(chunk)
                downloaded += len(chunk)
                # The precise moment we cut the cord
                if downloaded >= max_bytes:
                    print(f"Limit reached ({downloaded} bytes). Closing connection.")
                    break

Solution 3: The “Nuclear” Option (Process on the Fly)

Sometimes you don’t want the start of the file. Sometimes you need to grep for a specific string inside a 50GB file, and you don’t have 50GB of disk space to store it locally. Since the server won’t let us jump to the middle, we have to download the stream, but we don’t have to save it.

We treat the network stream purely as input for processing. This burns bandwidth (you still download the data), but it saves your disk IO and storage.

In the second example (getting the last 100 lines), realize that curl will download the entire file, but tail holds only the last buffer in memory. You download 50GB over the network, but only write 10KB to disk. It’s inefficient for the network, but it solves the “My disk is full” problem.

Warning: Be careful running these “stream processing” commands on metered connections (like AWS NAT Gateways). You are still paying for the data transfer even if you aren’t saving the file!

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions