Tired of slow, sequential downloads from a massive list of URLs? A senior DevOps engineer shares battle-tested shell commands using `xargs` and `parallel` to get the job done efficiently without crashing your server.

From My Terminal to Yours: The Battle-Tested Guide to Bulk Downloading URLs

I remember it like it was yesterday. It was 1:30 AM, and a critical data migration for our largest e-commerce client was failing spectacularly. A list of about 80,000 product image URLs, dumped from a legacy system into a simple `product_images.txt` file, needed to be downloaded to our new asset server, `prod-assets-01`. The junior engineer on call, bless his heart, had written a simple bash loop. The estimated completion time? Sometime next Tuesday. The site was live with thousands of broken images, and my phone was melting. That night, under pressure, we didn’t just fix the problem; we solidified a playbook for a task that seems simple but is deceptively tricky.

The “Why”: Why Your Simple Loop is Killing Performance

When you’re faced with a file full of URLs, the first instinct for many is to write a simple `for` or `while` loop in bash. It makes sense, it’s readable, and it works. But it’s also incredibly inefficient for a large number of files. Why? Because it’s sequential.

Your script does the following, one by one:

• Starts a `wget` or `curl` process.
• Waits for the DNS lookup.
• Waits for the TCP connection.
• Waits for the file to download completely.
• Closes the connection.
• Finally, starts the entire process over for the next URL.

Most of that time is spent waiting on network I/O. Your powerful, multi-core CPU is just sitting there, bored, waiting for a server halfway across the world to respond. We can do better. We have to do better.

Solution 1: The Quick & Dirty Fix (The Simple Loop)

First, let’s look at the approach that got my junior colleague into trouble. I call this the “it works for 10 files” method. It’s fine for a quick, one-off task, but it doesn’t scale and has zero error handling.

# urls.txt contains one URL per line
while read url; do
  wget "$url"
done < urls.txt

This is the classic, readable, and painfully slow method. If one download hangs, the whole script hangs. It’s fragile, but for a handful of URLs from a reliable source, it’ll get you out of a pinch. Just don’t use it when the VP of Engineering is watching you on a shared screen at 2 AM.

Solution 2: The “Smarter, Not Harder” Fix (Using `xargs`)

This is my go-to for most situations. `xargs` is a standard Unix utility that turns input from standard in into arguments for a command. Its magic ingredient for us is the `-P` flag, which specifies the maximum number of processes to run in parallel.

Instead of one download at a time, we can tell it to run, say, 16 downloads simultaneously.

# Download up to 16 files at once
cat urls.txt | xargs -n 1 -P 16 wget -q

Let’s break that down:

• cat urls.txt: Reads our file and pipes its content to the next command.
• xargs: Is waiting to receive that content.
• -n 1: Tells `xargs` to use just one line of input (one URL) per command it builds.
• -P 16: The magic. Run up to 16 `wget` processes at the same time.
• wget -q: The command to run. I added `-q` (quiet) to prevent our terminal from being flooded with 80,000 progress bars.

Pro Tip from the Trenches: How many processes for `-P`? A good starting point is the number of CPU cores on your machine (`nproc` or `sysctl -n hw.ncpu` can tell you). For network-bound tasks like this, you can often go higher, like 2x or 4x your core count. Experiment! Start with 8 or 16 and see how the system load (`uptime`) and network performance (`iftop`) respond.

Solution 3: The “Production-Grade” Fix (Using GNU `parallel`)

When you need more power, better reporting, and bulletproof execution, you bring in the heavy machinery. GNU `parallel` is `xargs` on steroids. It’s not always installed by default, but a quick `sudo apt-get install parallel` or `brew install parallel` is well worth it.

It gives us things `xargs` can’t easily do, like a progress bar, ETA, and built-in retries.

# Use parallel for a progress bar, retries, and job control
cat urls.txt | parallel --jobs 16 --eta --retries 3 wget {}

What’s new here?

• parallel: The command itself.
• --jobs 16: Same as `-P 16` in `xargs`.
• --eta: This is beautiful. It shows you a progress bar and an Estimated Time of Arrival. No more guessing!
• --retries 3: If a `wget` command fails (e.g., a network blip), `parallel` will automatically retry it up to 3 times. This is huge for reliability.
• {}: This is the placeholder that `parallel` replaces with the line from the input (our URL).

Which One Should You Use?

Here’s how I decide which tool to pull out of my toolbox.

At the end of the day, a simple task like “download files from a list” separates the rookies from the seasoned engineers. It’s not about knowing the most obscure command, but about understanding the bottleneck—in this case, sequential I/O—and reaching for the right tool to solve it efficiently. Now go make your terminal work for you, not the other way around.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions