Struggling with a slow product catalog built on a tangled mess of databases and APIs? A Senior DevOps Lead breaks down three real-world architectural patterns to fix it, from a quick caching fix to a full-blown headless rebuild.

That Reddit Thread on Product Catalogs? I’ve Lived It. Here’s How We Fixed It.

I remember it was a Tuesday. 3 AM. A pager alert screamed me awake. The e-commerce site was down. Again. The culprit? The product catalog. A monolithic Postgres instance, prod-db-01, was choking under the load of a marketing campaign, trying to serve up thousands of products with complex filters. The marketing team was furious, the devs were pointing fingers, and I was staring at a psql prompt, realizing we had built a sports car on a lawnmower engine. That Reddit thread hit a little too close to home for me, so I felt I had to write this down.

The “Why”: You’re Asking One System to Do Two Wildly Different Jobs

Here’s the thing we all get wrong at the start: we think of the product catalog as a single problem. It’s not. It’s two distinct problems masquerading as one:

• The System of Record (SOR): This is where your business team manages products. It needs to be accurate, transactional, and consistent. It’s optimized for writes and updates. Think of your ERP, a custom admin panel writing to a SQL database, etc.
• The Query & Discovery System: This is what your customers hit. It needs to be blazing fast, handle massive read traffic, and support complex filtering, faceting, and full-text search. It’s optimized for reads.

The core issue is trying to use your SOR database (like Postgres or MySQL) to also power your customer-facing search and discovery. It will eventually fall over. It’s not designed for that kind of load or query complexity.

So, how do we fix it? We choose a strategy that fits our scale, budget, and timeline.

Solution 1: The Quick Fix (The “Duct Tape & Cache” Play)

This is the pragmatic, “we need the site to be up yesterday” solution. You keep your existing monolithic database but stick a powerful caching layer in front of it to absorb the read traffic.

How it works: Your application first checks a cache like Redis or Memcached for the product data. If it’s there (a cache hit), it returns it instantly. If not (a cache miss), it queries the slow primary database, gets the data, stores it in the cache for next time, and then returns it.

A simplified application logic might look like this:

function getProduct(productId) {
    // Generate a unique key for this product
    const cacheKey = `product:${productId}`;

    // 1. Try to get it from Redis first
    const cachedProduct = redis.get(cacheKey);

    if (cachedProduct) {
        // Cache HIT! Return it fast.
        return JSON.parse(cachedProduct);
    } else {
        // Cache MISS. Go to the slow database.
        const productFromDb = postgres.query('SELECT * FROM products WHERE id = ?', productId);

        // 2. Store the result in Redis for next time. Set a Time-To-Live (TTL) of 1 hour.
        redis.set(cacheKey, JSON.stringify(productFromDb), 'EX', 3600);

        return productFromDb;
    }
}

Warning: Cache invalidation is one of the two hard problems in computer science. When a product’s price or name is updated in the database, how do you ensure the old data is removed from the cache? You can set short TTLs (Time-To-Live) and live with slightly stale data, or build a more complex eventing system to bust the cache on every update. This approach buys you time, it doesn’t solve the root problem.

Solution 2: The Permanent Fix (The “Search-First” Architecture)

This is my recommended approach for 90% of growing e-commerce or catalog-heavy applications. You formally separate the read and write concerns by introducing a dedicated search engine.

How it works: Your primary database (e.g., prod-catalog-db-01) remains the source of truth for all product data. However, whenever data is created or updated in that database, a process (either a cron job or an event-driven hook) pushes that data into a dedicated search engine like Elasticsearch or Algolia. Your public-facing website/API only ever queries the search engine, never the primary database.

• Writes/Updates: App -> PostgreSQL -> Asynchronous Indexing Job -> Elasticsearch
• Reads/Searches: Customer Browser -> App API -> Elasticsearch

This gives you the best of both worlds: the transactional integrity of a SQL database and the insane read performance and powerful query capabilities of a real search engine. You can do faceting, filtering, type-ahead search, and more, with sub-50ms response times.

A query to Elasticsearch for “leather boots” in size 10 looks nothing like SQL, and that’s a good thing:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "leather boots" } }
      ],
      "filter": [
        { "term": { "variants.size": "10" } },
        { "term": { "in_stock": true } }
      ]
    }
  }
}

Solution 3: The ‘Nuclear’ Option (The “Headless Microservice” Rebuild)

This is for when you’ve hit massive scale or your product catalog is so complex it needs its own dedicated team. You break the product catalog out into its own standalone microservice, completely decoupled from the rest of your monolith.

How it works: The “Product Catalog Service” owns its own database, its own API, and its own infrastructure. It doesn’t share a database with Billing, Users, or Orders. It communicates with other services asynchronously via an event bus like Kafka or RabbitMQ. When a price is updated, it doesn’t update a column in a shared DB; it publishes a ProductPriceUpdated event to a Kafka topic. Other services, like the “Checkout Service,” listen to these events and react accordingly.

This pattern is powerful but introduces significant operational complexity. You’re no longer debugging a single application; you’re debugging a distributed system. You need robust monitoring, tracing, and a dedicated platform engineering team to manage the infrastructure.

Pro Tip: Don’t even think about this unless you’re hitting the scaling limits of Solution 2 and have the engineering maturity to manage it. Adopting microservices before you need them is a classic case of premature optimization and a recipe for headaches.

Which Path Is Right For You?

There’s no single right answer, only a series of trade-offs. Here’s how I break it down for my teams:

My advice? Start where you are. If you’re feeling the pain, the “Cache” play can buy you a few months. But start planning your migration to a proper “Search-First” architecture immediately. It will save you from that 3 AM pager alert. Trust me.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions