Quick Summary: relying solely on Google Search Console query overlap misses the semantic conflicts that actually confuse search engines; implementing vector similarity allows you to detect intent-based cannibalisation before it tanks your rankings.

Beyond Exact Match: Detecting Keyword Cannibalisation with Vector Similarity

I still remember the deployment that almost got me fired from a consulting gig back in 2019. We had just migrated a massive documentation portal for a SaaS client to a shiny new headless CMS. Two weeks later, traffic on their core “API Integration” segment dropped by 40%. The marketing lead was screaming about lost keywords. I pulled the logs from prod-analytics-01 and looked at GSC. Nothing. Zero exact-match cannibalisation.

The problem? We had one page titled “API Setup Guide” and another legacy page titled “Integrating our API.” To a string-matching algorithm (and GSC’s basic overlap report), these were different topics. To Google’s BERT model (and any human with a pulse), they were the exact same thing. The search engine was flip-flopping between the two URLs, eventually deciding to rank neither. That’s when I realized: checking for string equality in a semantic world is like bringing a knife to a drone fight.

The Root Cause: Search Engines Don’t Read, They “Feel”

Here is the hard truth that most SEO tools gloss over: Google doesn’t store your content as words; it stores it as numerical vectors. When you rely exclusively on GSC query overlap, you are looking for instances where URL A and URL B rank for the exact string “best devops tools.”

But real cannibalisation happens at the intent level. If you have two pages that sit close to each other in vector space (high cosine similarity), they are fighting for the same oxygen, even if they don’t share a single keyword. You need to stop auditing keywords and start auditing semantic proximity.

The Fixes

So, does the vector approach make sense? Absolutely. But you don’t need to spin up a Kubernetes cluster just to test it. Here are three ways I’ve tackled this, ranging from “quick hack” to “production-grade architecture.”

Solution 1: The Quick Script (The “Local” Fix)

Before you go buying enterprise tools, run a sanity check. I use a simple Python script with sentence-transformers to audit critical site sections. This runs locally on my machine, scrapes the H1s and Meta Descriptions, generates embeddings, and flags anything with a similarity score over 0.85.

It’s hacky, but it works.

import pandas as pd
from sentence_transformers import SentenceTransformer, util

# Load a lightweight model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Mock data from your CMS export
pages = [
    {'url': '/blog/docker-guide', 'title': 'Complete Guide to Docker Containers'},
    {'url': '/blog/containerization-101', 'title': 'Intro to Containerization'},
    {'url': '/blog/k8s-vs-docker', 'title': 'Kubernetes vs Docker'}
]

# Generate embeddings
titles = [p['title'] for p in pages]
embeddings = model.encode(titles, convert_to_tensor=True)

# Compute cosine similarity
cosine_scores = util.cos_sim(embeddings, embeddings)

# Find pairs with > 0.85 similarity (excluding self-matches)
# In production, we iterate through the tensor to find conflicts.
print("Checking for semantic conflicts...")

Pro Tip: Don’t embed the whole body content initially. It introduces too much noise. Start with the H1 and the Title Tag. If the titles are semantically identical, you have a problem regardless of the body text.

Solution 2: The Automated Pipeline (The Permanent Fix)

If you are managing a site with thousands of pages, running a script manually is technical debt. At TechResolve, we integrated this into our CI/CD pipeline for the marketing site.

We use a vector database (like Pinecone or Weaviate) to store the embeddings of our live sitemap. When a content writer commits a new Markdown file to the repo:

• The pipeline generates an embedding for the new title.
• It queries the vector DB for existing pages with similarity > 0.9.
• If a conflict is found, the PR build fails with a warning: “Cannibalisation Risk detected with /existing-page-url.”

Solution 3: The “Cluster and Kill” (The Nuclear Option)

Sometimes, the analysis reveals that your site architecture is fundamentally broken. You might find you have 15 distinct pages all trying to rank for variations of “cloud migration.”

In this scenario, vector analysis isn’t just for detection—it’s for cleanup. We perform K-Means clustering on the page vectors to group them into semantic topics. Once clustered, we pick the strongest URL (highest backlinks, best current traffic) and 301 Redirect all other cluster members to it.

This is drastic. You are deleting pages. But in my experience, consolidating 5 weak pages into 1 strong page often results in a traffic gain greater than the sum of the parts. Just make sure you update your internal links or nginx configs carefully, or you’ll create a redirect loop nightmare.

Darian Vance

Lead Cloud Architect & DevOps Strategist

With over 12 years in system architecture and automation, Darian specializes in simplifying complex cloud infrastructures. An advocate for open-source solutions, he founded TechResolve to provide engineers with actionable, battle-tested troubleshooting guides and robust software alternatives.

🤖 Frequently Asked Questions