Competition Research With a Crawler and a Keyword API.

The team I was training had a folder of screenshots of a competitor’s pages and a shared belief that the competitor won on “content quality”. Nobody could say which of its several thousand URLs actually ranked, or whether the pages they were admiring were the ones bringing visitors. The practice domain was a large public website in the tax-filing space, chosen because it has clear templates, a lot of rankings in India, and nobody in the room worked for it. The method below took an afternoon, cost under a few hundred rupees in API credit, and ended with a table that showed the rankings sat on two templates the team had not looked at once.

You need a licensed copy of Screaming Frog SEO Spider (the free version stops at 500 URLs and does not run headless), a DataForSEO account with a small credit balance, and either Python 3 with the requests and pandas packages or a spreadsheet you are comfortable doing lookups in. Replace competitor.example with the domain you are studying throughout.

Step 1. Pick a competitor that ranks for what you want

The domain to study is the one that appears in the results you want, not the one your client names as a rival. Search five or six of your target queries from a clean browser profile with the location set to India and note which domains appear for most of them. A domain that shows up on four of six queries is a better teacher than a famous brand that shows up on one. For the session we picked the tax site because it appeared for calculator queries, form-name queries and glossary-style “what is” queries, which is the mix the team wanted.

If you want a number rather than a feeling, the API in Step 3 will give you a count of ranking keywords for any domain, so you can run it against three candidates with limit set to 1 and compare the total_count field before spending on a full pull.

Step 2. Crawl the competitor with Screaming Frog

Set the configuration in the user interface first, save it as a file, and then run the crawl headless so it can be repeated later without clicking. Four settings matter.

Under Configuration, Spider, Crawl, leave Respect robots.txt on (it is the default under Configuration, robots.txt) and untick Images, CSS, JavaScript and SWF so the crawl is HTML only. Under Configuration, Speed, set a limit of around 2 URLs per second; you are a guest on someone else’s server and a fast crawl of a large site is the quickest way to get your IP blocked before the crawl finishes. Under Configuration, Custom, Extraction, add two extractors, both set to XPath, extract Text.

If the body class is empty on the competitor, try //meta[@property='og:type']/@content or a distinctive element such as //form[contains(@class,'calculator')]/@class. The point is to capture one thing per page that identifies the template. Save the configuration with File, Configuration, Save As, and call it competitor.seospiderconfig.

Then run headless. On macOS:

/Applications/Screaming\ Frog\ SEO\ Spider.app/Contents/MacOS/ScreamingFrogSEOSpiderLauncher \
  --crawl https://competitor.example/ \
  --headless \
  --config "$HOME/sf/competitor.seospiderconfig" \
  --save-crawl \
  --output-folder "$HOME/sf/out" \
  --overwrite \
  --export-format csv \
  --export-tabs "Internal:HTML,Custom Extraction:All"

On Linux the binary is screamingfrogseospider and the arguments are the same. The Internal:HTML filter exports one row per HTML page with the Address, Status Code, Title, H1 and Word Count columns, and Custom Extraction:All exports the two extractors keyed by Address. When the crawl finishes you should see internal_html.csv and custom_extraction_all.csv in the output folder. Check the row count against what you expected before moving on, because a site that blocks you halfway through produces a clean-looking export with half the URLs missing.

wc -l "$HOME/sf/out/internal_html.csv"

The tax site came back at a little over 6,000 HTML URLs in about 50 minutes at 2 URLs per second.

Step 3. Pull the domain’s ranking keywords from the API

DataForSEO Labs has an endpoint that returns every keyword its index has seen the domain rank for, together with the URL that ranks. The endpoint is POST https://api.dataforseo.com/v3/dataforseo_labs/google/ranked_keywords/live. Authentication is HTTP Basic with your API login and password. The parameters that matter for this job are target (the bare domain, without https:// or www), location_code (2356 for India [VERIFY: current code in the Labs locations list]), language_code (en), limit (maximum 1000 per call) and offset for paging. Filters are optional; I usually add one so I only get positions 1 to 30, since a keyword ranking at 87 tells you nothing about the template.

import base64
import json
import time
import requests

LOGIN = "YOUR_DATAFORSEO_LOGIN"
PASSWORD = "YOUR_DATAFORSEO_PASSWORD"
TARGET = "competitor.example"
URL = "https://api.dataforseo.com/v3/dataforseo_labs/google/ranked_keywords/live"

auth = base64.b64encode(f"{LOGIN}:{PASSWORD}".encode()).decode()
headers = {"Authorization": f"Basic {auth}", "Content-Type": "application/json"}

rows = []
offset = 0
while True:
    payload = [{
        "target": TARGET,
        "location_code": 2356,
        "language_code": "en",
        "limit": 1000,
        "offset": offset,
        "filters": ["ranked_serp_element.serp_item.rank_group", "<=", 30],
        "order_by": ["ranked_serp_element.serp_item.etv,desc"],
    }]
    r = requests.post(URL, headers=headers, data=json.dumps(payload), timeout=120)
    r.raise_for_status()
    task = r.json()["tasks"][0]
    if task["status_code"] != 20000:
        raise SystemExit(f"API error {task['status_code']}: {task['status_message']}")
    result = task["result"][0]
    items = result.get("items") or []
    for it in items:
        kd = it["keyword_data"]
        si = it["ranked_serp_element"]["serp_item"]
        rows.append({
            "keyword": kd["keyword"],
            "search_volume": kd["keyword_info"]["search_volume"],
            "rank_group": si["rank_group"],
            "rank_absolute": si["rank_absolute"],
            "url": si["url"],
            "etv": si["etv"],
        })
    print(f"offset {offset}: {len(items)} items, total_count {result['total_count']}")
    if len(items) < 1000:
        break
    offset += 1000
    time.sleep(1)

with open("ranked_keywords.json", "w") as f:
    json.dump(rows, f)

The fields you keep are the keyword, its monthly search volume, the rank, the ranking URL and etv, which is DataForSEO’s estimated monthly traffic to that URL from that keyword. Print total_count on the first call and decide whether you need all of it. On the tax site the count in the top 30 for India was in the tens of thousands, so we paged through to about 20,000 rows, which was enough to see the shape.

Two operational notes. The documentation states a limit of 2,000 API calls a minute and 30 simultaneous calls, which you will not approach with a sequential loop, but the one-second sleep is still polite. Cost for this endpoint is billed per request plus a per-row component [VERIFY: current price per request and per returned row on the DataForSEO pricing page], and the account needs a minimum top-up of 50 US dollars, so budget in rupees at the day’s rate and check the balance in the dashboard before running a large domain.

The same endpoint also accepts a full page URL as target if you include https://, which returns only the keywords that page ranks for. That is useful in Step 6 when you want to inspect one representative page of a template without pulling the whole domain again.

Step 4. Join keywords to crawled URLs

You now have two files keyed on URL, and they will not match cleanly. The crawl records URLs as the server returned them; the API records them as Google indexed them. Trailing slashes, uppercase in paths, http versus https and tracking parameters all break a naive lookup. Normalise both sides the same way before joining.

import pandas as pd
from urllib.parse import urlsplit, urlunsplit

def norm(u):
    p = urlsplit(str(u).strip())
    path = p.path.rstrip("/") or "/"
    return urlunsplit(("https", p.netloc.lower().removeprefix("www."), path.lower(), "", ""))

crawl = pd.read_csv("out/internal_html.csv")
crawl = crawl[crawl["Status Code"] == 200][["Address", "Title 1", "H1-1", "Word Count"]]
crawl["key"] = crawl["Address"].map(norm)

ext = pd.read_csv("out/custom_extraction_all.csv")[["Address", "h1_text 1", "template_hint 1"]]
ext["key"] = ext["Address"].map(norm)

kw = pd.DataFrame(json.load(open("ranked_keywords.json")))
kw["key"] = kw["url"].map(norm)

pages = crawl.merge(ext.drop(columns="Address"), on="key", how="left")
joined = kw.merge(pages, on="key", how="left", indicator=True)
print(joined["_merge"].value_counts())

The _merge column tells you how many ranking URLs were found in the crawl. Expect 85 to 95 percent to match. A large unmatched share means either the crawl was incomplete or the site ranks with URLs it no longer links internally, both of which are findings in themselves. Save the unmatched rows and look at a sample of twenty before you trust the rest.

In a spreadsheet the same join is a VLOOKUP or XLOOKUP on a normalised URL column, with the normalisation done by LOWER and SUBSTITUTE formulas, which is fine for a few thousand rows and painful past that.

Step 5. Group URLs by template and sum what they carry

A template on most sites shows up as a URL folder, and where it does not, the body class from the custom extraction fills the gap. Take the first path segment as the pattern and aggregate.

def pattern(key):
    parts = urlsplit(key).path.strip("/").split("/")
    return "/" + parts[0] + "/" if parts[0] else "/"

joined["pattern"] = joined["key"].map(pattern)
by_template = (joined.groupby("pattern")
    .agg(urls=("key", "nunique"),
         keywords=("keyword", "nunique"),
         est_traffic=("etv", "sum"),
         top3=("rank_group", lambda s: (s <= 3).sum()))
    .sort_values("est_traffic", ascending=False))
print(by_template.head(15))

If the first folder is too coarse (some sites put everything under /blog/), group on template_hint instead, or on a regex over the path such as -calculator$ or ^/glossary/. Add the crawl’s urls per pattern so you can compute keywords per URL, which is the number that separates a template that works from one that merely has many pages.

On the tax site the top of the table looked like this, with figures rounded and the folder names paraphrased.

Forty-one calculator pages carried more estimated traffic than the other 5,000 URLs combined, and the blog the team had been screenshotting was the weakest template on the site by every measure.

Step 6. Write down what to copy and what to leave

Open three or four pages from each of the top two templates and record the structure, not the words. For the calculator template on the tax site that meant an H1 naming the calculation, the input form above the fold, a short “how this is calculated” section with the formula, an FAQ block with four to six questions, and a strip of links to related calculators and to the glossary terms used on the page. The glossary template was shorter, with a definition in the first paragraph, a worked example in rupees, and links back to the calculators that used the term, which is how the two templates fed each other rankings.

Run the page-level version of the Step 3 call on one calculator URL to confirm what a single page ranks for. You will usually find one head term and a long tail of phrasings that the FAQ block answers, which tells you the FAQ is doing real work and is worth carrying over.

What to copy is the template structure, the section order, the internal link pattern between templates, and the decision to build a page per calculation and per term. What not to copy is the copy itself, the exact keyword targets (you will pick your own from your own keyword data, since ranking for the competitor’s tail is not the goal), and any page count for its own sake. The tax site’s 2,900 blog posts are a warning, not a model.

Put the findings in one page with the table from Step 5, a screenshot of each winning template with the sections labelled, and a list of five things to build first. That page is the deliverable; the crawl and the API export are working files.

Check it worked

You have finished when you can produce this table for the competitor, with real numbers in every cell, and point to the template that carries the most estimated traffic per URL.

Three checks on the way. The crawl export row count should be close to the site’s sitemap count or to a site: search estimate; a large shortfall means the crawl was cut off. The join match rate from Step 4 should be above 85 percent. And the sum of est_traffic across all templates should be within a sensible distance of the domain-level traffic estimate the API returns in the metrics object of the first call, since both come from the same index and a large gap means rows were dropped in the join.

Where I could be wrong

The estimated traffic figures are a model, not analytics. DataForSEO computes etv from search volume and an assumed click curve, so treat the numbers as a ranking of templates against each other and not as visitors the competitor actually receives. Two templates within 20 percent of each other are tied.

Folder-based grouping fails on sites whose URLs are flat, which is common on Indian news and finance sites where every page sits at the root. On those the body-class extractor is the only template signal, and if it is empty you will need a more creative XPath, such as the presence of a specific widget, and a second crawl to collect it.

The keyword index is a snapshot and its coverage of Hindi and Hinglish queries is thinner than its coverage of English, so a competitor whose strength is in vernacular search will look weaker in this method than it is. Cross-check with a few manual searches from a phone with the language set to Hindi.

Screaming Frog’s export column names (H1-1, Title 1, and the 1 suffix on custom extractors) change between versions. If the pandas code raises a KeyError, print crawl.columns and adjust the names rather than assuming the join is wrong.

Sources

• Screaming Frog SEO Spider user guide, command line interface
• Screaming Frog SEO Spider user guide, custom extraction
• DataForSEO Labs, Google Ranked Keywords (live) endpoint documentation
• DataForSEO Labs, supported locations and languages
• DataForSEO pricing
• Python requests documentation
• pandas merge documentation

How to cite this guide