Crawlsmith

SimilarWeb Scraper

Scrape SimilarWeb data for any domain: global/country/category rank, monthly visits, traffic sources, top countries, top keywords, engagement (bounce rate, time on site), competitors, and AI traffic details.

What it does

This scraper automatically pulls SimilarWeb's domain analytics for any website you give it and returns structured, ready-to-use data. No manual browsing through dashboards — just paste your list of domains and hit Start.

You get global, country, and category rankings, three months of monthly visit history, the full traffic source breakdown (direct / search / social / referrals / mail / paid), top countries, top SEO keywords, competitors, and the new AI Traffic split (which chatbots are sending people to the site).


Who uses it

Use Case How It Helps
SEO Agencies Audit thousands of client and competitor domains in minutes — rank, keywords, traffic sources
Sales & Lead Gen Teams Score and prioritize prospects by real traffic and rank instead of guessing
Market Researchers Track market share across an industry with bulk traffic and engagement data
E-commerce Owners See exactly how much traffic competitors get and where it comes from
Investors & VCs Validate startup growth claims with independent traffic and engagement metrics
Partnership & BD Teams Find high-traffic partners and qualify opportunities with hard numbers

Input

6 options. Required ones are marked.

Option What it does
Domains List of domains to look up on SimilarWeb (e.g. apify.com, stackoverflow.com).
Start URLs (optional) Alternative to Domains - paste similarweb.com/website// URLs or just the website URLs.
Max Concurrency How many domains to fetch in parallel.
Max Retries Per Domain How many times to retry (with proxy rotation) when SimilarWeb blocks or rate-limits a request.
Request Timeout (seconds) Per-request timeout.
Proxy Configuration Proxy settings.

Output fields

32 fields, grouped the way the dataset views show them. Export to JSON, CSV, Excel or pull them through the API.

Overview

  • Domain
  • Category
  • Global Rank
  • Country Rank
  • Country
  • Monthly Visits
  • Bounce Rate %
  • Pages/Visit
  • Avg Visit (s)
  • Snapshot Date

Traffic Sources

  • Domain
  • Direct %
  • Search %
  • Social %
  • Mail %
  • Referrals %
  • Paid Referrals %
  • Monthly Visits

Rankings

  • Domain
  • Global Rank
  • Country Rank
  • Country
  • Category Rank
  • Category

Full Details

  • Domain
  • Site Name
  • Title
  • Description
  • Category
  • Snapshot Date
  • Global Rank
  • Country Rank
  • Country
  • Category Rank
  • Category Path
  • Monthly Visits
  • monthlyVisitsHistory
  • Stats Month
  • Bounce Rate %
  • Pages/Visit
  • Avg Visit (s)
  • Direct %
  • Search %
  • Social %
  • Mail %
  • Referrals %
  • Paid Referrals %
  • topCountries
  • topKeywords
  • competitors
  • AI Total Visits
  • AI Referral %
  • aiTrafficSplit
  • Screenshot
  • SimilarWeb URL
  • Scraped At

Example output

One record, shortened. Real runs return one row like this per result.

{
  "domain": "stackoverflow.com",
  "siteName": "stackoverflow.com",
  "title": "Newest 'postgresql' Questions",
  "description": "Stack Overflow | The World's Largest Online Community for Developers",
  "category": "computers_electronics_and_technology/programming_and_developer_software",
  "largeScreenshot": "https://site-images.similarcdn.com/image?url=stackoverflow.com&t=1&s=1&h=04179ebf",
  "snapshotDate": "2026-04-01T00:00:00+00:00",
  "globalRank": 2240,
  "countryRank": 2170,
  "countryRankCountryCode": "US",
  "categoryRank": 63,
  "categoryRankCategory": "Computers_Electronics_and_Technology/Programming_and_Developer_Software",
  "globalCategoryRank": null,
  "monthlyVisits": 31837008,
  "engagementMonth": "2026-04",
  "bounceRate": 70.26,
  "pagesPerVisit": 1.69,
  "avgVisitDurationSeconds": 141.72,
  "monthlyVisitsHistory": [
    {
      "month": "2026-02",
      "visits": 42405407
    },
    {
      "month": "2026-03",
      "visits": 38044480
    },
    {
      "month": "2026-04",
      "visits": 31837008
    }
  ],
  "trafficSourceDirect": 7.01,
  "trafficSourceSearch": 85.6,
  "trafficSourceSocial": 1.52,
  "trafficSourceMail": 0.24,
  "trafficSourceReferrals": 5.21,
  "trafficSourcePaidReferrals": 0.03,
  "topCountries": [
    {
      "countryCode": "US",
      "countryNumeric": 840,
      "share": 22.21
    },
    {
      "countryCode": "IN",
      "countryNumeric": 356,
      "share": 8.07
    },
    {
      "countryCode": "DE",
      "countryNumeric": 276,
      "share": 7.29
    }
  ],
  "topKeywords": [
    {
      "keyword": "stack overflow",
      "estimatedValue": 149400,
      "volume": 149150,
      "cpc": 1.42
    },
    {
      "keyword": "stackoverflow",
      "estimatedValue": 43680,
      "volume": 48040,
      "cpc": 1.9
    }
  ],
  "competitors": [],
  "aiTotalVisits": 184250.5,
  "aiReferralTraffic": 0.0058,
  "aiTrafficSplit": [
    {
      "chatbot": "chatgpt.com",
      "rank": 1
    },
    {
      "chatbot": "perplexity.ai",
      "rank": 2
    },
    {
      "chatbot": "gemini.google.com",
      "rank": 3
    }
  ],
  "isSmall": false,
  "isDataFromGa": false,
  "url": "https://www.similarweb.com/website/stackoverflow.com/",
  "scrapedAt": "2026-05-06T20:35:00.000Z"
}

How to run it

  1. 1. Open the scraper on Apify and sign in (free plan works).
  2. 2. Fill in the input: search terms, URLs or filters, and a result limit.
  3. 3. Run it, then download JSON, CSV or Excel, or pull the dataset through the API.
  4. 4. Schedule it to get fresh data daily or weekly.

FAQ

Do I need a SimilarWeb account? +

No. You don't need any login, paid plan, or API key — just paste your list of domains and run.

What domains work? +

Anything SimilarWeb has data for. Popular sites return rich data; very small or new domains may have partial data or none at all.

How fresh is the data? +

SimilarWeb refreshes its dataset monthly — every record includes a snapshotDate and engagementMonth so you always know exactly which month the numbers represent.

What about subdomains and paths? +

SimilarWeb works at the registered domain level. Subdomains and paths in your input are normalized to the root domain automatically.

Can I scrape thousands of domains? +

Yes. The scraper processes domains in parallel (2 at a time by default, up to 50) with smart retries, so even very large lists complete in minutes — not hours.

Can I export the data? +

Yes — JSON, CSV, Excel, XML, HTML, RSS. You can also push data directly to Google Sheets, Zapier, Make, or any webhook/API endpoint.

How often should I run this? +

For fresh data, run monthly (right after SimilarWeb's data refresh). You can schedule automatic runs on Apify with just a few clicks.

Does it work with proxies? +

Yes — the scraper uses Apify residential proxies out of the box. SimilarWeb rate-limits datacenter IPs aggressively, so residential is required for reliable results.


Need something this scraper does not do?

Extra fields, a different filter, a recurring feed into your database or a one-off bulk export.

Ask for a custom version

Related scrapers