Crawlsmith

Academic Research MCP

MCP server for AI agents and a batch scraper: search papers across Europe PMC, PubMed, Crossref, arXiv and OpenAlex, resolve DOIs to abstracts and free PDFs, walk citations and search ClinicalTrials.gov.

What it does

This tool searches the world's biggest open research databases and returns clean, structured, ready-to-use records: title, authors, year, journal, abstract, citation count, open-access status and the best free PDF. Results from several databases are merged and de-duplicated by DOI, so one paper shows up once with the richest data from every source.

Two ways to use it:

  1. MCP server for AI agents: connect Claude Desktop, Claude Code, ChatGPT, Cursor, Windsurf or your own agent. The agent picks the right tool by itself and gets real citations instead of guessing.
  2. Classic run: choose a tool, type a query, hit Start, and download the results as JSON, CSV or Excel.

Who uses it

Use Case How It Helps
AI agent builders Ground answers in real papers with DOIs and links, so your assistant stops inventing references
Researchers & PhD students Literature reviews in minutes: search five databases at once and get abstracts plus free PDFs
Pharma & biotech teams Track competitor trials by drug, condition, sponsor, phase and recruitment status
Market & competitive intelligence Monitor what is being published and trialled in your therapy area or technology
Science writers & journalists Find the original study behind a headline and check how often it has been cited
Medical affairs & HEOR Pull PubMed evidence with MeSH queries and link it to the trials behind it

Input

22 options. Required ones are marked.

Option What it does
What do you want to do? required The search or lookup to run.
Search query Keywords.
Max results How many results to return (search_papers up to 50, other searches up to 100).
Databases (search papers) Which databases search_papers queries.
Published from (year) Only works published in or after this year.
Published to (year) Only works published in or before this year.
Open access only Return only works with free full text.
Sort by Most cited works first is available for search_papers only.
arXiv category For search_arxiv, e.g. cs.CL, cs.LG, q-bio.GN.
DOI For get_paper / get_citations, e.g. 10.1038/nature17946.
PubMed ID For get_paper / get_citations, e.g. 27096365.
arXiv ID For get_paper, e.g. 1706.03762.
Citations direction For get_citations.
Condition (clinical trials) Disease or condition, e.g. type 2 diabetes.

Plus 8 more options. Full input reference on Apify.

Output fields

18 fields, grouped the way the dataset views show them. Export to JSON, CSV, Excel or pull them through the API.

Papers

  • Title
  • Authors
  • Year
  • Venue
  • DOI
  • Citations
  • Open access
  • PDF
  • Link
  • Source

Clinical trials

  • NCT ID
  • Title
  • Status
  • Phases
  • Conditions
  • Sponsor
  • Enrollment
  • Start
  • Countries
  • Link

Example output

One record, shortened. Real runs return one row like this per result.

{
  "source": "crossref",
  "sourceId": "10.1038/nature17946",
  "title": "Programmable editing of a target base in genomic DNA without double-stranded DNA cleavage",
  "authors": [
    "Alexis C. Komor",
    "Yongjoo B. Kim",
    "Michael S. Packer"
  ],
  "year": 2016,
  "publishedDate": "2016-04-20",
  "venue": "Nature",
  "type": "journal-article",
  "abstract": "Current genome-editing technologies introduce double-stranded (ds) DNA breaks at a target locus as the first step to gene correction...",
  "doi": "10.1038/nature17946",
  "pmid": "27096365",
  "pmcid": "PMC4873371",
  "arxivId": null,
  "citationCount": 5661,
  "isOpenAccess": true,
  "pdfUrl": "https://www.nature.com/articles/nature17946.pdf",
  "url": "https://doi.org/10.1038/nature17946",
  "keywords": [
    "CRISPR and Genetic Engineering",
    "RNA regulation and disease",
    "DNA Repair Mechanisms"
  ],
  "foundIn": [
    "crossref",
    "europepmc",
    "openalex"
  ],
  "tool": "get_paper",
  "scrapedAt": "2026-11-16T12:00:00.000Z"
}

How to run it

  1. 1. Open the scraper on Apify and sign in (free plan works).
  2. 2. Fill in the input: search terms, URLs or filters, and a result limit.
  3. 3. Run it, then download JSON, CSV or Excel, or pull the dataset through the API.
  4. 4. Schedule it to get fresh data daily or weekly.

FAQ

Which AI assistants work with it? +

Any client that speaks MCP over HTTP: Claude Desktop, Claude Code, ChatGPT connectors, Cursor, Windsurf, VS Code, n8n and custom agents.

Do I get full-text PDFs? +

You get the best free, legal PDF link when one exists (open-access journals, PubMed Central, arXiv). Paywalled papers come with their DOI link.

Do I need any API keys? +

No. Optional free OpenAlex and NCBI keys add OpenAlex results and faster PubMed searches.

Can I use PubMed's advanced syntax? +

Yes. MeSH terms, [ti], [au], [dp] and boolean operators all work in search_pubmed.

Is personal data exported? +

Paper author names are public bibliographic data. For clinical trials we deliberately leave out investigators' e-mail addresses and phone numbers.

Need something this scraper does not do?

Extra fields, a different filter, a recurring feed into your database or a one-off bulk export.

Ask for a custom version

Related scrapers