Crawlsmith

ScienceDirect Search Scraper

Extract academic articles from ScienceDirect.com — title, DOI, authors, journal, abstract, keywords, dates, open-access status and PDF links. Search by keyword, author, journal, year and article type, or paste a search URL. Ideal for literature reviews and bibliometrics.

What it does

This scraper automatically searches ScienceDirect and collects structured, ready-to-use data for every matching article in the search results. No manual browsing, copy-pasting or page-flipping — just set your filters and hit Start.

Two ways to run it:

  1. Discovery Mode — Enter keywords plus optional filters (author, journal, year, article type, open access) and the scraper builds the search, pages through every result and collects the articles.
  2. Direct URL Mode — Already have a ScienceDirect search you like? Paste its search URL and the scraper pages through every matching result for you.

Who uses it

Use Case How It Helps
Researchers & PhD students Build a complete corpus for a systematic literature review without hours of manual searching
Bibliometrics & scientometrics teams Export thousands of citation records for trend, co-author and impact analysis
AI / NLP engineers Assemble large metadata datasets to train and evaluate models
Pharma & R&D analysts Track new publications in a field, journal or by a competitor's authors
Librarians & info specialists Deliver curated reading lists and current-awareness alerts to faculty
Science writers & journalists Find the latest open-access papers on any topic, fast

Input

23 options. Required ones are marked.

Option What it does
Search keywords Keywords to search across title, abstract and author-specified keywords (the main ScienceDirect search box).
Title contains Restrict to articles whose title contains these words.
Author(s) Filter by author name, e.g. "Y.
Journal or book title Restrict to a specific journal, book or publication title, e.g. "The Lancet".
Volume(s) Filter by journal volume(s), e.g. "75" or "75,76".
Issue Filter by issue number.
Page Filter by article page number.
Year(s) Filter by publication year(s).
Article types Limit to one or more article types.
Open access & open archive only Return only articles that are open access or in an open archive (freely readable).
Sort results by Sort search results by relevance (default) or newest first.
Start URLs (search URLs) Optional.
Include abstracts Add the abstract and the author highlights to every article.
Also collect keywords, affiliations and all authors Open every article page to add author-specified keywords, author affiliations and the complete author list — the fields the abstract endpoint does not carry.

Plus 9 more options. Full input reference on Apify.

Output fields

23 fields, grouped the way the dataset views show them. Export to JSON, CSV, Excel or pull them through the API.

Overview

  • Title
  • Authors
  • Journal
  • Published
  • Open Access
  • DOI
  • URL

Bibliographic

  • Title
  • Authors
  • Journal
  • ISSN
  • Volume/Issue
  • Pages
  • Type
  • Published
  • Online
  • DOI

Full Details

  • title
  • doi
  • authors
  • journalName
  • journalUrl
  • issn
  • articleType
  • contentType
  • volumeIssue
  • pages
  • publishedDate
  • onlineDate
  • isOpenAccess
  • language
  • pii
  • pdfUrl
  • articleUrl
  • searchUrl
  • scrapedAt

Abstracts

  • Title
  • Abstract
  • Keywords
  • Highlights
  • Authors
  • Affiliations
  • DOI
  • URL

Example output

One record, shortened. Real runs return one row like this per result.

{
  "doi": "10.1016/j.jhsa.2025.01.021",
  "title": "A Machine Learning Algorithm to Estimate the Probability of a True Scaphoid Fracture After Wrist Trauma",
  "articleUrl": "https://www.sciencedirect.com/science/article/pii/S036350232500036X",
  "pii": "S036350232500036X",
  "articleType": "Research article",
  "contentType": "JL",
  "journalName": "The Journal of Hand Surgery",
  "journalUrl": "https://www.sciencedirect.com/science/journal/03635023",
  "issn": "03635023",
  "publishedDate": "June 2025",
  "onlineDate": "2025-06-03T00:00:00.000Z",
  "volumeIssue": "Volume 50, Issue 6",
  "pages": "711-720",
  "authors": [
    "Anne Eva J. Bulstra"
  ],
  "language": "en",
  "isOpenAccess": false,
  "pdfUrl": "https://www.sciencedirect.com/science/article/pii/S036350232500036X/pdfft?pid=1-s2.0-S036350232500036X-main.pdf",
  "searchUrl": "https://www.sciencedirect.com/search?qs=machine+learning&show=100",
  "scrapedAt": "2026-07-01T08:10:34.590Z"
}

How to run it

  1. 1. Open the scraper on Apify and sign in (free plan works).
  2. 2. Fill in the input: search terms, URLs or filters, and a result limit.
  3. 3. Run it, then download JSON, CSV or Excel, or pull the dataset through the API.
  4. 4. Schedule it to get fresh data daily or weekly.

FAQ

Can I scrape a search I already built on the website? +

Yes — just paste the ScienceDirect search URL into Start URLs and the scraper will page through every result for you.

What data do I get for each article? +

You get rich search-listing metadata — title, DOI, authors, journal, ISSN, volume/issue, pages, publication and online dates, article type, language, open-access status, and the article and PDF links. Switch on Include abstracts to add the abstract and the author highlights, and Keywords & affiliations for author-specified keywords, author affiliations and the complete author list.

Does collecting abstracts slow the run down? +

Barely — abstracts come from the same place the website's own "Abstract" preview button reads them, so a run with abstracts takes about as long as one without. Keywords and affiliations are the slow part, because those only exist on the article pages themselves.

Can it collect open-access PDFs? +

The scraper returns the direct PDF link for every article where one is available, and you can filter to open-access articles only.

How many articles can I collect? +

From a few up to thousands per run — set the Max items limit to whatever you need.

Can I export the data? +

Yes — JSON, CSV, Excel, XML, HTML, RSS. You can also push data directly to Google Sheets, Zapier, Make, or any webhook/API endpoint.

How often should I run this? +

For fresh data, run daily or weekly. You can schedule automatic runs on Apify with just a few clicks.

Does it work with proxies? +

Yes — residential proxies are recommended for the best results. The scraper works with Apify's built-in proxy service.

Need something this scraper does not do?

Extra fields, a different filter, a recurring feed into your database or a one-off bulk export.

Ask for a custom version

Related scrapers