ScienceDirect Search Scraper
Extract academic articles from ScienceDirect.com — title, DOI, authors, journal, abstract, keywords, dates, open-access status and PDF links. Search by keyword, author, journal, year and article type, or paste a search URL. Ideal for literature reviews and bibliometrics.
What it does
This scraper automatically searches ScienceDirect and collects structured, ready-to-use data for every matching article in the search results. No manual browsing, copy-pasting or page-flipping — just set your filters and hit Start.
Two ways to run it:
- Discovery Mode — Enter keywords plus optional filters (author, journal, year, article type, open access) and the scraper builds the search, pages through every result and collects the articles.
- Direct URL Mode — Already have a ScienceDirect search you like? Paste its search URL and the scraper pages through every matching result for you.
Who uses it
| Use Case | How It Helps |
|---|---|
| Researchers & PhD students | Build a complete corpus for a systematic literature review without hours of manual searching |
| Bibliometrics & scientometrics teams | Export thousands of citation records for trend, co-author and impact analysis |
| AI / NLP engineers | Assemble large metadata datasets to train and evaluate models |
| Pharma & R&D analysts | Track new publications in a field, journal or by a competitor's authors |
| Librarians & info specialists | Deliver curated reading lists and current-awareness alerts to faculty |
| Science writers & journalists | Find the latest open-access papers on any topic, fast |
Input
23 options. Required ones are marked.
| Option | Type | What it does |
|---|---|---|
| Search keywords | Text | Keywords to search across title, abstract and author-specified keywords (the main ScienceDirect search box). |
| Title contains | Text | Restrict to articles whose title contains these words. |
| Author(s) | Text | Filter by author name, e.g. "Y. |
| Journal or book title | Text | Restrict to a specific journal, book or publication title, e.g. "The Lancet". |
| Volume(s) | Text | Filter by journal volume(s), e.g. "75" or "75,76". |
| Issue | Text | Filter by issue number. |
| Page | Text | Filter by article page number. |
| Year(s) | Text | Filter by publication year(s). |
| Article types | List | Limit to one or more article types. |
| Open access & open archive only | Yes / no | Return only articles that are open access or in an open archive (freely readable). |
| Sort results by | Text | Sort search results by relevance (default) or newest first. |
| Start URLs (search URLs) | List | Optional. |
| Include abstracts | Yes / no | Add the abstract and the author highlights to every article. |
| Also collect keywords, affiliations and all authors | Yes / no | Open every article page to add author-specified keywords, author affiliations and the complete author list — the fields the abstract endpoint does not carry. |
Plus 9 more options. Full input reference on Apify.
Output fields
23 fields, grouped the way the dataset views show them. Export to JSON, CSV, Excel or pull them through the API.
Overview
- Title
- Authors
- Journal
- Published
- Open Access
- DOI
- URL
Bibliographic
- Title
- Authors
- Journal
- ISSN
- Volume/Issue
- Pages
- Type
- Published
- Online
- DOI
Full Details
- title
- doi
- authors
- journalName
- journalUrl
- issn
- articleType
- contentType
- volumeIssue
- pages
- publishedDate
- onlineDate
- isOpenAccess
- language
- pii
- pdfUrl
- articleUrl
- searchUrl
- scrapedAt
Abstracts
- Title
- Abstract
- Keywords
- Highlights
- Authors
- Affiliations
- DOI
- URL
Example output
One record, shortened. Real runs return one row like this per result.
{
"doi": "10.1016/j.jhsa.2025.01.021",
"title": "A Machine Learning Algorithm to Estimate the Probability of a True Scaphoid Fracture After Wrist Trauma",
"articleUrl": "https://www.sciencedirect.com/science/article/pii/S036350232500036X",
"pii": "S036350232500036X",
"articleType": "Research article",
"contentType": "JL",
"journalName": "The Journal of Hand Surgery",
"journalUrl": "https://www.sciencedirect.com/science/journal/03635023",
"issn": "03635023",
"publishedDate": "June 2025",
"onlineDate": "2025-06-03T00:00:00.000Z",
"volumeIssue": "Volume 50, Issue 6",
"pages": "711-720",
"authors": [
"Anne Eva J. Bulstra"
],
"language": "en",
"isOpenAccess": false,
"pdfUrl": "https://www.sciencedirect.com/science/article/pii/S036350232500036X/pdfft?pid=1-s2.0-S036350232500036X-main.pdf",
"searchUrl": "https://www.sciencedirect.com/search?qs=machine+learning&show=100",
"scrapedAt": "2026-07-01T08:10:34.590Z"
} How to run it
- 1. Open the scraper on Apify and sign in (free plan works).
- 2. Fill in the input: search terms, URLs or filters, and a result limit.
- 3. Run it, then download JSON, CSV or Excel, or pull the dataset through the API.
- 4. Schedule it to get fresh data daily or weekly.
FAQ
Can I scrape a search I already built on the website? +
Yes — just paste the ScienceDirect search URL into Start URLs and the scraper will page through every result for you.
What data do I get for each article? +
You get rich search-listing metadata — title, DOI, authors, journal, ISSN, volume/issue, pages, publication and online dates, article type, language, open-access status, and the article and PDF links. Switch on Include abstracts to add the abstract and the author highlights, and Keywords & affiliations for author-specified keywords, author affiliations and the complete author list.
Does collecting abstracts slow the run down? +
Barely — abstracts come from the same place the website's own "Abstract" preview button reads them, so a run with abstracts takes about as long as one without. Keywords and affiliations are the slow part, because those only exist on the article pages themselves.
Can it collect open-access PDFs? +
The scraper returns the direct PDF link for every article where one is available, and you can filter to open-access articles only.
How many articles can I collect? +
From a few up to thousands per run — set the Max items limit to whatever you need.
Can I export the data? +
Yes — JSON, CSV, Excel, XML, HTML, RSS. You can also push data directly to Google Sheets, Zapier, Make, or any webhook/API endpoint.
How often should I run this? +
For fresh data, run daily or weekly. You can schedule automatic runs on Apify with just a few clicks.
Does it work with proxies? +
Yes — residential proxies are recommended for the best results. The scraper works with Apify's built-in proxy service.
Need something this scraper does not do?
Extra fields, a different filter, a recurring feed into your database or a one-off bulk export.
Ask for a custom versionRelated scrapers
Doctoralia Scraper
Global
Scrape doctors from the whole Doctoralia network — doctoralia.es/.com.br/.com.mx and 9 more sites incl. ZnanyLekarz, MioDottore, Doktortakvimi and Jameda. Profiles, license numbers, services with prices, GPS locations and every patient review. Incremental monitoring for cheap recurring runs.
Posh.vip Scraper
Global
Scrape live events, organizers, tickets, performer lineups and venue details from Posh.vip. Supports event pages, organizer/group pages, and explore (marketplace) URLs with location-based filtering.
Bazaarvoice & PowerReviews Reviews + Q&A Scraper
Global
Reviews, Q&A and rating summaries from any shop on Bazaarvoice or PowerReviews: Target, Home Depot, Sephora, Costco, Lowe's, Boots, Argos, MediaMarkt, Ulta and thousands more. Paste a product URL; filter by stars, date, language, photos, verified buyers; only-new mode.
SimilarWeb Scraper
Global
Scrape SimilarWeb data for any domain: global/country/category rank, monthly visits, traffic sources, top countries, top keywords, engagement (bounce rate, time on site), competitors, and AI traffic details.