Listopedia

Scrapy

Open-source Python framework for crawling and extraction

Overview

Scrapy is an open-source web scraping and crawling framework for Python, released under the BSD licence and maintained by Zyte together with more than 500 other contributors. The project describes itself as having run in production for over fifteen years and calls itself the most-used open-source data extraction framework, a claim the project makes about itself and that was not independently measured here.[7] Scrapy is a library rather than a service: it supplies request scheduling, parsing and export pipelines, while proxies, browser rendering and hosting must be provided separately, for example through a managed API such as Zyte API.[6][7]

Scrapy strengths and limitations

Strengths

  • No licence cost
  • Mature, widely used framework
  • Extensible pipelines and middleware

Limitations

  • Requires Python programming
  • No built-in proxy rotation or hosted execution

Appears in

Scrapy alternatives and competitors

Alternatives to Scrapy: companies ranked in the same lists.

Is this your company?

Update the description, logo, image and link through the submission form. Editors review every change.

Update this listing

Is your company missing from a list?

Send your logo, description and link. Editors review every submission, and approved listings are clearly labelled.

Add your company