Ecosyste.ms: Packages

An open API service providing package, version and dependency metadata of many open source software ecosystems and registries.

pypi.org "scrapy" keyword

scrapy-pipelines 0.2
A collection of scrapy item pipelines
2 versions - Latest release: almost 5 years ago - 1 dependent repositories - 63 downloads last month - 16 stars on GitHub - 1 maintainer
abu-quant 0.0.1
股票量化
1 version - Latest release: about 7 years ago - 1 dependent repositories - 34 downloads last month - 1 maintainer
themispy 0.2.7
Componente do projeto Themis para extração de dados e armazenamento em nuvem.
43 versions - Latest release: over 1 year ago - 588 downloads last month - 0 stars on GitHub - 1 maintainer
scrapfly-sdk 0.8.16
Scrapfly SDK for Scrapfly
37 versions - Latest release: about 1 month ago - 2 dependent repositories - 13.1 thousand downloads last month - 18 stars on GitHub - 1 maintainer
scrapy-elves 0.1.0
utils for parse html
1 version - Latest release: 9 months ago - 1 dependent repositories - 1 maintainer
scrapy-impersonate 1.3.0 💰
Scrapy download handler that can impersonate browser fingerprints
8 versions - Latest release: 28 days ago - 10.2 thousand downloads last month - 61 stars on GitHub - 1 maintainer
scrapy-scrapingbee 0.0.5
JavaScript support and proxy rotation for Scrapy with ScrapingBee
5 versions - Latest release: over 2 years ago - 1 dependent repositories - 3.05 thousand downloads last month - 32 stars on GitHub - 2 maintainers
scrapy-dot-items 1.0.4
A Scrapy addon that allows to access arguments via the dot
10 versions - Latest release: almost 2 years ago - 1 dependent repositories - 56 downloads last month - 1 stars on GitHub - 1 maintainer
scrapy-vampire 0.1.0
utils for scrapy
1 version - Latest release: 9 months ago - 1 dependent repositories - 1 maintainer
scrapy-wayback 1.0.9
Scrapy middleware with wayback machine support for more robust scrapers.
10 versions - Latest release: over 2 years ago - 1 dependent repositories - 37 downloads last month - 1 stars on GitHub - 1 maintainer
spider-brew-kit 0.1.6
A library for scrapy tools, including but not limited to the usual pipelines, middlewares, etc.
6 versions - Latest release: 3 months ago - 42 downloads last month - 0 stars on GitHub - 1 maintainer
thisisapogreq 21.3.3 💰
Faster & simpler requests replacement for Python
1 version - Latest release: about 2 years ago - 1 dependent repositories - 17 downloads last month - 1,079 stars on GitHub - 1 maintainer
habra-favorites 2.0.0
Sort your favorites posts from Habrahabr.ru
16 versions - Latest release: 4 months ago - 2 dependent repositories - 63 downloads last month - 10 stars on GitHub - 1 maintainer
scrapy-pyh2m 0.1.0
simple and flexible html to markdown python converter pipeline with scrapy
1 version - Latest release: over 3 years ago - 1 dependent repositories - 16 downloads last month - 1 maintainer
scrapy-influxdb-exporter 1.0.6
A simple package to export Scrapy spider stats to InfluxDB
7 versions - Latest release: about 1 month ago - 109 downloads last month - 1 stars on GitHub - 1 maintainer
scrapymon 0.1.0
Simple management UI for scrapyd
1 version - Latest release: about 7 years ago - 1 dependent repositories - 14 downloads last month - 49 stars on GitHub - 1 maintainer
Top 7.1% on pypi.org
scrapyjs 0.1.1
JavaScript support for Scrapy using Splash
3 versions - Latest release: about 9 years ago - 1 dependent package - 20 dependent repositories - 122 downloads last month - 3,089 stars on GitHub - 4 maintainers
Top 1.5% on pypi.org
scrapy-splash 0.9.0
JavaScript support for Scrapy using Splash
11 versions - Latest release: over 1 year ago - 5 dependent packages - 429 dependent repositories - 111 thousand downloads last month - 3,089 stars on GitHub - 4 maintainers
scrapy-puppeteer-client 0.1.5
A library to use Puppeteer-managed browser in Scrapy spiders
11 versions - Latest release: 4 months ago - 1 dependent package - 1 dependent repositories - 205 downloads last month - 48 stars on GitHub - 2 maintainers
aio-scrapy 2.1.0
A high-level Web Crawling and Web Scraping framework based on Asyncio
39 versions - Latest release: about 1 month ago - 1 dependent repositories - 304 downloads last month - 51 stars on GitHub - 1 maintainer
ayugespidertools 3.9.7
scrapy 扩展库:用于扩展 Scrapy 功能来解放双手。
88 versions - Latest release: 2 months ago - 1 dependent repositories - 224 downloads last month - 58 stars on GitHub - 1 maintainer
scrapy-dynamic-spiders 1.0.0a1
Dynamically generate spider subclasses. Run crawls sequentially with crochet. Do both.
1 version - Latest release: almost 4 years ago - 1 dependent repositories - 16 downloads last month - 0 stars on GitHub - 1 maintainer
Top 4.6% on pypi.org
scutils 1.2.0
Utilities for Scrapy Cluster
15 versions - Latest release: about 7 years ago - 13 dependent repositories - 910 downloads last month - 1,157 stars on GitHub - 1 maintainer
hypotonic 0.0.15
Fast asynchronous web scraper with minimalist API.
15 versions - Latest release: over 3 years ago - 1 dependent repositories - 138 downloads last month - 8 stars on GitHub - 1 maintainer
Top 3.7% on pypi.org
itemadapter 0.9.0
Common interface for data container classes
18 versions - Latest release: 12 days ago - 17 dependent packages - 1,988 dependent repositories - 1.36 million downloads last month - 60 stars on GitHub - 3 maintainers
scrapy-db-pipeline 1.1
persist item to the database table
2 versions - Latest release: over 4 years ago - 1 dependent repositories - 11 downloads last month - 2 stars on GitHub - 1 maintainer
scrapyq 1.0.0
A library to filter SQLAlchemy queries.
1 version - Latest release: 8 months ago - 18 downloads last month - 2 stars on GitHub - 1 maintainer
spider-admin-pro 2.0.16
a spider admin based vue, scrapyd api and APScheduler
32 versions - Latest release: 3 days ago - 1 dependent repositories - 530 downloads last month - 498 stars on GitHub - 1 maintainer
scrapyd-egg-checksum 0.1.2
Get the checksum of eggs in case of building distributed scrapy clusters
2 versions - Latest release: over 4 years ago - 1 dependent repositories - 23 downloads last month - 0 stars on GitHub - 1 maintainer
Top 1.6% on pypi.org
scrapy-redis 0.7.3
Redis-based components for Scrapy.
18 versions - Latest release: almost 2 years ago - 6 dependent packages - 392 dependent repositories - 7.15 thousand downloads last month - 5,448 stars on GitHub - 3 maintainers
Top 5.9% on pypi.org
scrapy-mongodb 0.12.0
Pipeline to MongoDB for Scrapy. Supports MongoDB replica sets
22 versions - Latest release: over 6 years ago - 41 dependent repositories - 631 downloads last month - 355 stars on GitHub - 1 maintainer
scrapy-status-mailer 0.3
Scrapy Status Mailer: Status mailer extension for Scrapy
3 versions - Latest release: over 7 years ago - 1 dependent repositories - 26 downloads last month - 1 stars on GitHub - 1 maintainer
scrapy-tls-client 0.0.5
tls client downloader middleware for scrapy, send request by tls client.
4 versions - Latest release: 8 months ago - 99 downloads last month - 10 stars on GitHub - 1 maintainer
inspire-crawler 3.0.4
Crawler integration with INSPIRE-HEP.
33 versions - Latest release: almost 5 years ago - 6 dependent repositories - 157 downloads last month - 4 stars on GitHub - 2 maintainers
py-dictionary 4.1.4
Dictionary module
12 versions - Latest release: over 2 years ago - 1 dependent repositories - 288 downloads last month - 4 stars on GitHub - 1 maintainer
scrapy-link-filter 0.2.0
Scrapy Middleware that allows a Scrapy Spider to filter requests.
2 versions - Latest release: over 4 years ago - 1 dependent repositories - 27 downloads last month - 2 stars on GitHub - 1 maintainer
scrapy-algolia-exporter 0.0.2
Scrapy item exporter for the Algolia API
3 versions - Latest release: over 6 years ago - 1 dependent repositories - 38 downloads last month - 1 stars on GitHub - 1 maintainer
scrapy-rss 0.3.1
RSS Tools for Scrapy Framework
15 versions - Latest release: about 2 years ago - 1 dependent repositories - 450 downloads last month - 29 stars on GitHub - 1 maintainer
scrapy-save-statistics 0.2
Scrapy Save Statistics: Save statistics extension for Scrapy
2 versions - Latest release: about 7 years ago - 1 dependent repositories - 14 downloads last month - 3 stars on GitHub - 1 maintainer
Top 9.8% on pypi.org
scrapple 0.3.0
A framework for creating web content extractors
10 versions - Latest release: over 7 years ago - 2 dependent repositories - 246 downloads last month - 494 stars on GitHub - 1 maintainer
gerapy-team 0.1.3
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, D...
1 version - Latest release: almost 2 years ago - 49 downloads last month - 3,206 stars on GitHub - 1 maintainer
Top 4.8% on pypi.org
gerapy 0.9.13
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, D...
47 versions - Latest release: 10 months ago - 34 dependent repositories - 1.01 thousand downloads last month - 3,206 stars on GitHub - 1 maintainer
zcbot-scrapy-redis 0.7.3.2110.2
Redis-based components for Scrapy 2.11.0+.
3 versions - Latest release: 6 months ago - 24 downloads last month - 5,448 stars on GitHub - 1 maintainer
Top 2.7% on pypi.org
icrawler 0.6.8
A multi-thread crawler framework with many builtin image crawlers provided.
43 versions - Latest release: 4 days ago - 2 dependent packages - 86 dependent repositories - 99.9 thousand downloads last month - 813 stars on GitHub - 2 maintainers
fara_principals 0.0.7
A web scraper designed to collect Foreign Principal information from fara.gov
3 versions - Latest release: about 7 years ago - 25 downloads last month - 0 stars on GitHub - 1 maintainer
frontoxy 1.0.3
Distributed URLs frontier for Scrapy with RabbitMQ
4 versions - Latest release: over 7 years ago - 1 dependent repositories - 23 downloads last month - 7 stars on GitHub - 1 maintainer
Top 6.1% on pypi.org
scrapy-user-agents 0.1.1
Automatically pick an User-Agent for every request
2 versions - Latest release: over 5 years ago - 2 dependent packages - 110 dependent repositories - 17.7 thousand downloads last month - 23 stars on GitHub - 1 maintainer
Top 2.8% on pypi.org
scrapy-playwright 0.0.34
Playwright integration for Scrapy
34 versions - Latest release: 5 months ago - 4 dependent packages - 22 dependent repositories - 35.5 thousand downloads last month - 839 stars on GitHub - 1 maintainer
Top 3.8% on pypi.org
random-user-agent 1.0.1
A package to get random user agents based filters provided by user
5 versions - Latest release: over 5 years ago - 24 dependent packages - 94 dependent repositories - 161 thousand downloads last month - 92 stars on GitHub - 1 maintainer
Top 9.3% on pypi.org
page_finder 0.1.9
Find which links on a web page are pagination links
10 versions - Latest release: over 7 years ago - 14 dependent repositories - 490 downloads last month - 30 stars on GitHub - 2 maintainers
page_clustering 0.0.1
Online k-means clustering of web pages
1 version - Latest release: almost 8 years ago - 13 dependent repositories - 239 downloads last month - 35 stars on GitHub - 2 maintainers
elasticstats-scrapy 0.1.5
A scrapy extension to send crawl stats to elasticsearch index.
5 versions - Latest release: about 7 years ago - 1 dependent repositories - 29 downloads last month - 0 stars on GitHub - 1 maintainer
m3u8-to-mp4 0.1.11
Python downloader for saving m3u8 video to local MP4 file.
12 versions - Latest release: almost 2 years ago - 1 dependent package - 1 dependent repositories - 2.58 thousand downloads last month - 60 stars on GitHub - 1 maintainer
scrapy-scraperapi-middleware 1.0
Middleware to easily implement ScraperAPI in Scrapy projects
3 versions - Latest release: over 2 years ago - 1 dependent repositories - 155 downloads last month - 2 stars on GitHub - 1 maintainer
Top 6.0% on pypi.org
anti-useragent 1.0.10
fake pc or app browser useragent, anti useragent, and other awesome tools
10 versions - Latest release: over 1 year ago - 4 dependent packages - 2 dependent repositories - 2.45 thousand downloads last month - 230 stars on GitHub - 1 maintainer
django-scratchy 0.4.0
Manage and run Scrapy spiders in Django
8 versions - Latest release: almost 4 years ago - 1 dependent repositories - 49 downloads last month - 2 stars on GitHub - 1 maintainer
scrapy-selenium2 0.0.8
Scrapy with selenium
1 version - Latest release: 2 months ago - 39 downloads last month - 887 stars on GitHub - 1 maintainer
scrapy-ai 0.0.1
AI-powered scrapy plugin
1 version - Latest release: 3 months ago - 28 downloads last month - 1 stars on GitHub - 1 maintainer
new-frontera 0.9.0 💰
A scalable frontier for web crawlers
2 versions - Latest release: 4 months ago - 9 downloads last month - 0 stars on GitHub - 1 maintainer
cli-yams 1.1
Yet Another Media Scraper
2 versions - Latest release: 4 months ago - 9 downloads last month - 3 stars on GitHub - 1 maintainer
scrapy-qos 0.0.2
implement QOS(TokenBucket) in scrapy download middleware
2 versions - Latest release: 4 months ago - 17 downloads last month - 1 maintainer
scrapy-playwright-full 0.0.3404
Playwright integration for Scrapy
6 versions - Latest release: 11 days ago - 67 downloads last month - 839 stars on GitHub - 1 maintainer
scrapy-scraper 1.7 💰
Web crawler and scraper based on Scrapy and Playwright's headless browser.
8 versions - Latest release: 8 days ago - 39 downloads last month - 4 stars on GitHub - 1 maintainer
scrapy-aiohttp 0.1.2
Scrapy middleware for sending requests with aiohttp.
3 versions - Latest release: 6 months ago - 9 downloads last month - 1 stars on GitHub - 1 maintainer
volleystats 0.8.1
Command-line tool to scrape volleyball statistics from Data Project Web Competition websites
8 versions - Latest release: 4 months ago - 72 downloads last month - 6 stars on GitHub - 1 maintainer
boost-siper 0.9 removed
横冲直闯无回调写法的高速爬虫框架
8 versions - Latest release: 7 months ago - 480 downloads last month - 4 stars on GitHub - 1 maintainer
scrapy-kit 0.1.12
A library for scrapy tools, including but not limited to the usual pipelines, middlewares, etc.
9 versions - Latest release: 7 months ago - 5 downloads last month - 0 stars on GitHub - 1 maintainer
scrapy-crawlbase-middleware 1.0.0
Scrapy Crawlbase Proxy Middleware: Crawlbase interfacing middleware for Scrapy
1 version - Latest release: 11 months ago - 12 downloads last month - 6 stars on GitHub - 1 maintainer
scrapy-manipulate-request 0.0.2
An async scrapy request downloader middleware, support random request and response manipulation.
2 versions - Latest release: 11 months ago - 67 downloads last month - 10 stars on GitHub - 1 maintainer
scrapy-processors 2.0.5
Provides processors for the itemloaders package, commonly used with scrapy.
9 versions - Latest release: 11 days ago - 49 downloads last month - 0 stars on GitHub - 1 maintainer
scrapy-mattermostbot 1.0.0
A Scrapy extension for sending notification to Mattermost channels
1 version - Latest release: about 1 year ago - 15 downloads last month - 1 maintainer
proxyport2 1.1.0
Proxy Port SDK
3 versions - Latest release: about 1 year ago - 2 dependent packages - 45 downloads last month - 2 stars on GitHub - 1 maintainer
yugioh-scraper 0.2.0
Yu-Gi-Oh! Scraper is a project that crawls websites and APIs and extracts Yu-Gi-Oh! related data ...
5 versions - Latest release: over 1 year ago - 46 downloads last month - 1 stars on GitHub - 1 maintainer
tkit-scrapy-mongo 0.0.0.116654862
Terry toolkit sdk for tkit_scrapy_mongo ,
1 version - Latest release: over 1 year ago - 8 downloads last month - 0 stars on GitHub - 1 maintainer
spdclient 0.0.1
Python Wrapper for Scrapyd WebService
1 version - Latest release: over 1 year ago - 11 downloads last month - 0 stars on GitHub - 1 maintainer
scrapy-athlinks 0.0.1
Web scraper for race results hosted on Athlinks.
1 version - Latest release: over 1 year ago - 14 downloads last month - 0 stars on GitHub - 1 maintainer
scrapy-helper 1.5.3
scrapy helper
2 versions - Latest release: 10 months ago - 1 dependent repositories - 14 downloads last month - 1 maintainer
herodotus 0.1.0
Package for fast integration between SQLAlchemy models and Scrapy spiders
1 version - Latest release: over 1 year ago - 643 downloads last month - 0 stars on GitHub - 1 maintainer
ax-spider 0.1.4
A simple Python crawler framework
11 versions - Latest release: about 1 year ago - 25 downloads last month - 2 stars on GitHub - 1 maintainer
zhihu-crawler 1.0.3
知乎关键词搜索、热榜、用户信息、回答、专栏文章、评论等信息的抓取程序
6 versions - Latest release: about 2 years ago - 1 dependent repositories - 24 downloads last month - 26 stars on GitHub - 1 maintainer
ze 0.0.17.dev1
Scaper to lager portal of news in Brazil.
2 versions - Latest release: about 7 years ago - 1 dependent repositories - 19 downloads last month - 5 stars on GitHub - 1 maintainer
Top 5.3% on pypi.org
wechatsogou 4.5.4 💰
Api for wechat mp with sogou
25 versions - Latest release: about 5 years ago - 17 dependent repositories - 595 downloads last month - 5,783 stars on GitHub - 1 maintainer
web-walker 3.1.5
your can crawl web pages with litte settings. based on scrapy.
43 versions - Latest release: almost 7 years ago - 1 dependent repositories - 145 downloads last month - 53 stars on GitHub - 1 maintainer
weblocust 1.0.3
A more Powerful Spider System in Python based on pyspider
4 versions - Latest release: over 7 years ago - 1 dependent repositories - 25 downloads last month - 6 stars on GitHub - 1 maintainer
web-crawler-plus 0.9.14
A micro-framework to crawl the web pages with crawlers configs. It can use MongoDB, Elasticsearch...
17 versions - Latest release: about 6 years ago - 1 dependent repositories - 61 downloads last month - 31 stars on GitHub - 1 maintainer
Top 7.7% on pypi.org
wayback-machine-scraper 1.0.8
A command-line utility for scraping Wayback Machine snapshots from archive.org.
6 versions - Latest release: over 3 years ago - 4 dependent repositories - 232 downloads last month - 400 stars on GitHub - 2 maintainers
take 0.2.0
A DSL for extracting data from a web page.
9 versions - Latest release: about 9 years ago - 10 dependent repositories - 57 downloads last month - 8 stars on GitHub - 1 maintainer
structure-spider 1.3.5
multi requests to combine a structure item.
49 versions - Latest release: over 4 years ago - 1 dependent repositories - 14 downloads last month - 29 stars on GitHub - 1 maintainer
stickymeta 0.0.5
Handy tools to maintain persistent meta values between requests in Scrapy spiders
3 versions - Latest release: over 7 years ago - 2 dependent repositories - 23 downloads last month - 1 stars on GitHub - 1 maintainer
stand 0.1.11
IP代理池
1 version - Latest release: over 4 years ago - 1 dependent repositories - 21 downloads last month - 22 stars on GitHub - 1 maintainer
spymanga 0.1
A lib to download manga chapters
1 version - Latest release: about 6 years ago - 1 dependent repositories - 8 downloads last month - 2 stars on GitHub - 1 maintainer
spydy 0.1.25
light-weight high-level web-crawling framework
25 versions - Latest release: about 3 years ago - 1 dependent repositories - 52 downloads last month - 2 stars on GitHub - 1 maintainer
spider-renderer 0.2.3
Building a modular crawler template system based on Jinja2.
12 versions - Latest release: almost 4 years ago - 1 dependent repositories - 92 downloads last month - 0 stars on GitHub - 1 maintainer
spidermanager 1.3.1
Admin ui for spider service
2 versions - Latest release: almost 3 years ago - 1 dependent repositories - 28 downloads last month - 2 stars on GitHub - 1 maintainer
spiderkeeper-deploy 0.1.3
Deploy to SpiderKeeper
2 versions - Latest release: over 5 years ago - 1 dependent repositories - 18 downloads last month - 2 stars on GitHub - 1 maintainer
spider-feeder 0.3.0
spider-feeder is a library to help loading inputs to scrapy spiders.
3 versions - Latest release: over 4 years ago - 1 dependent repositories - 31 downloads last month - 14 stars on GitHub - 1 maintainer
sodo 1.0.1
Redis-based scheduler and Message queue Spider for Scrapy, Provide more flexible and practical ...
2 versions - Latest release: about 5 years ago - 1 dependent repositories - 15 downloads last month - 4 stars on GitHub - 1 maintainer
sitesearcher 0.1a2
A command line tool that creates fulltext search indexes of your favourite websites on your machi...
2 versions - Latest release: over 7 years ago - 1 dependent repositories - 8 downloads last month - 1 stars on GitHub - 1 maintainer
shub-cli 2.0.1
A CLI at your hands to deal with the features of ScrapingHub.
5 versions - Latest release: over 7 years ago - 1 dependent repositories - 28 downloads last month - 16 stars on GitHub - 1 maintainer
sentry-scrapy 0.2
Scrapy integration with Sentry SDK (unofficial)
2 versions - Latest release: over 5 years ago - 1 dependent repositories - 38 downloads last month - 7 stars on GitHub - 1 maintainer