distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Main metrics
Overview
| Name With Owner | webpy/webpy |
|---|---|
| Primary Language | Python |
| Program language | Python (Language Count: 2) |
| Platform | |
| License: | Other |
| Release Count | 18 |
| Last Release Name | webpy-0.70 (Posted on ) |
| First Release Name | webpy-0.31 (Posted on ) |
| Created At | 2008-09-29 11:42:56 |
| Pushed At | 2025-09-18 21:57:43 |
| Last Commit At | 2025-09-18 21:57:43 |
| Stargazers Count | 5925 |
| Watchers Count | 327 |
| Fork Count | 1303 |
| Commits Count | 1084 |
| Has Issues Enabled | |
| Issues Count | 455 |
| Issue Open Count | 40 |
| Pull Requests Count | 190 |
| Pull Requests Open Count | 12 |
| Pull Requests Close Count | 153 |
| Has Wiki Enabled | |
| Is Archived | |
| Is Fork | |
| Is Locked | |
| Is Mirror | |
| Is Private |
