distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
Main metrics
Overview
| Name With Owner | webpy/webpy |
|---|---|
| Primary Language | Python |
| Program language | Python (Language Count: 2) |
| Platform | |
| License: | Other |
| Release Count | 18 |
| Last Release Name | webpy-0.70 (Posted on ) |
| First Release Name | webpy-0.31 (Posted on ) |
| Created At | 2008-09-29 03:42:56 |
| Pushed At | 2025-05-13 21:44:18 |
| Last Commit At | 2025-05-08 12:49:08 |
| Stargazers Count | 5914 |
| Watchers Count | 333 |
| Fork Count | 1307 |
| Commits Count | 1080 |
| Has Issues Enabled | |
| Issues Count | 452 |
| Issue Open Count | 37 |
| Pull Requests Count | 186 |
| Pull Requests Open Count | 14 |
| Pull Requests Close Count | 152 |
| Has Wiki Enabled | |
| Is Archived | |
| Is Fork | |
| Is Locked | |
| Is Mirror | |
| Is Private |
