ScrapeGraphAI

基于人工智能的 Python 网络爬虫。『Python scraper based on AI』

🕷️ ScrapeGraphAI: You Only Scrape Once

English | 中文

Downloads
linting: pylint
Pylint
CodeQL
License: MIT

ScrapeGraphAI is a web scraping python library that uses LLM and direct graph logic to create scraping pipelines for websites and local documents (XML, HTML, JSON, etc.).

Just say which information you want to extract and the library will do it for you!

🚀 Quick install

The reference page for Scrapegraph-ai is available on the official page of PyPI: pypi.

pip install scrapegraphai

Note: it is recommended to install the library in a virtual environment to avoid conflicts with other libraries 🐱

🔍 Demo

Official streamlit demo:

My Skills

Try it directly on the web using Google Colab:

Open In Colab

📖 Documentation

The documentation for ScrapeGraphAI can be found here.

Check out also the Docusaurus here.

💻 Usage

There are three main scraping pipelines that can be used to extract information from a website (or local file):

  • SmartScraperGraph: single-page scraper that only needs a user prompt and an input source;
  • SearchGraph: multi-page scraper that extracts information from the top n search results of a search engine;
  • SpeechGraph: single-page scraper that extracts information from a website and generates an audio file.
  • SmartScraperMultiGraph: multiple page scraper given a single prompt

It is possible to use different LLM through APIs, such as OpenAI, Groq, Azure and Gemini, or local models using Ollama.

Case 1: SmartScraper using Local Models

Remember to have Ollama installed and download the models using the ollama pull command.

from scrapegraphai.graphs import SmartScraperGraph

graph_config = {
    "llm": {
        "model": "ollama/mistral",
        "temperature": 0,
        "format": "json",  # Ollama needs the format to be specified explicitly
        "base_url": "http://localhost:11434",  # set Ollama URL
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",  # set Ollama URL
    },
    "verbose": True,
}

smart_scraper_graph = SmartScraperGraph(
    prompt="List me all the projects with their descriptions",
    # also accepts a string with the already downloaded HTML code
    source="https://perinim.github.io/projects",
    config=graph_config
)

result = smart_scraper_graph.run()
print(result)

The output will be a list of projects with their descriptions like the following:

{'projects': [{'title': 'Rotary Pendulum RL', 'description': 'Open Source project aimed at controlling a real life rotary pendulum using RL algorithms'}, {'title': 'DQN Implementation from scratch', 'description': 'Developed a Deep Q-Network algorithm to train a simple and double pendulum'}, ...]}

Case 2: SearchGraph using Mixed Models

We use Groq for the LLM and Ollama for the embeddings.

from scrapegraphai.graphs import SearchGraph

# Define the configuration for the graph
graph_config = {
    "llm": {
        "model": "groq/gemma-7b-it",
        "api_key": "GROQ_API_KEY",
        "temperature": 0
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": "http://localhost:11434",  # set ollama URL arbitrarily
    },
    "max_results": 5,
}

# Create the SearchGraph instance
search_graph = SearchGraph(
    prompt="List me all the traditional recipes from Chioggia",
    config=graph_config
)

# Run the graph
result = search_graph.run()
print(result)

The output will be a list of recipes like the following:

{'recipes': [{'name': 'Sarde in Saòre'}, {'name': 'Bigoli in salsa'}, {'name': 'Seppie in umido'}, {'name': 'Moleche frite'}, {'name': 'Risotto alla pescatora'}, {'name': 'Broeto'}, {'name': 'Bibarasse in Cassopipa'}, {'name': 'Risi e bisi'}, {'name': 'Smegiassa Ciosota'}]}

Case 3: SpeechGraph using OpenAI

You just need to pass the OpenAI API key and the model name.

from scrapegraphai.graphs import SpeechGraph

graph_config = {
    "llm": {
        "api_key": "OPENAI_API_KEY",
        "model": "gpt-3.5-turbo",
    },
    "tts_model": {
        "api_key": "OPENAI_API_KEY",
        "model": "tts-1",
        "voice": "alloy"
    },
    "output_path": "audio_summary.mp3",
}

# ************************************************
# Create the SpeechGraph instance and run it
# ************************************************

speech_graph = SpeechGraph(
    prompt="Make a detailed audio summary of the projects.",
    source="https://perinim.github.io/projects/",
    config=graph_config,
)

result = speech_graph.run()
print(result)

The output will be an audio file with the summary of the projects on the page.

Sponsors

🤝 Contributing

Feel free to contribute and join our Discord server to discuss with us improvements and give us suggestions!

Please see the contributing guidelines.

My Skills
My Skills
My Skills

📈 Roadmap

Check out the project roadmap here! 🚀

Wanna visualize the roadmap in a more interactive way? Check out the markmap visualization by copy pasting the markdown content in the editor!

❤️ Contributors

Contributors

🎓 Citations

If you have used our library for research purposes please quote us with the following reference:

  @misc{scrapegraph-ai,
    author = {Marco Perini, Lorenzo Padoan, Marco Vinciguerra},
    title = {Scrapegraph-ai},
    year = {2024},
    url = {https://github.com/VinciGit00/Scrapegraph-ai},
    note = {A Python library for scraping leveraging large language models}
  }

Authors

Contact Info
Marco Vinciguerra Linkedin Badge
Marco Perini Linkedin Badge
Lorenzo Padoan Linkedin Badge

📜 License

ScrapeGraphAI is licensed under the MIT License. See the LICENSE file for more information.

Acknowledgements

  • We would like to thank all the contributors to the project and the open-source community for their support.
  • ScrapeGraphAI is meant to be used for data exploration and research purposes only. We are not responsible for any misuse of the library.

主要指標

概覽
名稱與所有者ScrapeGraphAI/Scrapegraph-ai
主編程語言Python
編程語言 (語言數: 3)
平台
許可證MIT License
所有者活动
創建於2024-01-27 16:54:38
推送於2025-07-03 09:51:50
最后一次提交
發布數429
最新版本名稱v1.61.0 (發布於 )
第一版名稱release (發布於 )
用户参与
星數20.3k
關注者數130
派生數1.7k
提交數2.8k
已啟用問題?
問題數401
打開的問題數14
拉請求數467
打開的拉請求數0
關閉的拉請求數62
项目设置
已啟用Wiki?
已存檔?
是復刻?
已鎖定?
是鏡像?
是私有?