libextract

Extract data from websites using basic statistical magic

Github 星跟踪图

Libextract: extract data from websites

.. image:: https://travis-ci.org/datalib/libextract.svg?branch=master

:target: https://travis-ci.org/datalib/libextract

::

    ___ __              __                  __
   / (_) /_  ___  _  __/ /__________ ______/ /_
  / / / __ \/ _ \, /_/ __/ ___/ __ `/ ___/ __/
 / / / /_/ /  __/>  </ /_/ /  / /_/ / /__/ /_
/_/_/_.___/\___/_/, _, \__/_/   \__,_/\___/\__/

Libextract is a statistics-enabled <https://github.com/datalib/StatsCounter> data extraction library that works on HTML and XML documents and written in Python. Originating from eatiht <http://rodricios.github.io/eatiht/>, the extraction algorithm works by making one simple assumption: data appear as collections of repetitive elements. You can read about the reasoning here <http://rodricios.github.io/posts/solving_the_data_extraction_problem.html>_.

Overview

libextract.api.extract(document, encoding='utf-8', count=5)

Given an html *document*, and optionally the *encoding*, return
a list of nodes likely containing data (5 by default).

Installation

::

pip install libextract

Usage

Due to our simple definition of "data", we open up a single interfaceable method. Post-processing is up to you.

.. code-block:: python

from requests import get
from libextract.api import extract

r = get('http://en.wikipedia.org/wiki/Information_extraction')
textnodes = list(extract(r.content))

Using lxml's built-in methods for post-processing:

.. code-block:: python

>> print(textnodes[0].text_content())
Information extraction (IE) is the task of automatically extracting structured information...

The extraction algo is agnostic to article text as it is with tabular data:

.. code-block:: python

height_data = get("http://en.wikipedia.org/wiki/Human_height")
tabs = list(extract(height_data.content))

.. code-block:: python

>> [elem.text_content() for elem in tabs[0].iter('th')]
['Country/Region',
 'Average male height',
 'Average female height',
 ...]

Dependencies


::

    lxml
    statscounter

Disclaimer
~~~~~~~~~~

This project is still in its infancy; and advice and suggestions as
to what this library could and should be would be greatly appreciated

:)

项目概况

主要编程语言Python
编程语言Python
许可证MIT License
最后一次提交2015-05-23 01:52:06
创建于2015-04-06T08:27:32
推送于2020-10-02T11:31:24
提交数317
关注者数24
名称与所有者datalib/libextract
派生数39
星数496
问题数21
打开的问题数2
语言数2
拉请求数17
关闭的拉请求数2
已启用问题?
已启用Wiki?
已存档?
是分叉?
已锁定?
是镜像?
是私有?
To the top