disco

a Map/Reduce framework for distributed computing

  • Owner: discoproject/disco
  • Platform:
  • License:: BSD 3-Clause "New" or "Revised" License
  • Category::
  • Topic:
  • Like:
    0
      Compare:

Github stars Tracking Chart

Disco - Massive data, Minimal code

Disco Logo

Disco is a distributed map-reduce and big-data framework. Like
the original framework, which was publicized by Google, Disco supports
parallel computations over large data sets on an unreliable cluster of
computers. This makes it a perfect tool for analyzing and processing large
datasets without having to bother about difficult technical questions
related to distributed computing, such as communication protocols, load
balancing, locking, job scheduling or fault tolerance, all of which are taken
care by Disco.

Writing a Disco job is very simple. For example, the following job counts the number of words in a document:

from disco.core import Job, result_iterator

def map(line, params):
    for word in line.split():
        yield word, 1

def reduce(iter, params):
    from disco.util import kvgroup
    for word, counts in kvgroup(sorted(iter)):
        yield word, sum(counts)

if __name__ == '__main__':
    input = ["http://discoproject.org/media/text/chekhov.txt"]
    job = Job().run(input=input, map=map, reduce=reduce)
    for word, count in result_iterator(job.wait()):
        print word, count

Note: For installing Disco, you cannot use the zip or tar.gz packages generated by github, instead you should clone this repository.

The develop branch contains the newest features and is not recommended for use
in production. The master branch is the latest stable release and is tested in
production. Important bug fixes will be first merged into the develop branch
and then backported into the master branch.

Disco integrates with a lot of different tools. The following screenshot,
for example, shows using ipython notebook to write a Disco job and using
matplotlib to plot the results:
ipython example

To learn more about the Disco Ecosystem see Disco Integrations. For some other resources, check out the Talks on Disco. Visit [discoproject.org] (http://discoproject.org) for more information.

Build Status: Travis-CI :: Travis-CI

Overview

Name With Ownerdiscoproject/disco
Primary LanguageErlang
Program languageMakefile (Language Count: 8)
Platform
License:BSD 3-Clause "New" or "Revised" License
Release Count24
Last Release Name0.5.4 (Posted on )
First Release Name0.1 (Posted on 2008-09-02 01:48:47)
Created At2008-07-24 08:46:39
Pushed At2018-01-30 20:55:22
Last Commit At2014-10-27 22:01:49
Stargazers Count1.6k
Watchers Count85
Fork Count244
Commits Count3.7k
Has Issues Enabled
Issues Count418
Issue Open Count129
Pull Requests Count187
Pull Requests Open Count11
Pull Requests Close Count42
Has Wiki Enabled
Is Archived
Is Fork
Is Locked
Is Mirror
Is Private
To the top