Skip to content
This repository has been archived by the owner on May 8, 2024. It is now read-only.
/ dpark Public archive

Python clone of Spark, a MapReduce alike framework in Python

License

Notifications You must be signed in to change notification settings

douban/dpark

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Mar 9, 2017
6aa27bb · Mar 9, 2017
Aug 14, 2014
May 25, 2016
Mar 9, 2017
Jan 18, 2016
Mar 9, 2017
Mar 9, 2017
Dec 5, 2016
Dec 6, 2016
Oct 23, 2011
Dec 24, 2015
Oct 28, 2011
Dec 5, 2016
Feb 24, 2016
Oct 27, 2016
Mar 9, 2017
Dec 24, 2015
Mar 9, 2017

Repository files navigation

DPark

pypi status travis-ci status

DPark is a Python clone of Spark, MapReduce(R) alike computing framework supporting iterative computation.

Example for word counting (wc.py):

import dpark
file = dpark.textFile("/tmp/words.txt")
words = file.flatMap(lambda x:x.split()).map(lambda x:(x,1))
wc = words.reduceByKey(lambda x,y:x+y).collectAsMap()
print wc

This script can run locally or on a Mesos cluster without any modification, just using different command-line arguments:

$ python wc.py
$ python wc.py -m process
$ python wc.py -m host[:port]

See examples/ for more use cases.

Some more docs (in Chinese): https://github.com/jackfengji/test_pro/wiki

DPark can run with Mesos 0.9 or higher.

If a $MESOS_MASTER environment variable is set, you can use a shortcut and run DPark with Mesos just by typing

$ python wc.py -m mesos

$MESOS_MASTER can be any scheme of Mesos master, such as

$ export MESOS_MASTER=zk://zk1:2181,zk2:2181,zk3:2181/mesos_master

In order to speed up shuffling, you should deploy Nginx at port 5055 for accessing data in DPARK_WORK_DIR (default is /tmp/dpark), such as:

server {
        listen 5055;
        server_name localhost;
        root /tmp/dpark/;
}

Mailing list: [email protected] (http://groups.google.com/group/dpark-users)