map, reduce, filterといった関数と同様の処理をマルチスレッド下で行い、特にI/Oがボトルネックとなるような処理の高速化を行う関数群、tmap, treduce, tfilterを提供します．また，関数レベルの簡易な並列処理を可能にするparallelという関数も提供します．
使用例
def test_tmap(numThreads):
def double(x):
time.sleep(random.random()) #something time-consuming procedure
return x*2
return tmap(double,range(1,32),numThreads)
def test_treduce(numThreads):
def add(x,y):
time.sleep(random.random()) #something time-consuming procedure
return x+y
return treduce(add,range(1,32),numThreads)
def test_tfilter(numThreads):
def prime(n):
time.sleep(random.random()) #something time-consuming procedure
return n % 2 == 0
return tfilter(prime, range(1,32),numThreads)
def test_parallel():
def hoge(a): #something time-consuming procedure #1
time.sleep(random.random())
return a**2
def fuga(a): #something time-consuming procedure #2
time.sleep(random.random())
return a**5
return parallel((hoge,[5],{}),(fuga,[2],{}))
def mapreducetest():
import google
def search(args):
return google.doGoogleSearch(*args).results
def aggregate(a,b):
a.extend(b)
return a
return treduce(aggregate, tmap(search, (('hoge',0),('hoge',10))))
実行結果
### testing tmap() ### result: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 1 thread(s), 13.9450819492 sec result: 2 4 6 8 12 10 14 16 18 20 24 22 26 28 30 34 36 32 38 42 40 44 46 50 52 48 54 56 58 62 60 2 thread(s), 8.19523096085 sec result: 2 8 10 4 6 12 16 14 18 24 22 20 26 32 28 38 30 36 40 42 34 48 44 46 56 50 52 62 54 60 58 4 thread(s), 4.12061500549 sec result: 8 12 32 18 4 10 30 26 22 6 36 24 34 48 40 58 2 20 28 14 54 16 46 52 44 38 42 56 60 50 62 16 thread(s), 1.59210181236 sec result: 16 12 54 2 4 28 60 22 24 36 38 50 48 34 8 14 20 52 32 58 10 40 6 44 62 56 26 18 30 46 42 32 thread(s), 0.983188152313 sec result: 20 56 6 46 32 60 4 42 62 38 28 26 30 36 8 12 40 2 10 34 22 58 52 18 48 44 54 16 50 24 14 64 thread(s), 0.977565050125 sec result: 34 28 38 54 36 24 40 30 14 56 48 4 58 12 2 52 16 10 22 26 6 18 44 46 50 42 60 8 20 62 32 128 thread(s), 1.06019997597 sec ### testing treduce() ### result: 496 1 thread(s), 14.7298090458 sec result: 496 2 thread(s), 8.68240904808 sec result: 496 4 thread(s), 3.75789403915 sec result: 496 16 thread(s), 3.38547897339 sec result: 496 32 thread(s), 3.19397521019 sec result: 496 64 thread(s), 3.45076584816 sec result: 496 128 thread(s), 3.19514322281 sec ### testing tfilter() ### result: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 1 thread(s), 16.4703760147 sec result: 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 2 thread(s), 7.04814600945 sec result: 4 2 6 8 10 12 14 18 16 22 24 20 28 30 26 4 thread(s), 3.61463689804 sec result: 4 8 20 22 14 2 18 6 26 12 16 10 30 24 28 16 thread(s), 1.45710206032 sec result: 28 14 16 22 10 12 18 2 26 20 24 8 4 6 30 32 thread(s), 0.976119041443 sec result: 28 4 2 6 16 22 18 30 12 20 10 26 14 24 8 64 thread(s), 1.01319289207 sec result: 4 14 12 2 26 30 20 8 16 10 22 6 24 28 18 128 thread(s), 1.06458902359 sec ### testing parallel() ### [25, 32]
制限など
- tmap, treduce, tfilterは、処理順序及び返り値の順序が保存されません。なので、そうなっても問題ない処理に利用してください
- もちろん関数parallelにおいては，返り値の順序は保存されます