thread module lets you create pipelines using objects from python's threading module according to Pypeln's general architecture. Use this module when you are in need to perform some synchronous IO operations and DONT need to perform heavy CPU operations.
Most functions in this module return a
pl.thread.Stage object which implement the
Iterable interface which enables you to combine it seamlessly with regular Python code.
You can iterate over any
p.thread.Stage to get back the results of its computation:
import pypeln as pl import time from random import random def slow_add1(x): time.sleep(random()) # <= some slow computation return x + 1 def slow_gt3(x): time.sleep(random()) # <= some slow computation return x > 3 data = range(10) # [0, 1, 2, ..., 9] stage = pl.thread.map(slow_add1, data, workers=3, maxsize=4) stage = pl.thread.filter(slow_gt3, stage, workers=2) for x in stage: print(x) # e.g. 5, 6, 9, 4, 8, 10, 7
At each stage the you can specify the numbers of
maxsize parameter limits the maximum amount of elements that the stage can hold simultaneously.