pl.sync.concat

Concatenates / merges many stages into a single one by appending elements from each stage in order, that is, it yields an element from the frist stage, then an element from the second stage and so on until it reaches the last stage and starts again. When a stage has no more elements its taken out of the process.

import pypeln as pl

stage_1 = [1, 2, 3]
stage_2 = [4, 5, 6, 7]

stage_3 = pl.sync.concat([stage_1, stage_2]) # [1, 4, 2, 5, 3, 6, 7]

Parameters:

Name Type Description Default
stages List[Union[pypeln.sync.stage.Stage[~A], Iterable[~A]]]

A list of Stage or Iterable.

required
maxsize int

This parameter is not used and only kept for API compatibility with the other modules.

0

Returns:

Type Description
Stage

A stage object.

Source code in pypeln/sync/api/concat.py
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
def concat(
    stages: tp.List[tp.Union[Stage[A], tp.Iterable[A]]], maxsize: int = 0
) -> Stage:
    """
    Concatenates / merges many stages into a single one by appending elements from each stage in order, that is, it yields an element from the frist stage, then an element from the second stage and so on until it reaches the last stage and starts again. When a stage has no more elements its taken out of the process.

    ```python
    import pypeln as pl

    stage_1 = [1, 2, 3]
    stage_2 = [4, 5, 6, 7]

    stage_3 = pl.sync.concat([stage_1, stage_2]) # [1, 4, 2, 5, 3, 6, 7]
    ```

    Arguments:
        stages: A list of Stage or Iterable.
        maxsize: This parameter is not used and only kept for API compatibility with the other modules.

    Returns:
        A stage object.
    """

    dependencies = [to_stage(stage) for stage in stages]

    return Stage(
        process_fn=Concat(),
        timeout=0,
        dependencies=dependencies,
        on_start=None,
        on_done=None,
        f_args=[],
    )