pl.sync.to_iterable

Creates an iterable from a stage. Use this function to when you want to have more control over how the output stage is consumed, especifically, setting the maxsize argument can help you avoid OOM error if the consumer is slow.

Parameters:

Name Type Description Default
stage Union[pypeln.sync.stage.Stage[~A], Iterable[~A], pypeln.utils.Undefined]

A Stage or Iterable.

<pypeln.utils.Undefined object at 0x7f54c588db20>
maxsize int

This parameter is not used and only kept for API compatibility with the other modules.

0
return_index bool

When set to True the resulting iterable will yield the Elemen(index: Tuple[int, ...], value: Any) which contains both the resulting value and the index parameter which holds information about the order of creation of the elements at the source.

False

Returns:

Type Description
Union[Iterable[~A], pypeln.utils.Partial[Iterable[~A]]]

If the stage parameters is given then this function returns an iterable, else it returns a Partial.

Source code in pypeln/sync/api/to_iterable.py
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def to_iterable(
    stage: tp.Union[
        Stage[A], tp.Iterable[A], pypeln_utils.Undefined
    ] = pypeln_utils.UNDEFINED,
    maxsize: int = 0,
    return_index: bool = False,
) -> tp.Union[tp.Iterable[A], pypeln_utils.Partial[tp.Iterable[A]]]:
    """
    Creates an iterable from a stage. Use this function to when you want to have more control over how the output stage is consumed, especifically, setting the `maxsize` argument can help you avoid OOM error if the consumer is slow.

    Arguments:
        stage: A Stage or Iterable.
        maxsize: This parameter is not used and only kept for API compatibility with the other modules.
        return_index: When set to `True` the resulting iterable will yield the `Elemen(index: Tuple[int, ...], value: Any)` which contains both the resulting value and the index parameter which holds information about the order of creation of the elements at the source.

    Returns:
        If the `stage` parameters is given then this function returns an iterable, else it returns a `Partial`.
    """

    if isinstance(stage, pypeln_utils.Undefined):
        return pypeln_utils.Partial(
            lambda stage: to_iterable(stage, maxsize=maxsize, return_index=return_index)
        )

    if isinstance(stage, Stage):
        iterable = stage.to_iterable(maxsize=maxsize, return_index=return_index)
    else:
        iterable = stage

    return iterable