

pl.process.from_iterable

Creates a stage from an iterable.

Parameters:

Name	Type	Description	Default
iterable	Union[Iterable[~T], pypeln.utils.Undefined]	A source Iterable.	<pypeln.utils.Undefined object at 0x7f27e00aaaf0>
use_thread	bool	If set to True (default) it will use a thread instead of a process to consume the iterable. Threads start faster and use thread memory to the iterable is not serialized, however, if the iterable is going to perform slow computations it better to use a process.	True

Returns:

Type	Description
Union[pypeln.process.stage.Stage[~T], pypeln.utils.Partial[pypeln.process.stage.Stage[~T]]]	Returns a Stage if the iterable parameters is given, else it returns a Partial.

Source code in pypeln/process/api/from_iterable.py

def from_iterable(
    iterable: tp.Union[tp.Iterable[T], pypeln_utils.Undefined] = pypeln_utils.UNDEFINED,
    use_thread: bool = True,
    maxsize: int = 0,
) -> tp.Union[Stage[T], pypeln_utils.Partial[Stage[T]]]:
    """
    Creates a stage from an iterable.

    Arguments:
        iterable: A source Iterable.
        use_thread: If set to `True` (default) it will use a thread instead of a process to consume the iterable. Threads start faster and use thread memory to the iterable is not serialized, however, if the iterable is going to perform slow computations it better to use a process.

    Returns:
        Returns a `Stage` if the `iterable` parameters is given, else it returns a `Partial`.
    """

    if isinstance(iterable, pypeln_utils.Undefined):
        return pypeln_utils.Partial(
            lambda iterable: from_iterable(iterable, use_thread=use_thread)
        )

    return Stage(
        process_fn=FromIterable(iterable, maxsize=maxsize),
        workers=1,
        maxsize=maxsize,
        timeout=0,
        total_sources=1,
        dependencies=[],
        on_start=None,
        on_done=None,
        use_threads=use_thread,
        f_args=[],
    )