lenskit.parallel.ray#

Support for parallelism with Ray.

Attributes#

BATCH_SIZE

Classes#

TaskLimiter

Limit task concurrency using ray.wait().

Functions#

ensure_cluster()

init_cluster(*[, num_cpus, resources, global_logging])

Initialize or connect to a Ray cluster, with the LensKit options.

init_serializers()

ray_available()

Check if Ray is available.

ray_active()

Query whether Ray is active.

is_ray_worker()

Determine whether the current process is running on a Ray worker.

init_worker(*[, autostart])

Initialize a Ray worker process. Sets up logging, and returns the context.

Module Contents#

lenskit.parallel.ray.BATCH_SIZE = 200#
lenskit.parallel.ray.ensure_cluster()#
lenskit.parallel.ray.init_cluster(*, num_cpus=None, resources=None, global_logging=False, **kwargs)#

Initialize or connect to a Ray cluster, with the LensKit options.

The resulting cluster can be used by an invoker, or it can be used directly. The Ray invoker uses batching, though, so it only works well with many small tasks.

Parameters:

  • num_cpus (int | None) – The total number of CPUs to allow. Defaults to :fun:`effective_cpu_count`.

  • proc_slots – The number of “process slots” for LensKit parallel operations. Defaults to the LensKit process count. These slots are recorded as the lk_process resource on the Ray cluster.

  • resources (dict[str, float] | None) – Additional custom resources to register in the Ray cluster.

  • global_logging (bool) – True to wire up logging in the workers at startup, instead of only connecting logs when a task is run.

  • kwargs – Other options to pass to ray.init().

Stability:

Experimental

lenskit.parallel.ray.init_serializers()#
lenskit.parallel.ray.ray_available()#

Check if Ray is available.

Return type:

bool

lenskit.parallel.ray.ray_active()#

Query whether Ray is active.

Return type:

bool

lenskit.parallel.ray.is_ray_worker()#

Determine whether the current process is running on a Ray worker.

Return type:

bool

lenskit.parallel.ray.init_worker(*, autostart=True)#

Initialize a Ray worker process. Sets up logging, and returns the context.

Parameters:

autostart (bool) – Set to False to disable calling WorkerContext.start(), for when the caller will start and stop the context if it is new.

Return type:

lenskit.logging.multiprocess.WorkerContext

Exported Aliases#

lenskit.parallel.ray.get_logger()#

Re-exported alias for lenskit.logging.get_logger().

class lenskit.parallel.ray.WorkerContext#

Re-exported alias for lenskit.logging.multiprocess.WorkerContext.

class lenskit.parallel.ray.WorkerLogConfig#

Re-exported alias for lenskit.logging.multiprocess.WorkerLogConfig.

lenskit.parallel.ray.effective_cpu_count()#

Re-exported alias for lenskit.parallel.config.effective_cpu_count().

lenskit.parallel.ray.ensure_parallel_init()#

Re-exported alias for lenskit.parallel.config.ensure_parallel_init().

lenskit.parallel.ray.get_parallel_config()#

Re-exported alias for lenskit.parallel.config.get_parallel_config().