spawn.schedulers

Schedulers for spawn

A scheduler understands how to turn a spec into a list of jobs and related dependncies to run

class spawn.schedulers.LuigiScheduler(config)[source]

Scheduler implementation for Luigi

Because this is currently the only scheduler implementation it’s probable that the interface will evolve in time.

add_worker()[source]

Add a worker

run(spawner, spec)[source]

Run the spec by generating tasks using the spawner

Parameters

  • spawner (TaskSpawner) – The task spawner

  • spec (SpecificationModel) – The specification