curia.infra.databricks.task module

class curia.infra.databricks.task.DatabricksTaskDefinition(name: str, task_slug: str, inputs: ~typing.List[~curia.infra.task.TaskInputDefinition], outputs: ~typing.List[~curia.infra.task.TaskOutputDefinition], description: str | None = None, _function: ~typing.Callable | None = None, _module: str | None = None, code_src_type: ~curia.infra.types.DatabricksTaskCodeSourceType = DatabricksTaskCodeSourceType.S3WHEEL, code_src_cfg: dict = <factory>, min_workers: int = 2, max_workers: int = 4)

Bases: TaskDefinition

Task definition for a task executed through a databricks job

code_src_type: DatabricksTaskCodeSourceType = 'S3WHEEL'

code_src_cfg: dict

min_workers: int = 2

max_workers: int = 4

task_type: ClassVar[TaskType] = 'DatabricksJob'

run(task_execution_id: str, api_token: str, api_endpoint: str, spark: SparkSession, dbutils: DBUtils) → None

Run the analytics task flow definition :param task_execution_id: The task execution ID :param api_token: The API key to use to retrieve the task inputs from the Curia API :param api_endpoint: The API endpoint to use to retrieve the task inputs from the Curia API :return: The result

build_task_type_specific_input_arguments(context: dict)