hypermodel.kubeflow package

Submodules

hypermodel.kubeflow.deploy module

Helper function to deploy and run a pipeline to a production environment, deploying the pipeline as a part of the “Production” experiment.

hypermodel.kubeflow.deploy.deploy_pipeline(pipeline, environment: str = 'dev', host: Optional[str] = None, client_id: Optional[str] = None, namespace: Optional[str] = None)

Deploy the current pipeline Kubeflew in the provided namespace on the using the Kubeflow api found at host and authenticate using client_id.

Parameters:
  • environment (str) – The environment to create the pipelien in (e.g. “dev”, “prod”)
  • host (str) – The host we can find the Kubeflow API at (e.g. https://{APP_NAME}.endpoints.{PROJECT_ID}.cloud.goog/pipeline)
  • client_id (str) – The IAP client id we can use for authorisate (e.g. “XXXXXX-XXXXXXXXX.apps.googleusercontent.com”)
  • namespace (str) – The Kuberenetes / Kubeflow namespace to deploy to (e.g. kubeflow)

hypermodel.kubeflow.deploy_dev module

Helper function to deploy and run a pipeline to a development environment

hypermodel.kubeflow.deploy_dev.deploy_to_dev(pipeline)

Deploy the Kubeflow Pipelines Pipeline (e.g. a method decorated with @dsl.pipeline) to Kubeflow and execute it.

Parameters:pipeline (func) – The @dsl.pipeline method describing the pipeline
Returns:True if the deployment suceeds

hypermodel.kubeflow.kubeflow_client module

class hypermodel.kubeflow.kubeflow_client.KubeflowClient(host: Optional[str] = None, client_id: Optional[str] = None, namespace: Optional[str] = 'kubeflow')

Bases: object

A wrapper of the existing Kubeflow Pipelines Client which enriches it to be able to access more of the Kubeflow Pipelines API.

create_experiment(experiment_name)

Create a new Kubeflow Pipelines Experiment (grouping of pipeliens / runs)

Parameters:experiment_name (str) – The name of the experiment
Returns:The Kubeflow experiement object created
create_job(name: str, pipeline, experiment, description=None, enabled=True, max_concurrency=1, cron=None)

Create a new Kubeflow Pipelines Job

Parameters:
  • name (str) – The name of the Job
  • pipeline (Pipeline) – The Pipeline object to execute when the Job is called
  • experiment (Experiment) – The Experiment object to create the Job in.
  • description (str) – A description of what the Job is all about
  • enabled (bool) – Should be Job be enabled?
  • max_concurrency (int) – How many concurrent executions of the Job are allowed?
  • cron (str) – The CRON expression to use to execute the job periodicalls
Returns:

The Kubeflow API response object.
create_pipeline(pipeline_func, pipeline_name)

Create a new Kubeflow Pipeline using the provided pipeline function

Parameters:pipeline_func – The method decorated by @dsl.pipeline which defines the pipeline
Returns:The Kubeflow Pipeline object created
delete_job(job)

Delete a Job using its job.id

Parameters:job (KubeflowJob) – A Job object to delete
Returns:True if the Job was deleted succesfully
delete_pipeline(pipeline)

Delete the specified Pipeline

Parameters:pipeline – The pipeline object to delete
Returns:True if successfull
find_experiment(id=None, name=None)

Look up an Experiment by its name or id. Returns None if the Experiment cannot be found. Both id and name are optional, but atleast one must be provided. Where both a provided, the function will return with the first Experiment matching either id or name.

Parameters:
  • id (str) – The id of the Experiment to find
  • name (string) – The name of the Experiment to find
Returns:

A reference to the Experiment if found, and None if not.
find_job(job_name)

Look up a job by its name (in the current namespace). Returns None if the job cannot be found

Parameters:job_name (str) – The name of the job to find
Returns:A reference to the job if found, and None if not.
find_pipeline(name)

Look up a Pipeline by its name (in the current namespace). Returns None if the Pipeline cannot be found

Parameters:name (str) – The name of the Pipeline to find
Returns:A reference to the Pipeline if found, and None if not.
list_experiments()

List the Experiments in the current namespace

Returns:A list of all the Experiments
list_jobs()

List the Jobs in the current namespace

Returns:A list of all the Jobs
list_pipelines()

List the Pipelines in the current namespace

Returns:A list of all the Pipelines in the current Experiment
list_runs(experiment_name)

List the Runs in the specified Exper`iment

Parameters:experiment_name (str) – The name of the Experiment
Returns:A list of all the Runs in the current Experiment

Module contents