pixaris.experiment_handlers package

Submodules

pixaris.experiment_handlers.base module

class pixaris.experiment_handlers.base.ExperimentHandler[source]

Bases: object

When implementing a new Experiment Handler, inherit from this one and implement all the abstract methods.

abstractmethod load_experiment_results_for_dataset(project: str, dataset: str)[source]
abstractmethod load_images_for_experiment(project: str, dataset: str, experiment_run_name: str, local_results_directory: str)[source]
abstractmethod load_projects_and_datasets()[source]
abstractmethod store_results(project: str, dataset: str, experiment_run_name: str, image_name_pairs: Iterable[tuple[Image, str]], metric_values: dict[str, float], args: dict[str, any]) → None[source]

pixaris.experiment_handlers.gcp module

class pixaris.experiment_handlers.gcp.GCPExperimentHandler(gcp_project_id: str, gcp_bq_experiment_dataset: str, gcp_pixaris_bucket_name: str)[source]

Bases: ExperimentHandler

GCPExperimentHandler is a class for storing experiment results in Google Cloud Storage and BigQuery. It is also used to retrieve the images for display.

Parameters:

  • gcp_project_id (str) – The Google Cloud Platform project ID.

  • gcp_bq_experiment_dataset (str) – The BigQuery dataset for storing experiment results.

  • gcp_pixaris_bucket_name (str) – The name of the Google Cloud Storage bucket for storing images.

load_experiment_results_for_dataset(project: str, dataset: str) → DataFrame[source]

Loads the results of an experiment from a BigQuery dataset.

Parameters:

  • project (str) – The name of the project.

  • dataset (str) – The name of the dataset.

Returns:

The results of the experiment as a pandas DataFrame.

Return type:

pd.DataFrame

load_images_for_experiment(project: str, dataset: str, experiment_run_name: str, local_results_directory: str) → list[str][source]

Downloads images for a experiment_run_name from GCP bucket to local directory. Returns list of local image paths that belong to the experiment_run_name.

Parameters:

  • project (str) – The name of the project.

  • dataset (str) – The name of the dataset.

  • experiment_run_name (str) – Name of the experiment run.

  • local_results_directory (str) – The local directory to store the downloaded images.

Returns:

List of local image paths.

Return type:

list[str]

load_projects_and_datasets() → dict[source]

Loads the projects and datasets available in the Google Cloud Storage bucket.

Returns:

A dictionary mapping project names to lists of dataset names.

Return type:

dict

store_results(project: str, dataset: str, experiment_run_name: str, image_name_pairs: Iterable[tuple[Image, str]], metric_values: dict[str, float], args: dict[str, any] = {})[source]

Stores the results of an experiment, including images, metrics, and parameters.

Parameters:

  • project (str) – The name of the project.

  • dataset (str) – The name of the dataset.

  • experiment_run_name (str) – The name of the experiment run.

  • image_name_pairs (Iterable[tuple[Image.Image, str]]) – An iterable of tuples containing images and their names.

  • metric_values (dict[str, float]) – A dictionary of metric names and their values.

  • args (dict[str, any]) – Additional arguments, such as images or JSON data.

pixaris.experiment_handlers.local module

class pixaris.experiment_handlers.local.LocalExperimentHandler(local_results_folder: str = 'local_results')[source]

Bases: ExperimentHandler

LocalExperimentHandler is a class that handles the storage and retrieval of experiment results locally.

Parameters:

local_results_folder (str, optional) – The root folder where the experiment subfolder is located. Defaults to ‘local_results’.

load_experiment_results_for_dataset(project: str, dataset: str) → DataFrame[source]

Load the results of an experiment.

Parameters:

  • project (str) – The name of the project.

  • dataset (str) – The name of the evaluation set.

Returns:

The results of the experiment as a DataFrame.

Return type:

pd.DataFrame

load_images_for_experiment(project: str, dataset: str, experiment_run_name: str, local_results_directory: str)[source]

Returns list of local image paths that belong to the experiment_run_name.

Parameters:

experiment_run_name (str) – Name of the experiment run.

Returns:

List of local image paths.

Return type:

list[str]

load_projects_and_datasets()[source]

Load the projects and datasets from the local results folder.

Returns:

A dictionary containing the projects and datasets. Example:

{"project": ["dataset1", "dataset2"]}

Return type:

dict[str, list[str]]

store_results(project: str, dataset: str, experiment_run_name: str, image_name_pairs: Iterable[tuple[Image, str]], metric_values: dict[str, float], args: dict[str, any] = {}, dataset_tracking_file_name: str = 'experiment_tracking.jsonl')[source]

Save a collection of images locally under a specified experiment name.

Parameters:

  • project (str) – The name of the project. This will be used to create a subfolder where images will be saved.

  • dataset (str) – The name of the evaluation set.

  • experiment_run_name (str) – The name of the experiment. This will be used to create a subfolder where images will be saved.

  • image_name_pairs (Iterable[tuple[Image.Image, str]]) – An iterable collection of tuples, where each tuple contains a PIL Image object and its corresponding name.

  • metric_values (dict[str, float]) – The metrics of the experiment to be saved as a JSON file.

  • args (dict[str, any]) – The arguments of the experiment to be saved as a JSON file. If any argument is a PIL Image, it will be saved as an image file.

  • dataset_tracking_file_name (str) – The name of the tracking file. Defaults to ‘experiment_tracking.jsonl’.

Module contents