tbp.monty.frameworks.experiments

class MontyExperiment(config: DictConfig) → None

Bases: object

General Monty experiment class used to run sensorimotor experiments.

This class implements the framework for setting up an environment interface and Monty model, the outermost loops for training and evaluating (including run epoch and episode).

__init__(config: DictConfig) → None

Initialize the experiment based on the provided configuration.

Parameters:

config (DictConfig) – config specifying variables of the experiment.

close()

create_env_interface(env_interface_class, env_interface_args)

Environment interface used to collect data from environment observations.

Parameters:

• env_interface_class – The class of the environment interface.

• env_interface_args – The arguments for the environment interface.

Returns:

The instantiated environment interface.

Raises:

TypeError – If env_interface_class is not a subclass of EnvironmentInterface

evaluate()

Run n_eval_epochs.

get_epoch_state()

init_counters()

init_env(env_init_func, env_init_args)

init_loggers(logging_config: dict[str, Any]) → None

Initialize logger with specified log level.

Parameters:

logging_config – Logging configuration.

init_model(monty_config, model_path=None)

Initialize the Monty model.

Parameters:

• monty_config – configuration for the Monty class.

• model_path – Optional model checkpoint. Can be full file name or just the directory containing the “model.pt” file saved from a previous run.

Returns:

Monty class instance

Raises:

TypeError – If motor_system_class is not a subclass of MotorSystem or policy_class is not a subclass of MotorPolicy.

init_monty_data_loggers(logging_config: dict[str, Any]) → None

Initialize Monty data loggers.

Parameters:

logging_config – Logging configuration.

load_environment_interfaces(config)

load_state_dict(load_dir)

Load state_dict of previous experiment.

post_episode(steps)

Call post_episode on elements in experiment and increment counters.

General order of post episode should be:

logger_handler.post_episode model.post_episode increment counters env_interface.post_episode

If the logger_handler is called later it will not log the correct episode ID and target object. If model.post_episode is called before the logger we have already updated the target to graph mapping and will never get ‘confused’/’FP’.

post_epoch()

Call sub post_epoch functions and save state dict.

post_step(_step, _observation)

Hook for anything you want to do after a step.

pre_episode()

Call pre_episode on elements in experiment and set mode.

pre_epoch()

Set environment interface and call sub pre_epoch functions.

pre_step(_step, _observation)

Hook for anything you want to do before a step.

run_episode()

Run one episode until model.is_done.

run_epoch()

Run epoch -> Run one episode for each object.

save_state_dict(output_dir=None)

Save state_dict of experiment and model.

setup_experiment(config: dict[str, Any]) → None

Set up the basic elements of a Monty experiment and initialize counters.

Parameters:

config – config specifying variables of the experiment.

state_dict()

Return state_dict with total steps.

train()

Run n_train_epochs.

property logger_args

Get current status of counters for the logger.

Returns:

dict with current expirent state.

class MontyGeneralizationExperiment(config: DictConfig) → None

Bases: MontyObjectRecognitionExperiment

Remove the tested object model from memory to see what is recognized instead.

pre_episode()

Pre episode where we pass target object to the model for logging.

class MontyObjectRecognitionExperiment(config: DictConfig) → None

Bases: MontyExperiment

Experiment customized for object-pose recognition with a single object.

Adds additional logging of the target object and pose for each episode and specific terminal states for object recognition. It also adds code for handling a matching and an exploration phase during each episode when training.

Note that this experiment assumes a particular model configuration, in order for the show_observations method to work: a zoomed out “view_finder” rgba sensor and an up-close “patch” depth sensor

pre_episode()

Pre-episode hook.

Pre episode where we pass the primary target object, as well as the mapping between semantic ID to labels, both for logging/evaluation purposes.

run_episode()

Episode that checks the terminal states of an object recognition episode.

run_episode_steps()

Runs one episode of the experiment.

At each step, observations are collected from the env_interface and either passed to the model or sent directly to the motor system. We also check if a terminal condition was reached at each step and increment step counters.

Returns:

The number of total steps taken in the episode.

class MontySupervisedObjectPretrainingExperiment(config: DictConfig)

Bases: MontyExperiment

Just run exploratory steps and tell the model the object and pose.

This is used to pretrain models of objects such that we don’t always have to start learning from scratch. Since we provide the model with the object label and pose we do not have to perform matching steps. We just run exploratory steps and use the collected observations to update the models in memory.

NOTE: This is not really an experiment, it is more a pretraining step to generate models that can then be loaded at the beginning of an experiment.

__init__(config: DictConfig)

Initialize the experiment based on the provided configuration.

Parameters:

config (DictConfig) – config specifying variables of the experiment.

evaluate()

Use experiment just for supervised pretraining -> no eval.

post_epoch()

Post epoch without saving state_dict.

pre_episode()

Pre episode where we pass target object to the model for logging.

run_episode()

Run a supervised episode on one object in one pose.

In a supervised episode we only make exploratory steps (no object recognition is attempted) since the target label is provided. The target label and pose is then used to update the object model in memory. This can for instance be used to warm-up the training by starting with some models in memory instead of completely from scatch. It also makes testing easier as long as we don’t have a good solution to dealing with incomplete objects.

setup_experiment(config)

Set up the basic elements of a Monty experiment and initialize counters.

Parameters:

config – config specifying variables of the experiment.

train()

Save state_dict at the end of training.

class ProfileExperimentMixin

Bases: object

Save cProfile traces for each episode.

Example

class ProfiledExp(ProfileExperimentMixin, SomeExp):

pass

my_config[“experiment_class”] = ProfiledExp

NOTE: make sure this class is leftmost in mixin order.

close()

evaluate()

make_profile_dir()

run_episode()

setup_experiment(config)

train()

tbp.monty.frameworks.experiments.data_collection_experiments

class DataCollectionExperiment(config: DictConfig) → None

Bases: MontyObjectRecognitionExperiment

Collect data in environment without performing inference.

Stripped down experiment, to explore points on the object and save JUST the resulting observations as a .pt file. This was used to collect data that can then be used offline to quickly test other, non-Monty methods (like ICP). Mostly useful for methods that require batches of observations and do not work with inference through movement over the object. Otherwise would recommend to implement approaches directly in the Monty framework instead of using offline data.

pass_features_to_motor_system(observation, step)

post_episode()

Call post_episode on elements in experiment and increment counters.

General order of post episode should be:

logger_handler.post_episode model.post_episode increment counters env_interface.post_episode

If the logger_handler is called later it will not log the correct episode ID and target object. If model.post_episode is called before the logger we have already updated the target to graph mapping and will never get ‘confused’/’FP’.

post_epoch()

Call sub post_epoch functions and save state dict.

pre_episode()

Pre episode where we pass target object to the model for logging.

run_episode()

Episode that checks the terminal states of an object recognition episode.

tbp.monty.frameworks.experiments.monty_experiment

class MontyExperiment(config: DictConfig) → None

Bases: object

General Monty experiment class used to run sensorimotor experiments.

This class implements the framework for setting up an environment interface and Monty model, the outermost loops for training and evaluating (including run epoch and episode).

__init__(config: DictConfig) → None

Initialize the experiment based on the provided configuration.

Parameters:

config (DictConfig) – config specifying variables of the experiment.

close()

create_env_interface(env_interface_class, env_interface_args)

Environment interface used to collect data from environment observations.

Parameters:

• env_interface_class – The class of the environment interface.

• env_interface_args – The arguments for the environment interface.

Returns:

The instantiated environment interface.

Raises:

TypeError – If env_interface_class is not a subclass of EnvironmentInterface

evaluate()

Run n_eval_epochs.

get_epoch_state()

init_counters()

init_env(env_init_func, env_init_args)

init_loggers(logging_config)

Initialize logger with specified log level.

Parameters:

logging_config – Logging configuration.

init_model(monty_config, model_path=None)

Initialize the Monty model.

Parameters:

• monty_config – configuration for the Monty class.

• model_path – Optional model checkpoint. Can be full file name or just the directory containing the “model.pt” file saved from a previous run.

Returns:

Monty class instance

Raises:

TypeError – If motor_system_class is not a subclass of MotorSystem or policy_class is not a subclass of MotorPolicy.

init_monty_data_loggers(logging_config)

Initialize Monty data loggers.

Parameters:

logging_config – Logging configuration.

load_environment_interfaces(config)

load_state_dict(load_dir)

Load state_dict of previous experiment.

post_episode(steps)

Call post_episode on elements in experiment and increment counters.

General order of post episode should be:

logger_handler.post_episode model.post_episode increment counters env_interface.post_episode

If the logger_handler is called later it will not log the correct episode ID and target object. If model.post_episode is called before the logger we have already updated the target to graph mapping and will never get ‘confused’/’FP’.

post_epoch()

Call sub post_epoch functions and save state dict.

post_step(_step, _observation)

Hook for anything you want to do after a step.

pre_episode()

Call pre_episode on elements in experiment and set mode.

pre_epoch()

Set environment interface and call sub pre_epoch functions.

pre_step(_step, _observation)

Hook for anything you want to do before a step.

run_episode()

Run one episode until model.is_done.

run_epoch()

Run epoch -> Run one episode for each object.

save_state_dict(output_dir=None)

Save state_dict of experiment and model.

setup_experiment(config)

Set up the basic elements of a Monty experiment and initialize counters.

Parameters:

config – config specifying variables of the experiment.

state_dict()

Return state_dict with total steps.

train()

Run n_train_epochs.

property logger_args

Get current status of counters for the logger.

Returns:

dict with current expirent state.

tbp.monty.frameworks.experiments.object_recognition_experiments

class MontyGeneralizationExperiment(config: DictConfig) → None

Bases: MontyObjectRecognitionExperiment

Remove the tested object model from memory to see what is recognized instead.

pre_episode()

Pre episode where we pass target object to the model for logging.

class MontyObjectRecognitionExperiment(config: DictConfig) → None

Bases: MontyExperiment

Experiment customized for object-pose recognition with a single object.

Adds additional logging of the target object and pose for each episode and specific terminal states for object recognition. It also adds code for handling a matching and an exploration phase during each episode when training.

Note that this experiment assumes a particular model configuration, in order for the show_observations method to work: a zoomed out “view_finder” rgba sensor and an up-close “patch” depth sensor

pre_episode()

Pre-episode hook.

Pre episode where we pass the primary target object, as well as the mapping between semantic ID to labels, both for logging/evaluation purposes.

run_episode()

Episode that checks the terminal states of an object recognition episode.

run_episode_steps()

Runs one episode of the experiment.

At each step, observations are collected from the env_interface and either passed to the model or sent directly to the motor system. We also check if a terminal condition was reached at each step and increment step counters.

Returns:

The number of total steps taken in the episode.

tbp.monty.frameworks.experiments.pretraining_experiments

class MontySupervisedObjectPretrainingExperiment(config: DictConfig)

Bases: MontyExperiment

Just run exploratory steps and tell the model the object and pose.

This is used to pretrain models of objects such that we don’t always have to start learning from scratch. Since we provide the model with the object label and pose we do not have to perform matching steps. We just run exploratory steps and use the collected observations to update the models in memory.

NOTE: This is not really an experiment, it is more a pretraining step to generate models that can then be loaded at the beginning of an experiment.

__init__(config: DictConfig)

Initialize the experiment based on the provided configuration.

Parameters:

config (DictConfig) – config specifying variables of the experiment.

evaluate()

Use experiment just for supervised pretraining -> no eval.

post_epoch()

Post epoch without saving state_dict.

pre_episode()

Pre episode where we pass target object to the model for logging.

run_episode()

Run a supervised episode on one object in one pose.

In a supervised episode we only make exploratory steps (no object recognition is attempted) since the target label is provided. The target label and pose is then used to update the object model in memory. This can for instance be used to warm-up the training by starting with some models in memory instead of completely from scatch. It also makes testing easier as long as we don’t have a good solution to dealing with incomplete objects.

setup_experiment(config)

Set up the basic elements of a Monty experiment and initialize counters.

Parameters:

config – config specifying variables of the experiment.

train()

Save state_dict at the end of training.

tbp.monty.frameworks.experiments.profile

class ProfileExperimentMixin

Bases: object

Save cProfile traces for each episode.

Example

class ProfiledExp(ProfileExperimentMixin, SomeExp):

pass

my_config[“experiment_class”] = ProfiledExp

NOTE: make sure this class is leftmost in mixin order.

close()

evaluate()

make_profile_dir()

run_episode()

setup_experiment(config)

train()

make_stats_df(stats)

Convert cProfile.Profile() stats to dataframe.

Take a cProfile.Profile() object, gather stats, put in dataframe, sort by cumtime.

Returns:

The dataframe with the stats.