tbp.monty.frameworks.models

tbp.monty.frameworks.models.abstract_monty_classes

class GoalStateGenerator

Bases: object

Generate goal-states that other learning modules and motor-systems will attempt.

Generate goal-states potentially (in the case of LMs) by outputing their own sub-goal-states. Provides a mechanism for implementing hierarchical action policies that are informed by world models/hypotheses.

abstract get_output_goal_state()

Return current output goal-state.

abstract set_driving_goal_state()

Set the driving goal state.

e.g., from a human operator or a high-level LM.

abstract step_gsg()

Called on each step of the LM to which the GSG belongs.

class LMMemory

Bases: object

Like a long-term memory storing all the knowledge an LM has.

abstract load_state_dict()

Take a state dict as an argument and set state for the memory.

abstract memory_consolidation()

Consolidate/clean up models stored in memory.

abstract state_dict()

Return a serializable dict with everything needed to save/load the memory.

abstract update_memory(observations)

Update models stored in memory given new observation & classification.

class LearningModule

Bases: object

abstract exploratory_step()

Model building step called inside of monty._step_learning_modules.

abstract get_output()

Return learning module output (same format as input).

abstract load_state_dict(state_dict)

Take a state dict as an argument and set state for this LM.

abstract matching_step()

Matching / inference step called inside of monty._step_learning_modules.

abstract post_episode()

Do things like update object models with stored data after an episode.

abstract pre_episode()

Do things like reset buffers or possible_matches before training.

abstract propose_goal_state()

Return the goal-state proposed by this LM’s GSG.

abstract receive_votes(votes)

Process voting data sent out from other learning modules.

abstract reset()

Do things like reset buffers or possible_matches before training.

abstract send_out_vote()

This method defines what data are sent to other learning modules.

abstract set_experiment_mode(mode)

Set the experiment mode.

Update state variables based on which method (train or evaluate) is being called at the experiment level.

abstract state_dict()

Return a serializable dict with everything needed to save/load this LM.

class Monty

Bases: object

abstract aggregate_sensory_inputs(observation)

Receive data from dataloader/env, organize on a per sensor module basis.

abstract is_done()

Return bool to tell the experiment if we are done with this episode.

abstract load_state_dict(state_dict)

Take a state dict as an argument and set state for monty and children.

abstract post_episode()

Recursively call post_episode on child classes.

abstract pre_episode()

Recursively call pre_episode on child classes.

abstract set_experiment_mode(mode)

Set the experiment mode.

Update state variables based on which method (train or evaluate) is being called at the experiment level.

abstract state_dict()

Return a serializable dict with everything needed to save/load monty.

abstract step(observation)

Take a matching, exploratory, or custom user-defined step.

Step taken depends on the value of self.step_type.

class ObjectModel

Bases: object

Model of an object. Is stored in Memory and used by LM.

abstract build_model(observations)

Build a new model.

abstract update_model(obersevations)

Update an existing model with new observations.

class SensorModule(*args, **kwds)

Bases: ABC, Callable

abstract post_episode()

This method is called after each episode.

abstract pre_episode()

This method is called before each episode.

abstract state_dict()

Return a serializable dict with this sensor module’s state.

Includes everything needed to save/load this sensor module.

abstract step(data)

Called on each step.

Parameters:

data – Sensor observations

abstract update_state(state)

tbp.monty.frameworks.models.buffer

class BaseBuffer

Bases: object

abstract classmethod append()

abstract classmethod reset()

class BufferEncoder(**kwargs)

Bases: JSONEncoder

Encoder to turn the buffer into a JSON compliant format.

default(obj)

Turn non compliant types into right format.

Parameters:

obj – The object to turn into a JSON compliant format.

Returns:

The object in a JSON compliant format.

class FeatureAtLocationBuffer

Bases: BaseBuffer

Buffer which stores features at locations coming into one LM. Also stores stats.

Used for building graph models and logging detailed stats about an episode. The location buffer is also used to calculate displacements.

add_overall_stats(stats)

Add overall episode stats to self.stats.

append(list_of_data)

Add an observation to the buffer. Must be features at locations.

TODO S: Store state objects instead of list of data? A provisional version of this is implemented below, as the GSG uses State objects for computations.

append_input_states(input_state)

get_all_features_on_object()

Get all observed features that were on the object.

Like in get_all_locations_on_object the feature arrays should have the length of np.where(self.on_object) and contain all features that were observed when at least one of the input channels was on the object. However, we also check for each feature whether its input channel indicated on_object=True and if not set its value to nan.

Note

Since all inputs to an LM should have overlapping receptive fields, there should be no difference between the two at the moment (except due to noisy estimates of on_object).

Returns:

All observed features that were on the object.

get_all_input_states()

Get all the input states that the buffer’s parent LM has observed.

Returns:

All the input states that the buffer’s parent LM has observed.

get_all_locations_on_object(input_channel=None)

Get all observed locations that were on the object.

Returns:

All observed locations that were on the object.

get_buffer_len_by_channel(input_channel)

Return the number of observations stored for that input channel.

get_current_displacement(input_channel)

Get the current displacement.

Returns:

The current displacement.

get_current_features(keys)

Get the current value of a specific feature.

Returns:

The current features.

get_current_location(input_channel)

Get the current location.

Note

May have to add on_object check at some point.

Returns:

The current location.

get_current_pose(input_channel)

Get currently sensed location and orientation.

Returns:

The currently sensed location and orientation.

get_current_ppf(input_channel)

Get the current ppf.

Returns:

The current ppf.

get_currently_on_object()

Check whether last sensation was on object.

Returns:

Whether the last sensation was on object.

get_first_displacement_len(input_channel)

Get length of first observed displacement.

Use for scale in DisplacementLM.

Returns:

The length of the first observed displacement.

get_first_sensory_input_channel()

Get name of first sensory (coming from SM) input channel in buffer.

Returns:

The name of the first sensory (coming from SM) input channel in buffer.

Raises:

ValueError – If no sensor channels are found in the buffer

get_infos_for_graph_update()

Return all stored infos require to update a graph in memory.

get_last_obs_processed()

Check whether last sensation was processed by LM.

Returns:

Whether the last sensation was processed by the LM.

get_matching_step_when_output_goal_set()

Return matching step when last goal-state was generated.

Return the LM matching step associated with the last time a goal-state was generated.

get_nth_displacement(n, input_channel)

Get the nth displacement.

Returns:

The nth displacement.

get_num_goal_states_generated()

Return number of goal states generated by the LM’s GSG since episode start.

Note use of length not sum.

get_num_matching_steps()

Return number of matching steps performed by this LM since episode start.

Returns:

The number of matching steps performed by this LM since episode start.

get_num_observations_on_object()

Get the number of steps where at least 1 input was on the object.

Returns:

The number of steps where at least 1 input was on the object.

get_num_steps_post_output_goal_generated()

Return number of steps since last output goal-state.

Return the number of Monty-matching steps that have elapsed since the last time an output goal-state was generated.

get_previous_input_states()

Get previous State inputs received by the buffer’s parent LM.

i.e. in the last time step.

Returns:

The previous input states.

reset()

Reset the buffer.

set_individual_ts(object_id, pose)

Update self.stats with the individual LMs terminal state.

update_last_stats_entry(stats)

Use this to overwrite last entry (for example after voting).

update_stats(stats, update_time=True, append=True, init_list=True)

Update statistics for this step in the episode.

tbp.monty.frameworks.models.displacement_matching

class DisplacementGraphLM(k=None, match_attribute=None, tolerance=0.001, use_relative_len=False, graph_delta_thresholds=None)

Bases: GraphLM

Learning module that uses displacement stored in graphs to recognize objects.

get_object_rotation(sensed_displacements, model_displacements, get_reverse_r=False)

Calculate the rotation between two sets of displacement vectors.

Parameters:

• sensed_displacements – The displacements that were sensed.

• model_displacements – The displacements in the model that were matched to the sensed displacements.

• get_reverse_r – Whether to get the rotation that turns the model such that it would produce the sensed_displacements (False) or the rotation needed to turn the sensed_displacements into the model displacements.

Returns:

The rotation in Euler angles, as a matrix, and as a Rotation object.

get_object_scale(sensed_displacement, model_displacement)

Calculate the objects scale given sensed and model displacements.

Returns:

The scale of the object.

get_unique_pose_if_available(object_id)

Compute (location, rotation, scale) of object.

If we are sure about where on the object we are compare the sensed displacements to the observed displacements to calculate the pose, else return None.

Returns:

The pose and scale of the object.

reset()

Call this before each episode.

class DisplacementGraphMemory(match_attribute, *args, **kwargs)

Bases: GraphMemory

Graph memory that stores graphs with displacements as edges.

get_initial_hypotheses()

load_state_dict(state_dict)

Load graphs into memory from a state_dict and add point pair features.

tbp.monty.frameworks.models.evidence_matching

class EvidenceGraphLM(max_match_distance, tolerances, feature_weights, feature_evidence_increment=1, max_nneighbors=3, initial_possible_poses='informed', evidence_update_threshold='all', vote_evidence_threshold=0.8, past_weight=1, present_weight=1, vote_weight=1, object_evidence_threshold=1, x_percent_threshold=10, path_similarity_threshold=0.1, pose_similarity_threshold=0.35, required_symmetry_evidence=5, graph_delta_thresholds=None, max_graph_size=0.3, max_nodes_per_graph=2000, num_model_voxels_per_dim=50, use_multithreading=True, gsg_class=<class 'tbp.monty.frameworks.models.goal_state_generation.EvidenceGoalStateGenerator'>, gsg_args=None, *args, **kwargs)

Bases: GraphLM

Learning module that accumulates evidence for objects and poses.

Matching Attributes:

max_match_distance: Maximum distance of a tested and stored location to

be matched.

tolerances: How much can each observed feature deviate from the stored

features to still be considered a match.

feature_weights: How much should each feature be weighted when calculating

the evidence update for hypotheses. Weights are stored in a dictionary with keys corresponding to features (same as keys in tolerances)

feature_evidence_increment: Feature evidence (between 0 and 1) is multiplied

by this value before being added to the overall evidence of a hypothesis. This factor is only multiplied with the feature evidence (not the pose evidence as opposed to the present_weight).

max_nneighbors: Maximum number of nearest neighbors to consider in the

radius of a hypothesis for calculating the evidence.

initial_possible_poses: initial possible poses that should be tested for.

In [“uniform”, “informed”, list]. default = “informed”.

evidence_update_threshold: How to decide which hypotheses should be updated.

In [int, float, ‘mean’, ‘median’, ‘all’, ‘x_percent_threshold’].

vote_evidence_threshold: Only send votes that have a scaled evidence above

this threshold. Vote evidences are in the range of [-1, 1] so the threshold should not be outside this range.

past_weight: How much should the evidence accumulated so far be weighted

when combined with the evidence from the most recent observation.

present_weight: How much should the current evidence be weighted when added

to the previous evidence. If past_weight and present_weight add up to 1, the evidence is bounded and can’t grow infinitely. NOTE: right now this doesn’t give as good performance as with unbounded evidence since we don’t keep a full history of what we saw. With a more efficient policy and better parameters that may be possible to use though and could help when moving from one object to another and to generally make setting thresholds etc. more intuitive.

vote_weight: Vote evidence (between -1 and 1) in multiplied by this value

when being added to the overall evidence of a hypothesis. If past and current_weight add up to 1, it is use as weight in np.average to keep the evidence in a fixed range.

Terminal Condition Attributes:

object_evidence_threshold: Minimum required evidence for an object to be

recognized. We additionally check that the evidence for this object is significantly higher than for all other objects.

x_percent_threshold: Used in two places:

1. All objects whose highest evidence is greater than the most likely

   objects evidence - x_percent of the most like objects evidence are considered possible matches. That means to only have one possible match, no other object can have more evidence than the candidate match’s evidence - x percent of it.

2. Within one object, possible poses are considered possible if their

   evidence is larger than the most likely pose of this object - x percent of this poses evidence.

# TODO: should we use a separate theshold for within and between objects? If this value is larger, the model is usually more robust to noise and reaches a better performance but also requires a lot more steps to reach a terminal condition, especially if there are many similar object in the data set.

path_similarity_threshold: How similar do paths have to be to be

considered the same in the terminal condition check.

pose_similarity_threshold: difference between two poses to be considered

unique when checking for the terminal condition (in radians).

required_symmetry_evidence: number of steps with unchanged possible poses

to classify an object as symetric and go into terminal condition.

Model Attributes:

graph_delta_thresholds: Thresholds used to compare nodes in the graphs being

learned, and thereby whether to include a new point or not. By default, we only consider the distance between points, using a threshold of 0.001 (determined in remove_close_points). Can also specify thresholds based on e.g. point-normal angle difference, or principal curvature magnitude difference.

max_graph_size: Maximum size of a graph in meters. Any observations that fall

out of this range will be discarded/used for building a new model. This constraints the size of models that an LM can learn and enforces learning models of sub-components of objects.

max_nodes_per_graph: Maximum number of nodes in a graph. This will be k when

picking the k-winner voxels to add their content into the graph used for matching.

num_model_voxels_per_dim: Number of voxels per dimension in the model grid.

This constraints the spatial resolution that the model can represent. max_graph_size/num_model_voxels_per_dim = how much space is lumped into one voxel. All locations that fall into the same voxel will be averaged and represented as one value. num_model_voxels_per_dim should not be too large since the memory requirements grow cubically with this number.

gsg_class: The type of goal-state-generator to associate with the LM. gsg_args: Dictionary of configuration parameters for the GSG.

Debugging Attributes:

use_multithreading: Whether to calculate evidence updates for different

objects in parallel using multithreading. This can be done since the updates to different objects are completely independent of each other. In general it is recommended to use this but it can be usefull to turn it off for debugging purposes.

collect_stats_to_save()

Get all stats that this LM should store in the buffer for logging.

Returns:

The stats dictionary.

get_all_evidences()

Return evidence for each pose on each graph (pointer).

get_current_mlh()

Return the current most likely hypothesis of the learning module.

Returns:

graph_id, location, rotation, scale, evidence

Return type:

dict with keys

get_evidence_for_each_graph()

Return maximum evidence count for a pose on each graph.

get_mlh_for_object(object_id)

Get mlh for a specific object ID.

Note

When trying to retrieve the MLH for the current most likely object and not any other object, it is better to use self.current_mlh

Returns:

The most likely hypothesis for the object ID.

get_output()

Return the most likely hypothesis in same format as LM input.

The input to an LM at the moment is a dict of features at a location. The output therefor has the same format to keep the messaging protocol consistent and make it easy to stack multiple LMs on top of each other.

If the evidence for mlh is < object_evidence_threshold, interesting_features == False

get_possible_hypothesis_ids(object_id)

get_possible_matches()

Return graph ids with significantly higher evidence than median.

get_possible_poses(as_euler=True)

Return possible poses for each object (for logging).

Here this list doesn’t get narrowed down. This is not really used for evidence matching since we threshold in other places.

get_top_two_mlh_ids()

Retrieve the two most likely object IDs for this LM.

Returns:

The two most likely object IDs.

get_top_two_pose_hypotheses_for_graph_id(graph_id)

Return top two hypotheses for a given graph_id.

get_unique_pose_if_available(object_id)

Get the most likely pose of an object if narrowed down.

If there is not one unique possible pose or symmetry detected, return None

Returns:

The pose and scale if a unique pose is available, otherwise None.

receive_votes(vote_data)

Get evidence count votes and use to update own evidence counts.

Weighted by distance to votes and their evidence. TODO: also take into account rotation vote

vote_data contains:

pos_location_votes: shape=(N, 3) pos_rotation_votes: shape=(N, 3, 3) pose_evidences: shape=(N,)

reset()

Reset evidence count and other variables.

send_out_vote()

Send out hypotheses and the evidence for them.

Votes are a dict and contain the following:

pose_hypotheses: locations (V, 3) and rotations (V, 3, 3) pose_evidence: Evidence (V) for each location-rotation pair in the

pose hypotheses. Scaled into range [-1, 1] where 1 is the hypothesis with the largest evidence in this LM and -1 the one with the smallest evidence. When thresholded, pose_evidence will be in range [self.vote_evidence_threshold, 1]

sensed_pose_rel_body: sensed location and rotation of the input to this

LM. Rotation is represented by the pose vectors (point normal and curvature directions) for the SMs. For input from LMs it is also represented as 3 unit vectors, these are calculated from the estimated rotation of the most likely object. This pose is used to calculate the displacement between two voting LMs and to translate the votes between their reference frames. Shape=(4,3).

Where V is the number of votes (V=number of hypotheses if not thresholded) If none of the hypotheses of an object are > vote_evidence_threshold, this object will not send out a vote.

Returns:

possible_states: The possible states. sensed_pose_rel_body: The sensed pose relative to the body.

Return type:

None or dict

set_detected_object(terminal_state)

Set the current graph ID.

If we didn’t recognize the object this will be new_object{n} where n is len(graph_memory) + 1. Otherwise it is the id of the graph that we recognized. If we timed out it is None and we will not update the graph memory.

class EvidenceGraphMemory(max_nodes_per_graph, max_graph_size, num_model_voxels_per_dim, *args, **kwargs)

Bases: GraphMemory

Custom GraphMemory that stores GridObjectModel instead of GraphObjectModel.

get_initial_hypotheses()

get_rotation_features_at_all_nodes(graph_id, input_channel)

Get rotation features from all N nodes. shape=(N, 3, 3).

Returns:

The rotation features from all N nodes. shape=(N, 3, 3).

class MontyForEvidenceGraphMatching(*args, **kwargs)

Bases: MontyForGraphMatching

Monty model for evidence based graphs.

Customize voting and union of possible matches.

switch_to_exploratory_step()

Switch to exploratory step.

Also, set mlh evidence high enough to generate output during exploration.

tbp.monty.frameworks.models.evidence_sdr_matching

class EncoderSDR(sdr_length=2048, sdr_on_bits=41, lr=0.01, n_epochs=1000, stability=0.0, log_flag=False)

Bases: object

The SDR Encoder class.

This class keeps track of the dense representations, and trains them to output SDRs when binarized. This class also contains its own optimizer and function to add more objects/representations.

The representations are stored as dense vectors and binarized using top-k to convert them to SDRs. During training, the pairwise overlaps between the sdrs are compared to the target overlaps. This error signal trains the dense representations.

Refer to the self.train_sdrs function for more information on the training details

sdr_length

The size of the SDRs (total number of bits).

sdr_on_bits

The number of on bits in the SDRs. Controls sparsity.

lr

The learning rate of the encoding algorithm.

n_epochs

The number of training epochs per episode

stability

The stability parameter controls by how much old SDRs change relative to new SDRs. Value range is [0.0, 1.0], where 0.0 is no stability constraint applied and 1.0 is fixed SDRs. Values in between are for partial stability.

log_flag

Flag to activate the logger.

add_objects(n_objects)

Adds more objects to the available objects and re-initializes the optimizer.

We keep track of the stable representation ids (old objects) when adding new objects.

Parameters:

n_objects – Number of objects to add

binarize(emb)

Convert dense representations to SDRs (0s and 1s) using Top-k function.

Returns:

The SDRs.

get_sdr(index)

Return the SDR at a specific index.

This index refers to the object index in the SDRs dictionary (i.e., self.obj_sdrs)

optimize(overlap_error, mask)

Compute and apply local gradient descent.

Compute based on the overlap error and mask.

Note there is no use of the chain rule, i.e. each SDR is optimized based on the derivative of its clustering error with respect to its values, with no intermediary functions.

The overlap error helps correct the sign and also provides a magnitude for the representation updates.

Parameters:

• overlap_error – The difference between target and predicted overlaps.

• mask – Mask indicating valid entries in the overlap matrix.

Note

num_objects = self.n_objects

Note

A vectorized version of the algorithm is provided below, although it would need to be modified to avoid repeated creation of arrays in order to be more efficient. Leaving for now as this algorithm is not a bottle-neck (circa 10-20 seconds to learn 60 object SDRs).:

# Initialize gradients grad = np.zeros_like(self.obj_sdrs)

# Compute the pairwise differences between SDRs diff_matrix = (

self.obj_sdrs[:, np.newaxis, :] - self.obj_sdrs[np.newaxis, :, :]

)

# Compute the absolute differences for each pair abs_diff = np.sum(np.abs(diff_matrix), axis=2)

# Create a mask for non-zero differences non_zero_mask = abs_diff > 0

# Apply the mask to the original mask valid_mask = mask & non_zero_mask

# Calculate the summed distance and gradient contributions where the mask # is valid for logging summed_distance = np.sum(overlap_error * valid_mask * abs_diff)

# Calculate the gradients grad_contrib = overlap_error[:, :, np.newaxis] * 2 * diff_matrix grad += np.sum(grad_contrib * valid_mask[:, :, np.newaxis], axis=1) grad -= np.sum(grad_contrib * valid_mask[:, :, np.newaxis], axis=0)

# Update the SDRs using the gradient self.obj_sdrs -= self.lr * grad

Returns:

The summed distance for logging.

train_sdrs(target_overlaps, log_epoch_every=10)

Main SDR training function.

This function receives a copy of the average target overlap 2D tensor and trains the sdr representations for n_epochs to achieve these target overlap scores.

We use the overlap target as a learning signal to move the dense representations towards or away from each other. The magnitude of the overlap error controls the strength of moving dense representations. Also the sign of the overlap error controls whether the representations will be moving towards or away from each other.

We want to limit the amount by which trained representation change relative to untrained object representations such that higher-level LMs would not suffer from significant changes in lower-level representations that were used to build higher-level graphs.

When adding new representations, we keep track of the ids of the older representations (i.e., self.stable_ids). This allows us to control by how much the older representations move relative to the newer ones during training. This behavior is controlled by the stability value. During each training iteration, we update these older representations with an average of the optimizer output and the original representation (weighted by the stability value). Note that too much stability restricts the SDRs from adapting to desired changes in the target overlaps caused by normalization or distribution shift, affecting the overall encoding performance.

Consider two dense representations, A_dense and B_dense. We apply top-k operation on both to convert them to A_sdr and B_sdr, then calculate their overlaps. If the overlap is less than the target overlap, we move dense representations (A_dense and B_dense) closer to eachother with strength proportional to the error in overlaps. We move them apart if they have more overlap than the target.

Note

The distance_matrix variable is calculated using the cdist function and it denotes the pairwise euclidean distances between dense representations. The term “overlap” always refers to the overlap in bits between SDRs.

Note

The overlap_error is only used to weight the distance_matrix for each pair of objects, and gradients do not flow through the sparse overlap calculations.

Returns:

The stats dictionary for logging.

property n_objects

Return the available number of objects.

property sdrs

Return the available SDRs.

class EvidenceSDRGraphLM(*args, **kwargs)

Bases: EvidenceSDRLMMixin, EvidenceGraphLM

Class that incorporates the EvidenceSDR Mixin with the EvidenceGraphLM.

class EvidenceSDRLMMixin(*args, **kwargs)

Bases: object

This Mixin adds training of SDR representations to the EvidenceGraphLM.

It overrides the __init__ and post_episode functions of the LM

To use this Mixin, pass the EvidenceSDRGraphLM class as the learning_module_class in the learning_module_configs.

Additionally pass the sdr_args dictionary as an additional key in the learning_module_args.

The sdr_args dictionary should contain:

• log_path (string): A string that points to a temporary location for saving

  experiment logs. “None” means don’t save to file

• sdr_length (int): The size of the SDR to be used for encoding

• sdr_on_bits (int): The number of active bits to be used with these SDRs

• sdr_lr (float): The learning rate of the encoding algorithm

• n_sdr_epochs (int): The number of epochs to train the encoding algorithm

• stability (float): Stability of older object SDRs.

  Value range is [0.0, 1.0], where 0.0 is no stability applied and 1.0 is fixed SDRs.

• sdr_log_flag (bool): Flag indicating whether to log the results or not

See the monty_lab repo for reference. Specifically, experiments/configs/evidence_sdr_evaluation.py

collect_evidences()

Collect evidence scores from the Learning Module.

We do this in three steps:

• Step 1: We use the number of objects in the LM

  to update the sdr_encoder and id <-> obj tracking dictionaries, as well as the target overlap tensor

• Step 2: We collect evidences relative to the current

  most likely hypothesis (mlh). Evidences are stored in a 2d tensor.

• Step 3: We use the stored evidences to update the target overlap

  which stores the running average. Refer to EvidenceSDRTargetOverlaps for more details.

Note: We sort the ids in step 2 because the overlap values are suppossed to be symmetric (e.g., “2,5” = “5,2”). This way the target overlaps for the ids “x,y” and “y,x” will be averaged together in the EvidenceSDRTargetOverlaps class.

post_episode(*args, **kwargs)

Overrides the LM post_episode function.

This function collects evidences, trains SDRs and logs the output.

class EvidenceSDRTargetOverlaps

Bases: object

Keep track of the running average of target overlaps for each episode.

The target overlaps is implemented as a 2D tensor where the indices of the tensor represent the ids of the objects, and the values of the tensor represent a running average of the overlap target.

To achieve this, we implement functions for expanding the size of the overlap target tensor, linear mapping for normalization, and updating the overlap tensor (i.e., running average).

Note

We are averaging over multiple overlap targets. Multiple targets can happen for different reasons:

• Asymmetric evidences: the target overlap for object 1 w.r.t object 2

  ([2,1]) is averaged with object 2 w.r.t object 1 ([1,2]). This is possible because we sort the ids when we add the evidences to overlaps. Both evidences get added to the location [1,2].

• Additional episodes with similar MLO (most-likely object): More episodes

  can accumulate additional evidences on to the same key if the MLO is similar to previous MLO of another episode.

add_evidence(evidence, mapping_output_range)

Main function for updating the running average with evidence.

This function receives as input the relative evidence scores and maps them to overlaps in the mapping_output_range. The mapped overlaps are added to the running average in the function add_overlaps.

add_objects(new_size)

Expands the overlaps and the counts 2D tensors to accomodate new objects.

add_overlaps(mapped_overlaps)

Main function for updating the running average with overlaps.

The running average equation we use is: new_average = ((old_average * counts) + (new_val * 1))/ (counts + 1)

This calculates equally-weighted average, assuming that we keep track of the counts and increment them every time we add a new value to the average.

map_to_overlaps(evidence, output_range)

Linear mapping of values from input range to output range.

Only applies to real values (i.e., ignores nan values).

Returns:

?

property overlaps

Returns the target overlap values rounded to the nearest integer.

class LoggerSDR(path)

Bases: object

A simple logger that saves the data passed to it.

This logger maintains an episode counter and logs the data it receives under different files named by the episode counter.

See more information about what data is being logged under the `log_episode` function.

log_episode(data)

Receives data dictionary and saves it as a pth file.

This function will save all the data passed to it. Here is a breakdown of the data to be logged by this function.

The data dictionary contains these key-value pairs:

• mask: 2d tensor of the available overlap targets after this

  episode

• target_overlap: 2d tensor of the target overlap at the end of

  this episode

• training: Dictionary of training statistics for every epoch.

  Includes overlap_error, training_summed_distance, dense representations, and sdrs

• obj2id: Objects to ids dictionary mapping

• id2obj: Ids to objects dictionary mapping

tbp.monty.frameworks.models.feature_location_matching

class FeatureGraphLM(max_match_distance, tolerances, path_similarity_threshold=0.1, pose_similarity_threshold=0.35, required_symmetry_evidence=5, graph_delta_thresholds=None, initial_possible_poses='informed')

Bases: GraphLM

Learning module that uses features at locations to recognize objects.

get_object_rotation(graph_id, get_reverse_r=False)

Get the rotation of an object from the possible poses if resolved.

This first checks whether we have recognized a unique pose of the object or if a symmetry is detected. If one of the two is true it returns the unique rotation(s), otherwise returns None.

Parameters:

• graph_id – The object to check poses for.

• get_reverse_r – Whether to get the rotation that turns the model such that it would produce the sensed_displacements (False) or the rotation needed to turn the sensed_displacements into the model displacements.

Returns:

The rotation of the object if we know it.

get_unique_pose_if_available(object_id)

Get the pose of an object if we know it.

Scale is not implemented.

Returns:

The pose of the object if we know it.

receive_votes(vote_data)

Use votes to remove objects and poses from possible matches.

NOTE: Add object back into possible matches if majority of other modules

think it is correct? Could help with dealing with noise but may also prevent LMs from narrowing down quickly. Since we are not working with this LM anymore, we probably wont add that.

Parameters:

vote_data – positive and negative votes on object IDs + positive votes for locations and rotations on the object.

reset()

Call this before each episode.

send_out_vote()

Send out list of objects that are not possible matches.

By sending out the negative matches we avoid the problem that every LM needs to know about the same objects. We could think of this as more of an inhibitory signal (I know it can’t be this object so you all don’t need to check that anymore).

Returns:

List of objects that are not possible matches.

class FeatureGraphMemory(graph_delta_thresholds)

Bases: GraphMemory

Graph memory that matches objects by using features at locations.

get_initial_hypotheses()

get_nodes_with_matching_features(graph_id, features, list_of_lists=False)

Get only nodes with matching features.

Get a reduced list of nodes that includes only nodes with features that match the features dict passed here

Parameters:

• graph_id (str) – The graph descriptor e.g. ‘mug’

• features (dict) – The observed features to be matched

• list_of_lists (bool, optional) – should each location in the list be embedded

• list (in its own) –

• False. (Defaults to) –

Returns:

The reduced lists of ids / locs.

Return type:

tuple(list, list)

get_rotation_features_at_node(graph_id, node_id, channel)

Get the rotation features at a node in the graph.

Returns:

The rotation features at a node in the graph.

tbp.monty.frameworks.models.goal_state_generation

class EvidenceGoalStateGenerator(parent_lm, goal_tolerances=None, elapsed_steps_factor=10, min_post_goal_success_steps=inf, x_percent_scale_factor=0.75, desired_object_distance=0.03, **kwargs)

Bases: GraphGoalStateGenerator

Generator of goal states for an evidence-based graph LM.

GSG specifically setup for generating goal states for an evidence-based graph LM, which can therefore leverage the hypothesis-testing action policy. This policy uses hypotheses about the most likely objects, as well as knowledge of their structure from long-term memory, to propose test-points that should efficiently disambiguate the ID or pose of the object the agent is currently observing.

TODO M separate out the hypothesis-testing policy (which is one example of a model-based policy), from the GSG, which is the system that is capable of leveraging a variety of model-based policies.

reset()

Reset additional parameters specific to the Evidence GSG.

class GraphGoalStateGenerator(parent_lm, goal_tolerances=None, **kwargs)

Bases: GoalStateGenerator

Generate sub-goal states until the received goal state is achieved.

A component associated with each learning module that receives a high level goal state, and generates sub-goal states until the received goal state is achieved.

Generated goal-states are received by either:

i) other learning modules, which may model world-objects (e.g. a mug), or may model internal systems (e.g. the agent’s robotic limb) ii) motor actuators, in which case they represent simpler, primitive goal-states for the actuator to achieve (e.g. location and orientation of an actuator-sensor pair)

As well as the high-level, “driving” goal-state, generated goal-states can also be conditioned on other information, such as the LMs current most-likely hypothesis, and the structure of known object models (i.e. information local to the LM).

Note all goal-states conform to the State-class cortical messaging protocol (CMP).

get_output_goal_state()

Retrieve the output goal-state of the GSG.

This is the goal-state projected to other LM’s GSGs +/- motor-actuators.

Returns:

Output goal-state of the GSG.

reset()

Reset any stored attributes of the GSG.

set_driving_goal_state(received_goal_state)

Receive a new high-level goal to drive this goal-state-generator (GSG).

If none is provided, the goal-state generator should default to pursuing a goal-state of high confidence, with no other attributes of the state specified; in other words, it attempts to reduce uncertainty about the LM’s output (object ID and pose, whatever these may be).

TODO M: Currently GSGs always use the default, however future work will implement hierarchical action policies/GSGs, as well as the ability to specify a top goal-state by the experimenter.

TODO M : we currently just use “None” as a placehodler for the default goal-state > plan : set the default driving goal-state to a meaningful, non-None value that is compatible with the current method for checking convergence of an LM, such that achieving the driving goal-state can be used as a test for Monty convergence. This might be something like the below.

step_gsg(observations)

Step the GSG.

Check whether the GSG’s output and driving goal-states are achieved, and generate a new output goal-state if necessary.

tbp.monty.frameworks.models.graph_matching

class GraphLM(initialize_base_modules=True)

Bases: LearningModule

General Learning Module that contains a graph memory.

Subclasses are DisplacementGraphLM, FeatureGraphLM, and EvidenceGraphLM.

add_lm_processing_to_buffer_stats(lm_processed)

Update the buffer stats with whether the LM processed an observation.

Add boolean of whether the LM processed an observation on this particular episode step.

Parameters:

lm_processed – Boolean of whether the LM processed an observation on this particular episode step

collect_stats_to_save()

Get all stats that this LM should store in the buffer for logging.

Returns:

Stats to store in the buffer.

exploratory_step(observations)

Step without trying to recognize object (updating possible matches).

get_all_known_object_ids()

Get the IDs of all object models stored in memory.

Returns:

IDs of all object models stored in memory.

get_graph(model_id, input_channel=None)

Get learned graph from graph memory.

Note

May generalize this in the future to get_object_model which doesn’t have to be a graph but currently a lot of code expects a graph to be returned so this name is more meaningful.

Returns:

Graph.

get_input_channels_in_graph(model_id)

Get input channels stored for a graph in graph memory.

Returns:

Input channels stored for a graph in graph memory.

get_object_scale(object_id)

Get object scale. TODO: implement solution for detecting scale.

Returns:

1

get_output()

Return the output of the learning module.

Is currently only implemented for the evidence LM since the other LM versions do not have a notion of MLH and therefore can’t produce an output until the last step of the episode.

get_possible_locations()

get_possible_matches()

Get list of current possible objects.

TODO: Maybe make this private -> check terminal condition

Returns:

List of current possible objects.

get_possible_paths()

Return possible paths for each object.

This is used for logging/plotting and to check if we know where on the object we are.

Returns:

Possible paths for each object.

get_possible_poses(as_euler=True)

Return possible poses for each object (for logging).

Possible poses are narrowed down in the feature matching version. When using displacements or PPF this is empty.

Returns:

Possible poses for each object.

get_unique_pose_if_available(object_id)

Return a 7d pose array if pose is uniquely identified.

This method should return a 7d pose array containing the detected object location, rotation and scale if the pose is uniquely identified. If not, it should contain None. This is used in the Monty class to determine whether we have reached a terminal state.

Returns:

7d pose array or None.

load_state_dict(state_dict)

Load state dict.

Parameters:

state_dict – State dict to load.

matching_step(observations)

Update the possible matches given an observation.

post_episode()

If training, update memory after each episode.

pre_episode(primary_target)

Set target object var and reset others from last episode.

primary_targetthe primary target for the learning module/

Monty system to recognize (e.g. the object the agent begins on, or an important object in the environment; NB that a learning module can also correctly classify a “stepwise_target”, corresponding to the object that it is currently on, while it is attempting to classify the primary_target)

propose_goal_state()

Return the goal-state proposed by this LM’s GSG.

Only returned if the LM/GSG was stepped, otherwise returns None goal-state.

receive_votes(vote_data)

Remove object ids that come in from the votes.

Parameters:

vote_data – set of objects that other LMs excluded from possible matches

reset()

NOTE: currently not used in public interface.

send_out_vote()

Send out list ob objects that are not possible matches.

By sending out the negavtive matches we avoid the problem that every LM needs to know about the same objects. We could think of this as more of an inhibitory signal (I know it can’t be this object so you all don’t need to check that anymore).

Returns:

Set of objects that are not possible matches.

set_detected_object(terminal_state)

Set the current graph ID.

If we didn’t recognize the object this will be new_object{n} where n is len(graph_memory) + 1. Otherwise it is the id of the graph that we recognized. If we timed out it is None and we will not update the graph memory.

set_experiment_mode(mode)

Set LM and GM mode to train or eval.

set_individual_ts(terminal_state)

state_dict()

Get the full state dict for logging and saving.

Returns:

Full state dict for logging and saving.

update_terminal_condition()

Check if we have reached a terminal condition for this episode.

Returns:

Terminal state of the LM.

class GraphMemory(graph_delta_thresholds=None, k=None)

Bases: LMMemory

General GraphMemory that stores & manipulates GraphObjectModel instances.

You can think of the GraphMemory as a library of object models with a librarian managing them. The books ate GraphObjectModel instances. The LearningModule classes access the information stored in the books and can request books to be added to the library.

Subclasses are DisplacementGraphMemory, FeatureGraphMemory and EvidenceGraphMemory.

get_all_models_in_memory()

Return models stored in memory.

get_feature_array(graph_id)

get_feature_order(graph_id)

get_features_at_node(graph_id, input_channel, node_id, feature_keys=None)

Get features at a specific node in the graph.

Parameters:

• graph_id – Name of graph.

• input_channel – Input channel.

• node_id – Node ID of the node to get features from. Can also be an array of node IDs to return an array of features.

• feature_keys – Feature keys.

Returns:

Dict of features at this node.

TODO: look into getting node_id > graph.x.shape[0] (by 1)

get_graph(graph_id, input_channel=None)

Return graph from graph memory.

Parameters:

• graph_id – id of graph to retrieve

• input_channel –

  ?

Raises:

ValueError – If input_channel is defined, not “first”, and not in the graph

get_graph_node_ids(graph_id, input_channel)

get_initial_hypotheses()

get_input_channels_in_graph(graph_id)

get_locations_in_graph(graph_id, input_channel)

get_memory_ids()

Get list of all objects in memory.

Returns:

List of all objects in memory.

get_num_nodes_in_graph(graph_id, input_channel=None)

Get number of nodes in graph.

If input_channel is None, return sum over all input channels for this object.

Returns:

Number of nodes in graph.

initialize_feature_arrays()

load_state_dict(state_dict)

Load graphs from state dict and add to memory.

memory_consolidation()

Is here just as a placeholder.

This could be a function that cleans up graphs in memory to make more efficient use of their nodes by spacing them out evenly along the approximated object surface. It could be something that happens during sleep. During clean up, similar graphs could also be merged.

Q: Should we implement something like this?

remove_graph_from_memory(graph_id)

state_dict()

Return state_dict.

update_memory(locations, features, graph_id, object_location_rel_body, location_rel_model, object_rotation, object_scale)

Determine how to update memory and call corresponding function.

class MontyForGraphMatching(*args, **kwargs)

Bases: MontyBase

General Monty model for recognizing object using graphs.

check_if_any_lms_updated()

True if any LM received sensory information on the current episode step.

Returns:

True if any LM received sensory information on the current episode step, False otherwise.

check_terminal_conditions()

Check if all LMs have reached a terminal state.

This could be no_match, match, or time_out. If all LMs have reached one of these states, end the episode.

Currently the episode just ends if

• min_lms_match lms have reached “match”

• all lms have reached “no_match”

• We have exceeded max_total_steps

Note

In the future we may want to allow ending an episode when all states are either match or no_match. Right now, the match lms will have to convince the no_match lms of their detected object and pose for the episode to end which may be more difficult if not all LMs know about all objects.

Returns:

True if all LMs have reached a terminal state, False otherwise.

deal_with_time_out()

Set LM terminal states to time_out.

load_state_dict_from_parallel(parallel_dirs, save=False)

pre_episode(primary_target, semantic_id_to_label=None)

Reset values and call sub-pre_episode functions.

reset()

Reset monty status.

send_vote_to_lm(lm, lm_id, combined_votes)

Route correct votes to a given LM.

set_is_done()

Set the model is_done flag.

Method that e.g. experiment class can use to set model is_done flag if e.g. total number of episode steps possible has been exceeded

update_stats_after_vote(lm)

Add voting stats to buffer and check individual terminal condition.

LOGGING_REGISTRY = {'BASIC': <class 'tbp.monty.frameworks.loggers.graph_matching_loggers.BasicGraphMatchingLogger'>, 'DETAILED': <class 'tbp.monty.frameworks.loggers.graph_matching_loggers.DetailedGraphMatchingLogger'>, 'SELECTIVE': <class 'tbp.monty.frameworks.loggers.graph_matching_loggers.SelectiveEvidenceLogger'>, 'SILENT': <class 'tbp.monty.frameworks.loggers.exp_logger.BaseMontyLogger'>}

tbp.monty.frameworks.models.monty_base

class LearningModuleBase

Bases: LearningModule

Dummy placeholder class used only for tests.

exploratory_step(inputs)

Model building step called inside of monty._step_learning_modules.

get_output()

Return learning module output (same format as input).

load_state_dict(state_dict)

Take a state dict as an argument and set state for this LM.

matching_step(inputs)

Matching / inference step called inside of monty._step_learning_modules.

post_episode()

Do things like update object models with stored data after an episode.

pre_episode()

Do things like reset buffers or possible_matches before training.

propose_goal_state()

Return the goal-state proposed by this LM’s GSG.

receive_votes(inputs)

Process voting data sent out from other learning modules.

reset()

Do things like reset buffers or possible_matches before training.

send_out_vote()

This method defines what data are sent to other learning modules.

set_experiment_mode(inputs)

Set the experiment mode.

Update state variables based on which method (train or evaluate) is being called at the experiment level.

state_dict()

Return a serializable dict with everything needed to save/load this LM.

class MontyBase(sensor_modules, learning_modules, motor_system, sm_to_agent_dict, sm_to_lm_matrix, lm_to_lm_matrix, lm_to_lm_vote_matrix, min_eval_steps, min_train_steps, num_exploratory_steps, max_total_steps)

Bases: Monty

aggregate_sensory_inputs(observation)

Receive data from dataloader/env, organize on a per sensor module basis.

check_reached_max_matching_steps(max_steps)

Check if max_steps was reached and deal with time_out.

Returns:

True if max_steps was reached, False otherwise.

deal_with_time_out()

Call any functions and logging in case of a time out.

get_agent_state()

Get state of agent (dict).

Returns:

State of the agent.

get_observations(observations, sensor_module_id)

Get observations from all agents pertaining to a single sensor module.

Observations are returned in the format

{“agent_1”:

{“sm_1”:

{“rgba”: data,

“depth”: data “semantic”: data}

} {“sm_2”:

{“rgba”: data, … }

“agent_2”:

{“sm_3”:

{“rgba”: data, … }

… “agent_n”:

{“sm_k”:

{“rgba”: data, … }

}

}

Returns:

Observations from all agents pertaining to a single sensor module.

load_state_dict(state_dict)

Take a state dict as an argument and set state for monty and children.

pass_features_directly_to_motor_system(observation)

Pass features directly to motor system without stepping LMs.

post_episode()

Recursively call post_episode on child classes.

pre_episode()

Recursively call pre_episode on child classes.

reset_episode_steps()

set_done()

set_experiment_mode(mode)

Set the experiment mode.

Update state variables based on which method (train or evaluate) is being called at the experiment level.

state_dict()

Return a serializable dict with everything needed to save/load monty.

step(observation)

Take a matching, exploratory, or custom user-defined step.

Step taken depends on the value of self.step_type.

switch_to_exploratory_step()

switch_to_matching_step()

update_step_counters()

LOGGING_REGISTRY = {'TEST': <class 'tbp.monty.frameworks.loggers.exp_logger.TestLogger'>}

property exceeded_min_steps

property is_done

Return bool to tell the experiment if we are done with this episode.

property is_motor_only_step

property min_steps

property step_type_count

class SensorModuleBase(sensor_module_id)

Bases: SensorModule

post_episode()

This method is called after each episode.

pre_episode()

This method is called before each episode.

set_experiment_mode(mode)

state_dict()

Return a serializable dict with this sensor module’s state.

Includes everything needed to save/load this sensor module.

step(data)

Called on each step.

Parameters:

data – Sensor observations

update_state(state)

tbp.monty.frameworks.models.motor_policies

tbp.monty.frameworks.models.object_model

exception GridTooSmallError

Bases: Exception

Exception raised when grid is too small to fit all observations.

class GraphObjectModel(object_id)

Bases: ObjectModel

Object model class that represents object as graphs.

add_ppf_to_graph()

Add point pair features to graph edges.

build_model(locations, features, k_n, graph_delta_thresholds)

Build graph from locations and features sorted into grids.

get_values_for_feature(feature)

set_graph(graph)

Set self._graph property with given graph (i.e. from pretraining).

update_model(locations, features, location_rel_model, object_location_rel_body, object_rotation, object_scale=1)

Add new locations and features into grids and rebuild graph.

property edge_attr

property edge_index

property feature_ids_in_graph

property feature_mapping

property norm

property num_nodes

property pos

property x

class GridObjectModel(object_id, max_nodes, max_size, num_voxels_per_dim)

Bases: GraphObjectModel

Model of an object and all its functions.

This model has the same basic functionality as the NumpyGraph models used in older LM versions. On top of that we now have a grid representation of the object that constraints the model size and resultion. Additionally, this model class implements a lot of functionality that was previously implemented in the graph_utils.py file.

TODO: General cleanups that require more changes in other code

• remove node_ids from input_channels and have as graph attribute

• remove .norm as attribute and store as feature instead?

build_model(locations, features)

Build graph from locations and features sorted into grids.

find_nearest_neighbors(search_locations, num_neighbors, return_distance=False)

Find nearest neighbors in graph for list of search locations.

Note

This is currently using kd tree search. In the future we may consider doing this directly by indexing the grids. However, an initial implementation of this does not seem to be faster than the kd tree search (~5-10x slower). However one must consider that search directly in the grid would remove the cost of building the tree. TODO: Investigate this further.

Returns:

If return_distance is True, return distances. Otherwise, return indices of nearest neighbors.

set_graph(graph)

Set self._graph property and convert input graph to right format.

update_model(locations, features, location_rel_model, object_location_rel_body, object_rotation)

Add new locations and features into grids and rebuild graph.

tbp.monty.frameworks.models.sensor_modules

class DetailedLoggingSM(sensor_module_id, save_raw_obs, pc1_is_pc2_threshold=10, point_normal_method='TLS', weight_curvature=True)

Bases: SensorModuleBase

Sensor module that keeps track of raw observations for logging.

extract_and_add_features(features, obs_3d, rgba_feat, depth_feat, center_id, center_row_col, sensor_frame_data, world_camera)

Extract features specified in self.features from sensor patch.

Returns the features in the patch, and True if the point-normal or principal curvature directions were ill-defined.

Returns:

The features in the patch. morphological_features: ? invalid_signals: True if the point-normal or principal curvature directions

were ill-defined.

Return type:

features

observations_to_comunication_protocol(data, on_object_only=True)

Turn raw observations into instance of State class following CMP.

Parameters:

• data – Raw observations.

• on_object_only –

  If False, do the following: - If the center of the image is not on the object, but some other part

  of the object is in the image, continue with feature extraction

  • Get the point normal for the whole image, not just the parts of the

    image that include an object.

Returns:

Features and morphological features.

Return type:

State

pre_episode()

Reset buffer and is_exploring flag.

state_dict()

Return state_dict.

step(data)

Add raw observations to SM buffer.

class FeatureChangeSM(sensor_module_id, features, delta_thresholds, surf_agent_sm=False, save_raw_obs=False, noise_params=None)

Bases: HabitatDistantPatchSM, NoiseMixin

Sensor Module that turns Habitat camera obs into features at locations.

Takes in camera rgba and depth input and calculates locations from this. It also extracts features which are currently: on_object, rgba, point_normal, curvature.

check_feature_change(observed_features)

Check feature change between last transmitted observation.

Parameters:

observed_features – Features from the current observation.

Returns:

True if the features have changed significantly.

pre_episode()

Reset buffer and is_exploring flag.

step(data)

Return Features if they changed significantly.

class HabitatDistantPatchSM(sensor_module_id, features, save_raw_obs=False, pc1_is_pc2_threshold=10, noise_params=None, process_all_obs=False)

Bases: DetailedLoggingSM, NoiseMixin

Sensor Module that turns Habitat camera obs into features at locations.

Takes in camera rgba and depth input and calculates locations from this. It also extracts features which are currently: on_object, rgba, point_normal, curvature.

pre_episode()

Reset buffer and is_exploring flag.

state_dict()

Return state_dict.

step(data)

Turn raw observations into dict of features at location.

Parameters:

data – Raw observations.

Returns:

State with features and morphological features. Noise may be added. use_state flag may be set.

update_state(state)

Update information about the sensors location and rotation.

class HabitatSurfacePatchSM(sensor_module_id, features, save_raw_obs=False, noise_params=None)

Bases: HabitatDistantPatchSM

HabitatDistantPatchSM that continues feature extraction when patch not on object.

Identical to HabitatDistantPatchSM except that feature extraction continues even if the center of the sensor patch is not on the object. TODO: remove and replace with surf_agent_sm=True.

class NoiseMixin(noise_params, **kwargs)

Bases: object

add_noise_to_feat_value(feat_name, feat_val)

add_noise_to_sensor_data(sensor_data)

Add noise to features specified in noise_params.

Noise params should have structure {“features”:

{“feature_keys”: noise_amount, …},

“locations”: noise_amount}

noise_amount specifies the standard deviation of the gaussian noise sampled for real valued features. For boolian features it specifies the probability that the boolean flips. If we are dealing with normed vectors (point_normal or curvature_directions) the noise is applied by rotating the vector given a sampled rotation. Otherwise noise is just added onto the perceived feature value.

Parameters:

sensor_data – Sensor data to add noise to.

Returns:

Sensor data with noise added.

tbp.monty.frameworks.models.states

class GoalState(location, morphological_features, non_morphological_features, confidence, use_state, sender_id, sender_type, goal_tolerances)

Bases: State

Specialization of State for goal states with null (None) values allowed.

Specialized form of state that still adheres to the cortical messaging protocol, but can have null (None) values associated with the location and morphological features.

Used by goal-state generators (GSGs) to communicate goal states to other GSGs, and to motor actuators.

The state variables generally have the same meanign as for the base State class, and they represent the target values for the receiving system. Thus if a goal-state specifies a particular object ID (non-morphological feature) in a particular pose (location and morphological features), then the receiving system should attempt to achieve that state.

Note however that for the goal-state, the confidence corresponds to the conviction

with which a GSG believes that the current goal-state should be acted upon. Float bound in [0,1.0].

class State(location, morphological_features, non_morphological_features, confidence, use_state, sender_id, sender_type)

Bases: object

State class used as message packages passed in Monty using CMP.

The cortical messaging protocol (CMP) is used to pass messages between Monty components and makes sure we can easily set up arbitrary configurations of them. This class makes it easier to define the CMP in one place and defines the content and structure of messages passed between Monty components. It also contains some helper funtions to access and modify the message content.

States are represented in this format but can be interpreted by the receiver in different ways:

Observed states: states output py sensor modules Hypothesized states: states output by learning modules Goal states: motor output of learning modules

location

3D vector representing the location of the state

morphological_features

dictionary of morphological features. Should include pose_vectors of shape (3,3) and pose_fully_defined (bool).

non_morphological_features

dictionary of non-morphological features.

confidence

confidence in the state. In range [0,1].

use_state

boolean indicating whether the state should be used or not.

sender_id

string identifying the sender of the state.

sender_type

string identifying the type of sender. Can be “SM” or “LM”.

get_curvature_directions()

Return the curvature direction vectors.

Raises:

ValueError – If self.sender_type is not SM

get_feature_by_name(feature_name)

get_nth_pose_vector(pose_vector_index)

Return the nth pose vector.

When self.sender_type == “SM”, the first pose vector is the point normal and the second and third are the curvature directions. When self.sender_type == “LM”, the pose vectors correspond to the rotation of the object relative to the model learned of it.

get_on_object()

Return whether we think we are on the object or not.

This is currently used in the policy to stay on the object.

get_point_normal()

Return the point normal vector.

Raises:

ValueError – If self.sender_type is not SM

get_pose_vectors()

Return the pose vectors.

set_displacement(displacement, ppf=None)

Add displacement (represented as dict) to state.

TODO S: Add this to state or in another place?

transform_morphological_features(translation=None, rotation=None)

Apply translation and/or rotation to morphological features.