tbp.monty.frameworks.models#
tbp.monty.frameworks.models.abstract_monty_classes#
- class GoalStateGenerator[source]#
Bases:
objectGenerate goal-states that other learning modules and motor-systems will attempt.
Generate goal-states potentially (in the case of LMs) by outputing their own sub-goal-states. Provides a mechanism for implementing hierarchical action policies that are informed by world models/hypotheses.
- class LMMemory[source]#
Bases:
objectLike a long-term memory storing all the knowledge an LM has.
- class LearningModule[source]#
Bases:
object- abstract exploratory_step()[source]#
Model building step called inside of monty._step_learning_modules.
- abstract load_state_dict(state_dict)[source]#
Take a state dict as an argument and set state for this LM.
- abstract matching_step()[source]#
Matching / inference step called inside of monty._step_learning_modules.
- abstract post_episode()[source]#
Do things like update object models with stored data after an episode.
- class Monty[source]#
Bases:
object- abstract aggregate_sensory_inputs(observation)[source]#
Receive data from dataloader/env, organize on a per sensor module basis.
- abstract load_state_dict(state_dict)[source]#
Take a state dict as an argument and set state for monty and children.
- abstract set_experiment_mode(mode)[source]#
Set the experiment mode.
Update state variables based on which method (train or evaluate) is being called at the experiment level.
tbp.monty.frameworks.models.buffer#
- class BufferEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]#
Bases:
JSONEncoderEncoder to turn the buffer into a JSON compliant format.
- default(obj: Any) Any[source]#
Turn non-compliant types into a JSON-compliant format.
- Parameters:
obj – The object to turn into a JSON-compliant format.
- Returns:
The JSON-compliant data.
- classmethod register(obj_type: type, encoder: Callable | Type[JSONEncoder]) None[source]#
Register an encoder.
- Parameters:
obj_type – The type to associate with encoder.
encoder – A function or JSONEncoder class that converts objects of type obj_type into JSON-compliant data.
- Raises:
TypeError – If encoder is not a JSONEncoder subclass or a callable.
- class FeatureAtLocationBuffer[source]#
Bases:
BaseBufferBuffer which stores features at locations coming into one LM. Also stores stats.
Used for building graph models and logging detailed stats about an episode. The location buffer is also used to calculate displacements.
- append(list_of_data)[source]#
Add an observation to the buffer. Must be features at locations.
TODO S: Store state objects instead of list of data? A provisional version of this is implemented below, as the GSG uses State objects for computations.
- get_all_features_on_object()[source]#
Get all observed features that were on the object.
Like in get_all_locations_on_object the feature arrays should have the length of np.where(self.on_object) and contain all features that were observed when at least one of the input channels was on the object. However, we also check for each feature whether its input channel indicated on_object=True and if not set its value to nan.
Note
Since all inputs to an LM should have overlapping receptive fields, there should be no difference between the two at the moment (except due to noisy estimates of on_object).
- Returns:
All observed features that were on the object.
- get_all_input_states()[source]#
Get all the input states that the buffer’s parent LM has observed.
- Returns:
All the input states that the buffer’s parent LM has observed.
- get_all_locations_on_object(input_channel=None)[source]#
Get all observed locations that were on the object.
- Returns:
All observed locations that were on the object.
- get_buffer_len_by_channel(input_channel)[source]#
Return the number of observations stored for that input channel.
- get_current_displacement(input_channel)[source]#
Get the current displacement.
- Returns:
The current displacement.
- get_current_features(keys)[source]#
Get the current value of a specific feature.
- Returns:
The current features.
- get_current_location(input_channel)[source]#
Get the current location.
Note
May have to add on_object check at some point.
- Returns:
The current location.
- get_current_pose(input_channel)[source]#
Get currently sensed location and orientation.
- Returns:
The currently sensed location and orientation.
- get_currently_on_object()[source]#
Check whether last sensation was on object.
- Returns:
Whether the last sensation was on object.
- get_first_displacement_len(input_channel)[source]#
Get length of first observed displacement.
Use for scale in DisplacementLM.
- Returns:
The length of the first observed displacement.
- get_first_sensory_input_channel()[source]#
Get name of first sensory (coming from SM) input channel in buffer.
- Returns:
The name of the first sensory (coming from SM) input channel in buffer.
- Raises:
ValueError – If no sensor channels are found in the buffer
- get_last_obs_processed()[source]#
Check whether last sensation was processed by LM.
- Returns:
Whether the last sensation was processed by the LM.
- get_matching_step_when_output_goal_set()[source]#
Return matching step when last goal-state was generated.
Return the LM matching step associated with the last time a goal-state was generated.
- get_nth_displacement(n, input_channel)[source]#
Get the nth displacement.
- Returns:
The nth displacement.
- get_num_goal_states_generated()[source]#
Return number of goal states generated by the LM’s GSG since episode start.
Note use of length not sum.
- get_num_matching_steps()[source]#
Return number of matching steps performed by this LM since episode start.
- Returns:
The number of matching steps performed by this LM since episode start.
- get_num_observations_on_object()[source]#
Get the number of steps where at least 1 input was on the object.
- Returns:
The number of steps where at least 1 input was on the object.
- get_num_steps_post_output_goal_generated()[source]#
Return number of steps since last output goal-state.
Return the number of Monty-matching steps that have elapsed since the last time an output goal-state was generated.
- get_previous_input_states()[source]#
Get previous State inputs received by the buffer’s parent LM.
i.e. in the last time step.
- Returns:
The previous input states.
- set_individual_ts(object_id, pose)[source]#
Update self.stats with the individual LMs terminal state.
tbp.monty.frameworks.models.displacement_matching#
- class DisplacementGraphLM(k=None, match_attribute=None, tolerance=0.001, use_relative_len=False, graph_delta_thresholds=None)[source]#
Bases:
GraphLMLearning module that uses displacement stored in graphs to recognize objects.
- get_object_rotation(sensed_displacements, model_displacements, get_reverse_r=False)[source]#
Calculate the rotation between two sets of displacement vectors.
- Parameters:
sensed_displacements – The displacements that were sensed.
model_displacements – The displacements in the model that were matched to the sensed displacements.
get_reverse_r – Whether to get the rotation that turns the model such that it would produce the sensed_displacements (False) or the rotation needed to turn the sensed_displacements into the model displacements.
- Returns:
The rotation in Euler angles, as a matrix, and as a Rotation object.
- get_object_scale(sensed_displacement, model_displacement)[source]#
Calculate the objects scale given sensed and model displacements.
- Returns:
The scale of the object.
tbp.monty.frameworks.models.evidence_matching#
- class EvidenceGraphLM(max_match_distance, tolerances, feature_weights, feature_evidence_increment=1, max_nneighbors=3, initial_possible_poses='informed', evidence_update_threshold='all', vote_evidence_threshold=0.8, past_weight=1, present_weight=1, vote_weight=1, object_evidence_threshold=1, x_percent_threshold=10, path_similarity_threshold=0.1, pose_similarity_threshold=0.35, required_symmetry_evidence=5, graph_delta_thresholds=None, max_graph_size=0.3, max_nodes_per_graph=2000, num_model_voxels_per_dim=50, use_multithreading=True, gsg_class=<class 'tbp.monty.frameworks.models.goal_state_generation.EvidenceGoalStateGenerator'>, gsg_args=None, *args, **kwargs)[source]#
Bases:
GraphLMLearning module that accumulates evidence for objects and poses.
- Matching Attributes:
- max_match_distance: Maximum distance of a tested and stored location to
be matched.
- tolerances: How much can each observed feature deviate from the stored
features to still be considered a match.
- feature_weights: How much should each feature be weighted when calculating
the evidence update for hypotheses. Weights are stored in a dictionary with keys corresponding to features (same as keys in tolerances)
- feature_evidence_increment: Feature evidence (between 0 and 1) is multiplied
by this value before being added to the overall evidence of a hypothesis. This factor is only multiplied with the feature evidence (not the pose evidence as opposed to the present_weight).
- max_nneighbors: Maximum number of nearest neighbors to consider in the
radius of a hypothesis for calculating the evidence.
- initial_possible_poses: initial possible poses that should be tested for.
In [“uniform”, “informed”, list]. default = “informed”.
- evidence_update_threshold: How to decide which hypotheses should be updated.
When this parameter is either ‘[int]%’ or ‘x_percent_threshold’, then this parameter is applied to the evidence for the Most Likely Hypothesis (MLH) to determine a minimum evidence threshold in order for other hypotheses to be updated. Any hypotheses falling below the resulting evidence threshold do not get updated. The other options set a fixed threshold that does not take MLH evidence into account. In [int, float, ‘[int]%’, ‘mean’, ‘median’, ‘all’, ‘x_percent_threshold’].
- vote_evidence_threshold: Only send votes that have a scaled evidence above
this threshold. Vote evidences are in the range of [-1, 1] so the threshold should not be outside this range.
- past_weight: How much should the evidence accumulated so far be weighted
when combined with the evidence from the most recent observation.
- present_weight: How much should the current evidence be weighted when added
to the previous evidence. If past_weight and present_weight add up to 1, the evidence is bounded and can’t grow infinitely. NOTE: right now this doesn’t give as good performance as with unbounded evidence since we don’t keep a full history of what we saw. With a more efficient policy and better parameters that may be possible to use though and could help when moving from one object to another and to generally make setting thresholds etc. more intuitive.
- vote_weight: Vote evidence (between -1 and 1) in multiplied by this value
when being added to the overall evidence of a hypothesis. If past and current_weight add up to 1, it is use as weight in np.average to keep the evidence in a fixed range.
- Terminal Condition Attributes:
- object_evidence_threshold: Minimum required evidence for an object to be
recognized. We additionally check that the evidence for this object is significantly higher than for all other objects.
- x_percent_threshold: Used in two places:
- All objects whose highest evidence is greater than the most likely
objects evidence - x_percent of the most like objects evidence are considered possible matches. That means to only have one possible match, no other object can have more evidence than the candidate match’s evidence - x percent of it.
- Within one object, possible poses are considered possible if their
evidence is larger than the most likely pose of this object - x percent of this poses evidence.
# TODO: should we use a separate threshold for within and between objects? If this value is larger, the model is usually more robust to noise and reaches a better performance but also requires a lot more steps to reach a terminal condition, especially if there are many similar object in the data set.
- path_similarity_threshold: How similar do paths have to be to be
considered the same in the terminal condition check.
- pose_similarity_threshold: difference between two poses to be considered
unique when checking for the terminal condition (in radians).
- required_symmetry_evidence: number of steps with unchanged possible poses
to classify an object as symmetric and go into terminal condition.
- Model Attributes:
- graph_delta_thresholds: Thresholds used to compare nodes in the graphs being
learned, and thereby whether to include a new point or not. By default, we only consider the distance between points, using a threshold of 0.001 (determined in remove_close_points). Can also specify thresholds based on e.g. point-normal angle difference, or principal curvature magnitude difference.
- max_graph_size: Maximum size of a graph in meters. Any observations that fall
out of this range will be discarded/used for building a new model. This constraints the size of models that an LM can learn and enforces learning models of sub-components of objects.
- max_nodes_per_graph: Maximum number of nodes in a graph. This will be k when
picking the k-winner voxels to add their content into the graph used for matching.
- num_model_voxels_per_dim: Number of voxels per dimension in the model grid.
This constraints the spatial resolution that the model can represent. max_graph_size/num_model_voxels_per_dim = how much space is lumped into one voxel. All locations that fall into the same voxel will be averaged and represented as one value. num_model_voxels_per_dim should not be too large since the memory requirements grow cubically with this number.
gsg_class: The type of goal-state-generator to associate with the LM. gsg_args: Dictionary of configuration parameters for the GSG.
- Debugging Attributes:
- use_multithreading: Whether to calculate evidence updates for different
objects in parallel using multithreading. This can be done since the updates to different objects are completely independent of each other. In general it is recommended to use this but it can be useful to turn it off for debugging purposes.
- collect_stats_to_save()[source]#
Get all stats that this LM should store in the buffer for logging.
- Returns:
The stats dictionary.
- get_current_mlh()[source]#
Return the current most likely hypothesis of the learning module.
- Returns:
graph_id, location, rotation, scale, evidence
- Return type:
dict with keys
- get_mlh_for_object(object_id)[source]#
Get mlh for a specific object ID.
Note
When trying to retrieve the MLH for the current most likely object and not any other object, it is better to use self.current_mlh
- Returns:
The most likely hypothesis for the object ID.
- get_output()[source]#
Return the most likely hypothesis in same format as LM input.
The input to an LM at the moment is a dict of features at a location. The output therefor has the same format to keep the messaging protocol consistent and make it easy to stack multiple LMs on top of each other.
If the evidence for mlh is < object_evidence_threshold, interesting_features == False
- get_possible_poses(as_euler=True)[source]#
Return possible poses for each object (for logging).
Here this list doesn’t get narrowed down. This is not really used for evidence matching since we threshold in other places.
- get_top_two_mlh_ids()[source]#
Retrieve the two most likely object IDs for this LM.
- Returns:
The two most likely object IDs.
- get_top_two_pose_hypotheses_for_graph_id(graph_id)[source]#
Return top two hypotheses for a given graph_id.
- get_unique_pose_if_available(object_id)[source]#
Get the most likely pose of an object if narrowed down.
If there is not one unique possible pose or symmetry detected, return None
- Returns:
The pose and scale if a unique pose is available, otherwise None.
- receive_votes(vote_data)[source]#
Get evidence count votes and use to update own evidence counts.
Weighted by distance to votes and their evidence. TODO: also take into account rotation vote
- vote_data contains:
pos_location_votes: shape=(N, 3) pos_rotation_votes: shape=(N, 3, 3) pose_evidences: shape=(N,)
- send_out_vote()[source]#
Send out hypotheses and the evidence for them.
- Votes are a dict and contain the following:
pose_hypotheses: locations (V, 3) and rotations (V, 3, 3) pose_evidence: Evidence (V) for each location-rotation pair in the
pose hypotheses. Scaled into range [-1, 1] where 1 is the hypothesis with the largest evidence in this LM and -1 the one with the smallest evidence. When thresholded, pose_evidence will be in range [self.vote_evidence_threshold, 1]
- sensed_pose_rel_body: sensed location and rotation of the input to this
LM. Rotation is represented by the pose vectors (point normal and curvature directions) for the SMs. For input from LMs it is also represented as 3 unit vectors, these are calculated from the estimated rotation of the most likely object. This pose is used to calculate the displacement between two voting LMs and to translate the votes between their reference frames. Shape=(4,3).
Where V is the number of votes (V=number of hypotheses if not thresholded) If none of the hypotheses of an object are > vote_evidence_threshold, this object will not send out a vote.
- Returns:
possible_states: The possible states. sensed_pose_rel_body: The sensed pose relative to the body.
- Return type:
None or dict
- class EvidenceGraphMemory(max_nodes_per_graph, max_graph_size, num_model_voxels_per_dim, *args, **kwargs)[source]#
Bases:
GraphMemoryCustom GraphMemory that stores GridObjectModel instead of GraphObjectModel.
- class MontyForEvidenceGraphMatching(*args, **kwargs)[source]#
Bases:
MontyForGraphMatchingMonty model for evidence based graphs.
Customize voting and union of possible matches.
tbp.monty.frameworks.models.evidence_sdr_matching#
- class EncoderSDR(sdr_length=2048, sdr_on_bits=41, lr=0.01, n_epochs=1000, stability=0.0, log_flag=False)[source]#
Bases:
objectThe SDR Encoder class.
This class keeps track of the dense representations, and trains them to output SDRs when binarized. This class also contains its own optimizer and function to add more objects/representations.
The representations are stored as dense vectors and binarized using top-k to convert them to SDRs. During training, the pairwise overlaps between the sdrs are compared to the target overlaps. This error signal trains the dense representations.
Refer to the self.train_sdrs function for more information on the training details
- sdr_length#
The size of the SDRs (total number of bits).
- sdr_on_bits#
The number of on bits in the SDRs. Controls sparsity.
- lr#
The learning rate of the encoding algorithm.
- n_epochs#
The number of training epochs per episode
- stability#
The stability parameter controls by how much old SDRs change relative to new SDRs. Value range is [0.0, 1.0], where 0.0 is no stability constraint applied and 1.0 is fixed SDRs. Values in between are for partial stability.
- log_flag#
Flag to activate the logger.
- add_objects(n_objects)[source]#
Adds more objects to the available objects and re-initializes the optimizer.
We keep track of the stable representation ids (old objects) when adding new objects.
- Parameters:
n_objects – Number of objects to add
- binarize(emb)[source]#
Convert dense representations to SDRs (0s and 1s) using Top-k function.
- Returns:
The SDRs.
- get_sdr(index)[source]#
Return the SDR at a specific index.
This index refers to the object index in the SDRs dictionary (i.e., self.obj_sdrs)
- optimize(overlap_error, mask)[source]#
Compute and apply local gradient descent.
Compute based on the overlap error and mask.
Note there is no use of the chain rule, i.e. each SDR is optimized based on the derivative of its clustering error with respect to its values, with no intermediary functions.
The overlap error helps correct the sign and also provides a magnitude for the representation updates.
- Parameters:
overlap_error – The difference between target and predicted overlaps.
mask – Mask indicating valid entries in the overlap matrix.
Note
num_objects = self.n_objects
Note
A vectorized version of the algorithm is provided below, although it would need to be modified to avoid repeated creation of arrays in order to be more efficient. Leaving for now as this algorithm is not a bottle-neck (circa 10-20 seconds to learn 60 object SDRs).:
# Initialize gradients grad = np.zeros_like(self.obj_sdrs)
# Compute the pairwise differences between SDRs diff_matrix = (
self.obj_sdrs[:, np.newaxis, :] - self.obj_sdrs[np.newaxis, :, :]
)
# Compute the absolute differences for each pair abs_diff = np.sum(np.abs(diff_matrix), axis=2)
# Create a mask for non-zero differences non_zero_mask = abs_diff > 0
# Apply the mask to the original mask valid_mask = mask & non_zero_mask
# Calculate the summed distance and gradient contributions where the mask # is valid for logging summed_distance = np.sum(overlap_error * valid_mask * abs_diff)
# Calculate the gradients grad_contrib = overlap_error[:, :, np.newaxis] * 2 * diff_matrix grad += np.sum(grad_contrib * valid_mask[:, :, np.newaxis], axis=1) grad -= np.sum(grad_contrib * valid_mask[:, :, np.newaxis], axis=0)
# Update the SDRs using the gradient self.obj_sdrs -= self.lr * grad
- Returns:
The summed distance for logging.
- train_sdrs(target_overlaps, log_epoch_every=10)[source]#
Main SDR training function.
This function receives a copy of the average target overlap 2D tensor and trains the sdr representations for n_epochs to achieve these target overlap scores.
We use the overlap target as a learning signal to move the dense representations towards or away from each other. The magnitude of the overlap error controls the strength of moving dense representations. Also the sign of the overlap error controls whether the representations will be moving towards or away from each other.
We want to limit the amount by which trained representation change relative to untrained object representations such that higher-level LMs would not suffer from significant changes in lower-level representations that were used to build higher-level graphs.
When adding new representations, we keep track of the ids of the older representations (i.e., self.stable_ids). This allows us to control by how much the older representations move relative to the newer ones during training. This behavior is controlled by the stability value. During each training iteration, we update these older representations with an average of the optimizer output and the original representation (weighted by the stability value). Note that too much stability restricts the SDRs from adapting to desired changes in the target overlaps caused by normalization or distribution shift, affecting the overall encoding performance.
Consider two dense representations, A_dense and B_dense. We apply top-k operation on both to convert them to A_sdr and B_sdr, then calculate their overlaps. If the overlap is less than the target overlap, we move dense representations (A_dense and B_dense) closer to eachother with strength proportional to the error in overlaps. We move them apart if they have more overlap than the target.
Note
The distance_matrix variable is calculated using the cdist function and it denotes the pairwise euclidean distances between dense representations. The term “overlap” always refers to the overlap in bits between SDRs.
Note
The overlap_error is only used to weight the distance_matrix for each pair of objects, and gradients do not flow through the sparse overlap calculations.
- Returns:
The stats dictionary for logging.
- property n_objects#
Return the available number of objects.
- property sdrs#
Return the available SDRs.
- class EvidenceSDRGraphLM(*args, **kwargs)[source]#
Bases:
EvidenceSDRLMMixin,EvidenceGraphLMClass that incorporates the EvidenceSDR Mixin with the EvidenceGraphLM.
- class EvidenceSDRLMMixin(*args, **kwargs)[source]#
Bases:
objectThis Mixin adds training of SDR representations to the EvidenceGraphLM.
It overrides the __init__ and post_episode functions of the LM
To use this Mixin, pass the EvidenceSDRGraphLM class as the learning_module_class in the learning_module_configs.
Additionally pass the sdr_args dictionary as an additional key in the learning_module_args.
- The sdr_args dictionary should contain:
- log_path (string): A string that points to a temporary location for saving
experiment logs. “None” means don’t save to file
sdr_length (int): The size of the SDR to be used for encoding
sdr_on_bits (int): The number of active bits to be used with these SDRs
sdr_lr (float): The learning rate of the encoding algorithm
n_sdr_epochs (int): The number of epochs to train the encoding algorithm
- stability (float): Stability of older object SDRs.
Value range is [0.0, 1.0], where 0.0 is no stability applied and 1.0 is fixed SDRs.
sdr_log_flag (bool): Flag indicating whether to log the results or not
See the monty_lab repo for reference. Specifically, experiments/configs/evidence_sdr_evaluation.py
- collect_evidences()[source]#
Collect evidence scores from the Learning Module.
- We do this in three steps:
- Step 1: We use the number of objects in the LM
to update the sdr_encoder and id <-> obj tracking dictionaries, as well as the target overlap tensor
- Step 2: We collect evidences relative to the current
most likely hypothesis (mlh). Evidences are stored in a 2d tensor.
- Step 3: We use the stored evidences to update the target overlap
which stores the running average. Refer to EvidenceSDRTargetOverlaps for more details.
Note: We sort the ids in step 2 because the overlap values are suppossed to be symmetric (e.g., “2,5” = “5,2”). This way the target overlaps for the ids “x,y” and “y,x” will be averaged together in the EvidenceSDRTargetOverlaps class.
- class EvidenceSDRTargetOverlaps[source]#
Bases:
objectKeep track of the running average of target overlaps for each episode.
The target overlaps is implemented as a 2D tensor where the indices of the tensor represent the ids of the objects, and the values of the tensor represent a running average of the overlap target.
To achieve this, we implement functions for expanding the size of the overlap target tensor, linear mapping for normalization, and updating the overlap tensor (i.e., running average).
Note
We are averaging over multiple overlap targets. Multiple targets can happen for different reasons:
- Asymmetric evidences: the target overlap for object 1 w.r.t object 2
([2,1]) is averaged with object 2 w.r.t object 1 ([1,2]). This is possible because we sort the ids when we add the evidences to overlaps. Both evidences get added to the location [1,2].
- Additional episodes with similar MLO (most-likely object): More episodes
can accumulate additional evidences on to the same key if the MLO is similar to previous MLO of another episode.
- add_evidence(evidence, mapping_output_range)[source]#
Main function for updating the running average with evidence.
This function receives as input the relative evidence scores and maps them to overlaps in the mapping_output_range. The mapped overlaps are added to the running average in the function add_overlaps.
- add_objects(new_size)[source]#
Expands the overlaps and the counts 2D tensors to accomodate new objects.
- add_overlaps(mapped_overlaps)[source]#
Main function for updating the running average with overlaps.
The running average equation we use is: new_average = ((old_average * counts) + (new_val * 1))/ (counts + 1)
This calculates equally-weighted average, assuming that we keep track of the counts and increment them every time we add a new value to the average.
- map_to_overlaps(evidence, output_range)[source]#
Linear mapping of values from input range to output range.
Only applies to real values (i.e., ignores nan values).
- Returns:
?
- property overlaps#
Returns the target overlap values rounded to the nearest integer.
- class LoggerSDR(path)[source]#
Bases:
objectA simple logger that saves the data passed to it.
This logger maintains an episode counter and logs the data it receives under different files named by the episode counter.
See more information about what data is being logged under the `log_episode` function.
- log_episode(data)[source]#
Receives data dictionary and saves it as a pth file.
This function will save all the data passed to it. Here is a breakdown of the data to be logged by this function.
- The data dictionary contains these key-value pairs:
- mask: 2d tensor of the available overlap targets after this
episode
- target_overlap: 2d tensor of the target overlap at the end of
this episode
- training: Dictionary of training statistics for every epoch.
Includes overlap_error, training_summed_distance, dense representations, and sdrs
obj2id: Objects to ids dictionary mapping
id2obj: Ids to objects dictionary mapping
tbp.monty.frameworks.models.feature_location_matching#
- class FeatureGraphLM(max_match_distance, tolerances, path_similarity_threshold=0.1, pose_similarity_threshold=0.35, required_symmetry_evidence=5, graph_delta_thresholds=None, initial_possible_poses='informed')[source]#
Bases:
GraphLMLearning module that uses features at locations to recognize objects.
- get_object_rotation(graph_id, get_reverse_r=False)[source]#
Get the rotation of an object from the possible poses if resolved.
This first checks whether we have recognized a unique pose of the object or if a symmetry is detected. If one of the two is true it returns the unique rotation(s), otherwise returns None.
- Parameters:
graph_id – The object to check poses for.
get_reverse_r – Whether to get the rotation that turns the model such that it would produce the sensed_displacements (False) or the rotation needed to turn the sensed_displacements into the model displacements.
- Returns:
The rotation of the object if we know it.
- get_unique_pose_if_available(object_id)[source]#
Get the pose of an object if we know it.
Scale is not implemented.
- Returns:
The pose of the object if we know it.
- receive_votes(vote_data)[source]#
Use votes to remove objects and poses from possible matches.
- NOTE: Add object back into possible matches if majority of other modules
think it is correct? Could help with dealing with noise but may also prevent LMs from narrowing down quickly. Since we are not working with this LM anymore, we probably wont add that.
- Parameters:
vote_data – positive and negative votes on object IDs + positive votes for locations and rotations on the object.
- send_out_vote()[source]#
Send out list of objects that are not possible matches.
By sending out the negative matches we avoid the problem that every LM needs to know about the same objects. We could think of this as more of an inhibitory signal (I know it can’t be this object so you all don’t need to check that anymore).
- Returns:
List of objects that are not possible matches.
- NUM_OTHER_LMS = 4#
- class FeatureGraphMemory(graph_delta_thresholds)[source]#
Bases:
GraphMemoryGraph memory that matches objects by using features at locations.
tbp.monty.frameworks.models.goal_state_generation#
- class EvidenceGoalStateGenerator(parent_lm, goal_tolerances=None, elapsed_steps_factor=10, min_post_goal_success_steps=numpy.inf, x_percent_scale_factor=0.75, desired_object_distance=0.03, wait_growth_multiplier=2, **kwargs)[source]#
Bases:
GraphGoalStateGeneratorGenerator of goal states for an evidence-based graph LM.
GSG specifically setup for generating goal states for an evidence-based graph LM, which can therefore leverage the hypothesis-testing action policy. This policy uses hypotheses about the most likely objects, as well as knowledge of their structure from long-term memory, to propose test-points that should efficiently disambiguate the ID or pose of the object the agent is currently observing.
TODO M separate out the hypothesis-testing policy (which is one example of a model-based policy), from the GSG, which is the system that is capable of leveraging a variety of model-based policies.
- class GraphGoalStateGenerator(parent_lm, goal_tolerances=None, **kwargs)[source]#
Bases:
GoalStateGeneratorGenerate sub-goal states until the received goal state is achieved.
A component associated with each learning module that receives a high level goal state, and generates sub-goal states until the received goal state is achieved.
- Generated goal-states are received by either:
i) other learning modules, which may model world-objects (e.g. a mug), or may model internal systems (e.g. the agent’s robotic limb) ii) motor actuators, in which case they represent simpler, primitive goal-states for the actuator to achieve (e.g. location and orientation of an actuator-sensor pair)
As well as the high-level, “driving” goal-state, generated goal-states can also be conditioned on other information, such as the LMs current most-likely hypothesis, and the structure of known object models (i.e. information local to the LM).
Note all goal-states conform to the State-class cortical messaging protocol (CMP).
- get_output_goal_state()[source]#
Retrieve the output goal-state of the GSG.
This is the goal-state projected to other LM’s GSGs +/- motor-actuators.
- Returns:
Output goal-state of the GSG.
- set_driving_goal_state(received_goal_state)[source]#
Receive a new high-level goal to drive this goal-state-generator (GSG).
If none is provided, the goal-state generator should default to pursuing a goal-state of high confidence, with no other attributes of the state specified; in other words, it attempts to reduce uncertainty about the LM’s output (object ID and pose, whatever these may be).
TODO M: Currently GSGs always use the default, however future work will implement hierarchical action policies/GSGs, as well as the ability to specify a top goal-state by the experimenter.
TODO M : we currently just use “None” as a placehodler for the default goal-state > plan : set the default driving goal-state to a meaningful, non-None value that is compatible with the current method for checking convergence of an LM, such that achieving the driving goal-state can be used as a test for Monty convergence. This might be something like the below.
tbp.monty.frameworks.models.graph_matching#
- class GraphLM(initialize_base_modules=True)[source]#
Bases:
LearningModuleGeneral Learning Module that contains a graph memory.
Subclasses are DisplacementGraphLM, FeatureGraphLM, and EvidenceGraphLM.
- add_lm_processing_to_buffer_stats(lm_processed)[source]#
Update the buffer stats with whether the LM processed an observation.
Add boolean of whether the LM processed an observation on this particular episode step.
- Parameters:
lm_processed – Boolean of whether the LM processed an observation on this particular episode step
- collect_stats_to_save()[source]#
Get all stats that this LM should store in the buffer for logging.
- Returns:
Stats to store in the buffer.
- exploratory_step(observations)[source]#
Step without trying to recognize object (updating possible matches).
- get_all_known_object_ids()[source]#
Get the IDs of all object models stored in memory.
- Returns:
IDs of all object models stored in memory.
- get_graph(model_id, input_channel=None)[source]#
Get learned graph from graph memory.
Note
May generalize this in the future to get_object_model which doesn’t have to be a graph but currently a lot of code expects a graph to be returned so this name is more meaningful.
- Returns:
Graph.
- get_input_channels_in_graph(model_id)[source]#
Get input channels stored for a graph in graph memory.
- Returns:
Input channels stored for a graph in graph memory.
- get_object_scale(object_id)[source]#
Get object scale. TODO: implement solution for detecting scale.
- Returns:
1
- get_output()[source]#
Return the output of the learning module.
Is currently only implemented for the evidence LM since the other LM versions do not have a notion of MLH and therefore can’t produce an output until the last step of the episode.
- get_possible_matches()[source]#
Get list of current possible objects.
TODO: Maybe make this private -> check terminal condition
- Returns:
List of current possible objects.
- get_possible_paths()[source]#
Return possible paths for each object.
This is used for logging/plotting and to check if we know where on the object we are.
- Returns:
Possible paths for each object.
- get_possible_poses(as_euler=True)[source]#
Return possible poses for each object (for logging).
Possible poses are narrowed down in the feature matching version. When using displacements or PPF this is empty.
- Returns:
Possible poses for each object.
- get_unique_pose_if_available(object_id)[source]#
Return a 7d pose array if pose is uniquely identified.
This method should return a 7d pose array containing the detected object location, rotation and scale if the pose is uniquely identified. If not, it should contain None. This is used in the Monty class to determine whether we have reached a terminal state.
- Returns:
7d pose array or None.
- pre_episode(primary_target)[source]#
Set target object var and reset others from last episode.
- primary_targetthe primary target for the learning module/
Monty system to recognize (e.g. the object the agent begins on, or an important object in the environment; NB that a learning module can also correctly classify a “stepwise_target”, corresponding to the object that it is currently on, while it is attempting to classify the primary_target)
- propose_goal_state()[source]#
Return the goal-state proposed by this LM’s GSG.
Only returned if the LM/GSG was stepped, otherwise returns None goal-state.
- receive_votes(vote_data)[source]#
Remove object ids that come in from the votes.
- Parameters:
vote_data – set of objects that other LMs excluded from possible matches
- send_out_vote()[source]#
Send out list ob objects that are not possible matches.
By sending out the negavtive matches we avoid the problem that every LM needs to know about the same objects. We could think of this as more of an inhibitory signal (I know it can’t be this object so you all don’t need to check that anymore).
- Returns:
Set of objects that are not possible matches.
- set_detected_object(terminal_state)[source]#
Set the current graph ID.
If we didn’t recognize the object this will be new_object{n} where n is len(graph_memory) + 1. Otherwise it is the id of the graph that we recognized. If we timed out it is None and we will not update the graph memory.
- class GraphMemory(graph_delta_thresholds=None, k=None)[source]#
Bases:
LMMemoryGeneral GraphMemory that stores & manipulates GraphObjectModel instances.
You can think of the GraphMemory as a library of object models with a librarian managing them. The books ate GraphObjectModel instances. The LearningModule classes access the information stored in the books and can request books to be added to the library.
Subclasses are DisplacementGraphMemory, FeatureGraphMemory and EvidenceGraphMemory.
- get_features_at_node(graph_id, input_channel, node_id, feature_keys=None)[source]#
Get features at a specific node in the graph.
- Parameters:
graph_id – Name of graph.
input_channel – Input channel.
node_id – Node ID of the node to get features from. Can also be an array of node IDs to return an array of features.
feature_keys – Feature keys.
- Returns:
Dict of features at this node.
TODO: look into getting node_id > graph.x.shape[0] (by 1)
- get_graph(graph_id, input_channel=None)[source]#
Return graph from graph memory.
- Parameters:
graph_id – id of graph to retrieve
input_channel –
?
- Raises:
ValueError – If input_channel is defined, not “first”, and not in the graph
- get_memory_ids()[source]#
Get list of all objects in memory.
- Returns:
List of all objects in memory.
- get_num_nodes_in_graph(graph_id, input_channel=None)[source]#
Get number of nodes in graph.
If input_channel is None, return sum over all input channels for this object.
- Returns:
Number of nodes in graph.
- memory_consolidation()[source]#
Is here just as a placeholder.
This could be a function that cleans up graphs in memory to make more efficient use of their nodes by spacing them out evenly along the approximated object surface. It could be something that happens during sleep. During clean up, similar graphs could also be merged.
Q: Should we implement something like this?
- class MontyForGraphMatching(*args, **kwargs)[source]#
Bases:
MontyBaseGeneral Monty model for recognizing object using graphs.
- check_if_any_lms_updated()[source]#
True if any LM received sensory information on the current episode step.
- Returns:
True if any LM received sensory information on the current episode step, False otherwise.
- check_terminal_conditions()[source]#
Check if all LMs have reached a terminal state.
This could be no_match, match, or time_out. If all LMs have reached one of these states, end the episode.
- Currently the episode just ends if
min_lms_match lms have reached “match”
all lms have reached “no_match”
We have exceeded max_total_steps
Note
In the future we may want to allow ending an episode when all states are either match or no_match. Right now, the match lms will have to convince the no_match lms of their detected object and pose for the episode to end which may be more difficult if not all LMs know about all objects.
- Returns:
True if all LMs have reached a terminal state, False otherwise.
- pre_episode(primary_target, semantic_id_to_label=None)[source]#
Reset values and call sub-pre_episode functions.
- set_is_done()[source]#
Set the model is_done flag.
Method that e.g. experiment class can use to set model is_done flag if e.g. total number of episode steps possible has been exceeded
- update_stats_after_vote(lm)[source]#
Add voting stats to buffer and check individual terminal condition.
- LOGGING_REGISTRY = {'BASIC': <class 'tbp.monty.frameworks.loggers.graph_matching_loggers.BasicGraphMatchingLogger'>, 'DETAILED': <class 'tbp.monty.frameworks.loggers.graph_matching_loggers.DetailedGraphMatchingLogger'>, 'SELECTIVE': <class 'tbp.monty.frameworks.loggers.graph_matching_loggers.SelectiveEvidenceLogger'>, 'SILENT': <class 'tbp.monty.frameworks.loggers.exp_logger.BaseMontyLogger'>}#
tbp.monty.frameworks.models.monty_base#
- class LearningModuleBase[source]#
Bases:
LearningModuleDummy placeholder class used only for tests.
- exploratory_step(inputs)[source]#
Model building step called inside of monty._step_learning_modules.
- matching_step(inputs)[source]#
Matching / inference step called inside of monty._step_learning_modules.
- class MontyBase(sensor_modules, learning_modules, motor_system: MotorSystem, sm_to_agent_dict, sm_to_lm_matrix, lm_to_lm_matrix, lm_to_lm_vote_matrix, min_eval_steps, min_train_steps, num_exploratory_steps, max_total_steps)[source]#
Bases:
Monty- aggregate_sensory_inputs(observation)[source]#
Receive data from dataloader/env, organize on a per sensor module basis.
- check_reached_max_matching_steps(max_steps)[source]#
Check if max_steps was reached and deal with time_out.
- Returns:
True if max_steps was reached, False otherwise.
- get_observations(observations, sensor_module_id)[source]#
Get observations from all agents pertaining to a single sensor module.
- Observations are returned in the format
- {“agent_1”:
- {“sm_1”:
- {“rgba”: data,
“depth”: data “semantic”: data}
} {“sm_2”:
{“rgba”: data, … }
- “agent_2”:
- {“sm_3”:
{“rgba”: data, … }
… “agent_n”:
- {“sm_k”:
{“rgba”: data, … }
}
}
- Returns:
Observations from all agents pertaining to a single sensor module.
- load_state_dict(state_dict)[source]#
Take a state dict as an argument and set state for monty and children.
- pass_features_directly_to_motor_system(observation)[source]#
Pass features directly to motor system without stepping LMs.
- set_experiment_mode(mode)[source]#
Set the experiment mode.
Update state variables based on which method (train or evaluate) is being called at the experiment level.
- step(observation)[source]#
Take a matching, exploratory, or custom user-defined step.
Step taken depends on the value of self.step_type.
- LOGGING_REGISTRY = {'TEST': <class 'tbp.monty.frameworks.loggers.exp_logger.TestLogger'>}#
- property exceeded_min_steps#
- property is_done#
Return bool to tell the experiment if we are done with this episode.
- property is_motor_only_step#
- property min_steps#
- property step_type_count#
- class SensorModuleBase(sensor_module_id)[source]#
Bases:
SensorModule
tbp.monty.frameworks.models.motor_policies#
- class BasePolicy(rng, action_sampler_args: Dict, action_sampler_class: Type[ActionSampler], agent_id: str, switch_frequency, file_name=None, file_names_per_episode=None)[source]#
Bases:
MotorPolicy- dynamic_call(_state: MotorSystemState | None = None) Action[source]#
Return a random action.
The MotorSystemState is ignored.
- Parameters:
_state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None. Unused.
- Returns:
A random action.
- Return type:
(Action)
- get_agent_state(state: MotorSystemState) AgentState[source]#
Get agent state (dict).
Note
Assumes we only have one agent.
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
Agent state.
- Return type:
- get_random_action(action: Action) Action[source]#
Returns random action sampled from allowable actions.
Enables expanding the action space of the base policy with actions that we don’t necessarily want to randomly sample
- is_motor_only_step(state: MotorSystemState)[source]#
Check if the current step is a motor-only step.
TODO: This information is currently stored in motor system state, but should be stored in the policy state instead as it is tracking policy state, not motor system state. This will remove MotorSystemState param.
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
True if the current step is a motor-only step, False otherwise.
- Return type:
- post_action(action: Action, _: MotorSystemState | None = None) None[source]#
This post action hook will automatically be called at the end of __call__.
- TODO: Remove state parameter as it is only used to serialize the state in
state.convert_motor_state() and should be done within the motor system.
- Parameters:
action (Action) – The action to process the hook for.
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- predefined_call() Action[source]#
Use this method when actions are predefined.
- Returns:
The action to take.
- Return type:
(Action)
- class GetGoodView(desired_object_distance: float, good_view_percentage: float, multiple_objects_present: bool, sensor_id: str, target_semantic_id: int, allow_translation: bool = True, max_orientation_attempts: int = 1, **kwargs)[source]#
Bases:
PositioningProcedurePositioning procedure to get a good view of the object before an episode.
Used to position the distant agent so that it finds the initial view of an object at the beginning of an episode with respect to a given sensor (the surface agent is positioned using the TouchObject positioning procedure instead). Also currently used by the distant agent after a “jump” has been initialized by a model-based policy.
First, the agent is moved towards the target object until the object fills a minimum of percentage (given by good_view_percentage) of the sensor’s field of view or the closest point of the object is less than desired_object_distance from the sensor. This makes sure that big and small objects all fill similar amount of space in the sensor’s field of view. Otherwise small objects may be too small to perform saccades or the sensor ends up inside of big objects. This step is performed by default but can be skipped by setting allow_translation=False.
Second, the agent will then be oriented towards the object so that the sensor’s central pixel is on-object. In the case of multi-object experiments, (i.e., when multiple_objects_present=True), there is an additional orientation step performed prior to the translational movement step.
- compute_look_amounts(relative_location: numpy.ndarray, state: MotorSystemState | None = None) Tuple[float, float][source]#
Compute the amount to look down and left given a relative location.
This function computes the amount needed to look down and left in order for the sensor to be aimed at the target. The returned amounts are relative to the agent’s current position and rotation. Looking up and right is done by returning negative amounts.
TODO: Test whether this function works when the agent is facing in the positive z-direction. It may be fine, but there were some adjustments to accommodate the z-axis positive direction pointing opposite the body’s initial orientation (e.g., using negative z in left_amount = -np.degrees(np.arctan2(x_rot, -z_rot))).
- Parameters:
relative_location – the x,y,z coordinates of the target with respect
sensor. (to the) –
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- Returns:
Amount to look down (degrees). left_amount: Amount to look left (degrees).
- Return type:
down_amount
- find_location_to_look_at(sem3d_obs: numpy.ndarray, image_shape: Tuple[int, int], state: MotorSystemState | None = None) numpy.ndarray[source]#
Find the location to look at in the observation.
Takes in a semantic 3D observation and returns an x,y,z location.
The location is on the object and surrounded by pixels that are also on the object. This is done by smoothing the on_object image and then taking the maximum of this smoothed image.
- Parameters:
sem3d_obs (np.ndarray) – The location of each pixel and the semantic ID associated with that location.
image_shape (Tuple[int, int]) – The shape of the camera image.
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- Returns:
- The x,y,z coordinates of the target with respect
to the sensor.
- Return type:
(np.ndarray)
- is_on_target_object(observation: Mapping) bool[source]#
Check if a sensor is on the target object.
- Parameters:
observation (Mapping) – The observation to use for positioning.
- Returns:
Whether the sensor is on the target object.
- Return type:
- move_close_enough(observation: Mapping) Action | None[source]#
Move closer to the object until we are close enough.
- Parameters:
observation (Mapping) – The observation to use for positioning.
- Returns:
- The next action to take, or None if we are already close
enough to the object.
- Return type:
(Action | None)
- Raises:
ValueError – If the object is not visible.
- orient_to_object(observation: Mapping, state: MotorSystemState | None = None) List[Action][source]#
Rotate sensors so that they are centered on the object using the view finder.
The view finder needs to be in the same position as the sensor patch and the object needs to be somewhere in the view finders view.
- Parameters:
observation (Mapping) – The observation to use for positioning.
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- Returns:
- A list of actions of length two composed of actions needed
to get us onto the target object.
- Return type:
(List[Action])
- positioning_call(observation: Mapping, state: MotorSystemState | None = None) PositioningProcedureResult[source]#
Return a list of actions to position the agent in the scene.
TODO: When this becomes a PositioningProcedure it can be a __call__ method.
- Parameters:
observation (Optional[Mapping]) – The observation to use for positioning.
state (Optional[MotorSystemState]) – The current state of the motor system.
- Returns:
- Any actions to take, whether the procedure
succeeded, whether the procedure terminated, and whether the procedure truncated.
- Return type:
- sensor_rotation_relative_to_world(state: MotorSystemState) Any[source]#
Derives the positioning sensor’s rotation relative to the world.
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
The positioning sensor’s rotation relative to the world.
- Return type:
(Any)
- class InformedPolicy(min_perc_on_obj, good_view_percentage, desired_object_distance, use_goal_state_driven_actions=False, **kwargs)[source]#
Bases:
BasePolicy,JumpToGoalStateMixinPolicy that takes observation as input.
Extension of BasePolicy that allows for taking the observation into account for action selection. Currently it uses the percentage of the observation that is on the object to reverse the last action if it is below min_perc_on_obj.
Additionally, this policy discouraces taking the reverse of the previous action if we are still on the object.
- guiding_sensors#
List of sensors that are used to calculate the percentage on object. When using multiple sensors or a visualization sensor we may want to ignore some when determining whether we need to move back.
- min_perc_on_obj#
How much percent of the observation needs to be on the object to sample a new action. Otherwise the previous action is reversed to get back on the object. TODO: Not used anywhere?
- compute_look_amounts(relative_location: numpy.ndarray, sensor_id: str, state: MotorSystemState) Tuple[float, float][source]#
Compute the amount to look down and left given a relative location.
This function computes the amount needed to look down and left in order for the sensor to be aimed at the target. The returned amounts are relative to the agent’s current position and rotation.
TODO: Test whether this function works when the agent is facing in the positive z-direction. It may be fine, but there were some adjustments to accommodate the z-axis positive direction pointing opposite the body’s initial orientation (e.g., using negative z in left_amount = -np.degrees(np.arctan2(x_rot, -z_rot))).
- Parameters:
relative_location – the x,y,z coordinates of the target with respect
sensor. (to the) –
sensor_id – the ID of the sensor used to produce the relative location.
state (MotorSystemState) – The current state of the motor system.
- Returns:
Amount to look down (degrees). left_amount: Amount to look left (degrees).
- Return type:
down_amount
- dynamic_call(state: MotorSystemState | None = None) Action[source]#
Return the next action to take.
This requires self.processed_observations to be updated at every step in the Monty class. self.processed_observations contains the features extracted by the sensor module for the guiding sensor (patch).
- Parameters:
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- Returns:
The action to take.
- Return type:
(Action)
- find_location_to_look_at(sem3d_obs: numpy.ndarray, image_shape: Tuple[int, int], target_semantic_id: int, multiple_objects_present: bool, sensor_id: str, state: MotorSystemState) numpy.ndarray[source]#
Takes in a semantic 3D observation and returns an x,y,z location.
The location is on the object and surrounded by pixels that are also on the object. This is done by smoothing the on_object image and then taking the maximum of this smoothed image.
- Parameters:
sem3d_obs – the location of each pixel and the semantic ID associated with that location
image_shape – the shape of the camera image
target_semantic_id – the semantic ID of the target object we’d like to saccade on to
multiple_objects_present – whether there are multiple objects present in the scene.
sensor_id – the ID of the sensor to use for the search. Used for computing the relative location of the new target.
state (MotorSystemState) – The current state of the motor system.
- Returns:
the x,y,z coordinates of the target with respect to the sensor.
- Return type:
relative_location
- fixme_undo_last_action()[source]#
Returns an action that undoes last action for supported actions.
Previous InformedPolicy.dynamic_call() implementation when not on object:
action, amount = (last_action, -last_amount)
This implementation duplicates the functionality and the implicit assumption in the code and configurations that InformedPolicy is working with one of the following actions: - LookUp - LookDown - TurnLeft - TurnRight
Additionally, this implementation adds support for: - MoveForward - MoveTangentially
Additional support for the above two actions is due to -last_amount working for these actions as well. This maintains the same code functionality during this refactoring.
For other actions, raise ValueError explicitly.
- Raises:
TypeError – If the last action is not supported
TODO These instance checks are undesirable and should be removed in the future. I am using these for now to express the implicit assumptions in the code. An Action.undo of some sort would be a better solution, however it is not yet clear to me what to do for actions that do not support undo.
- get_depth_at_center(raw_observation, view_sensor_id, initial_pose=True)[source]#
Determine the depth of the central pixel.
Method primarily used by surface-agent, but also by distant agent after performing a hypothesis-testing jump; determines the depth of the central pixel, to inform whether the object is visible at all
- initial_poseWhether we are checking depth-at center as part of the first
start of an experiment; if using get_depth_at_center to check for the observation after e.g. a hypothesis-testing jump, then we don’t want to throw an error if the object is nowhere to be seen; instead, we simply move back
- Returns:
Depth at the center of the view sensor.
- get_sensors_perc_on_obj(observation: Mapping) float[source]#
Calculate how much percent of the sensor is on the object.
Get the average percentage of pixels on the object for all sensors in the list of guiding sensors.
- Returns:
Percentage of pixels on the object.
- Return type:
perc_on_obj
- is_on_target_object(observation: Mapping, sensor_id: str, target_semantic_id: int, multiple_objects_present: bool) bool[source]#
Check if a sensor is on the target object.
- Parameters:
- Returns:
Whether the sensor is on the target object.
- Return type:
- move_close_enough(raw_observation: Mapping, view_sensor_id: str, target_semantic_id: int, multiple_objects_present: bool) Tuple[Action | None, bool][source]#
At beginning of episode move close enough to the object.
Used the raw observations returned from the dataloader and not the extracted features from the sensor module.
- Parameters:
raw_observation – The raw observations from the dataloader
view_sensor_id – The ID of the view sensor
target_semantic_id – The semantic ID of the primary target object in the scene.
multiple_objects_present – Whether there are multiple objects present in the scene. If so, we do additional checks to make sure we don’t get too close to these when moving forward
- Returns:
- The next action to take and whether the
episode is done.
- Return type:
- Raises:
ValueError – If the object is not visible
- orient_to_object(raw_observation: Mapping, sensor_id: str, target_semantic_id: int, multiple_objects_present: bool, state: MotorSystemState) List[Action][source]#
Rotate sensors so that they are centered on the object using a view finder.
The view finder needs to be in the same position as the sensor patch and the object needs to be somewhere in the view finders view.
- Parameters:
raw_observation – raw observations of the view finder
sensor_id – view finder id (str)
target_semantic_id – the integer corresponding to the semantic ID of the target object that we will try to fixate on
multiple_objects_present – whether there are multiple objects present in the scene.
state (MotorSystemState) – The current state of the motor system.
- Returns:
A (possibly empty) list of actions and a bool that indicates whether we are already on the target object. If we are not on the target object, the list of actions is of length two and is composed of actions needed to get us onto the target object.
- post_action(action: Action, state: MotorSystemState | None = None) None[source]#
This post action hook will automatically be called at the end of __call__.
- TODO: Remove state parameter as it is only used to serialize the state in
state.convert_motor_state() and should be done within the motor system.
- Parameters:
action (Action) – The action to process the hook for.
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- class JumpToGoalStateMixin[source]#
Bases:
objectConvert driving goal state to an action in Habitat-compatible coordinates.
Motor policy that enables us to take in a driving goal state for the motor agent, and specify the action in Habitat-compatible coordinates that must be taken to move there.
- derive_habitat_goal_state()[source]#
Derive the Habitat-compatible goal state.
Take the current driving goal state (in CMP format), and derive the corresponding Habitat compatible goal-state to pass through the Embodied Dataloader.
- Returns:
Target location. target_quat: Target quaternion.
- Return type:
target_loc
- class MotorPolicy[source]#
Bases:
ABCThe abstract scaffold for motor policies.
- abstract dynamic_call(state: MotorSystemState | None = None) Action[source]#
Use this method when actions are not predefined.
- Parameters:
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- Returns:
The action to take.
- Return type:
(Action)
- abstract post_action(action: Action, state: MotorSystemState | None = None) None[source]#
This post action hook will automatically be called at the end of __call__.
- TODO: Remove state parameter as it is only used to serialize the state in
state.convert_motor_state() and should be done within the motor system.
- Parameters:
action (Action) – The action to process the hook for.
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- abstract predefined_call() Action[source]#
Use this method when actions are predefined.
- Returns:
The action to take.
- Return type:
(Action)
- class NaiveScanPolicy(fixed_amount, **kwargs)[source]#
Bases:
InformedPolicyPolicy that just moves left and right along the object.
- check_cycle_action()[source]#
Makes sure we move in a spiral.
This method switches the current action if steps_per_action was reached. Additionally it increments steps_per_action after the second and forth action to make sure paths don’t overlap.
_ _ _ _
_ _ ||_ | ||_ _ _| |
corresponds to 1x left, 1x up, 2x right, 2x down, 3x left, 3x up, 4x right, 4x down, …
- dynamic_call(_state: MotorSystemState | None = None) Action[source]#
Return the next action in the spiral being executed.
The MotorSystemState is ignored.
- Parameters:
_state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None. Unused.
- Returns:
The action to take.
- Return type:
(Action)
- Raises:
StopIteration – If the spiral has completed.
- class PositioningProcedure(rng, action_sampler_args: Dict, action_sampler_class: Type[ActionSampler], agent_id: str, switch_frequency, file_name=None, file_names_per_episode=None)[source]#
Bases:
BasePolicyPositioning procedure to position the agent in the scene.
- TODO: Remove from MotorPolicy hierarchy and refactor to standalone
PositioningProcedure hierarchy when they get separated.
The positioning_call method should be repeatedly called until the procedure result indicates that the procedure has terminated or truncated.
- abstract positioning_call(observation: Mapping, state: MotorSystemState | None = None) PositioningProcedureResult[source]#
Return a list of actions to position the agent in the scene.
TODO: When this becomes a PositioningProcedure it can be a __call__ method.
- Parameters:
observation (Optional[Mapping]) – The observation to use for positioning.
state (Optional[MotorSystemState]) – The current state of the motor system.
- Returns:
- Any actions to take, whether the procedure
succeeded, whether the procedure terminated, and whether the procedure truncated.
- Return type:
- class PositioningProcedureResult(actions: ~typing.List[~tbp.monty.frameworks.actions.actions.Action] = <factory>, success: bool = False, terminated: bool = False, truncated: bool = False)[source]#
Bases:
objectResult of a positioning procedure.
For more on the terminated/truncated terminology, see https://farama.org/Gymnasium-Terminated-Truncated-Step-API.
- class SurfacePolicy(alpha, min_perc_on_obj=0.25, good_view_percentage=0.5, **kwargs)[source]#
Bases:
InformedPolicyPolicy class for a surface-agent.
i.e. an agent that moves to and follows the surface of an object. Includes functions for moving along an object based on its surface normal.
- dynamic_call(state: MotorSystemState | None = None) Action[source]#
Return the next action to take.
This requires self.processed_observations to be updated at every step in the Monty class. self.processed_observations contains the features extracted by the sensor module for the guiding sensor (patch).
- Parameters:
state (Optional[MotorSystemState]) – The current state of the motor system. Defaults to None.
- Returns:
The action to take.
- Return type:
(Action)
- get_inverse_agent_rot(state: MotorSystemState)[source]#
Get the inverse rotation of the agent’s current orientation.
Used to transform poses of e.g. point normals or principle curvature from global coordinates into the coordinate frame of the agent.
To intuit why we apply the inverse, imagine an e.g. point normal with the same pose as the agent; in the agent’s reference frame, this should have the identity pose, which will be aquired by transforming the original pose by the inverse
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
Inverse quaternion rotation.
- get_next_action(state: MotorSystemState)[source]#
Retrieve next action from a cycle of four actions.
First move forward to touch the object at the right distance Then orient toward the normal along direction 1 Then orient toward the normal along direction 2 Then move tangentially along the object surface Then start over
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
Next action in the cycle.
- horizontal_distances(rotation_degrees: float) Tuple[float, float][source]#
Compute the horizontal and forward distances to move to.
Compensate for a given rotation of a certain angle.
- Parameters:
rotation_degrees – The angle to rotate by
- Returns:
The left distance to move move_forward_distance: The forward distance to move
- Return type:
move_left_distance
- orienting_angle_from_normal(orienting: str, state: MotorSystemState) float[source]#
Compute turn angle to face the object.
Based on the point normal, compute the angle that the agent needs to turn in order to be oriented directly toward the object
- Parameters:
orienting (str) – “horizontal” or “vertical”
state (MotorSystemState) – The current state of the motor system.
- Returns:
degrees that the agent needs to turn
- tangential_direction(state: MotorSystemState) Tuple[float, float, float][source]#
Set the direction of the action to be a direction 0 - 2pi.
start at 0 (go up) in the reference frame of the agent; i.e. based on
the standard initialization of an agent, this will be up from the floor. To implement this convention, the theta is offset by 90 degrees when finding our x and y translations, i.e. such that theta of 0 results in moving up by 1 (y), and right by 0 (x), rather than vice-versa - random action -pi - +pi is given by (rand() - 0.5) * 2pi - These are combined and weighted by the alpha parameter
- Parameters:
state – The current state of the motor system.
- Returns:
direction of the action
- Return type:
VectorXYZ
- touch_object(raw_observation, view_sensor_id: str, state: MotorSystemState) Action[source]#
The surface agent’s policy for moving onto an object for sensing it.
Like the distant agent’s get_good_view, this is called at the beginning of every episode, and after a “jump” has been initialized by a model-based policy. In addition, it can be called when the surface agent cannot sense the object, e.g. because it has fallen off its surface.
Currently uses the raw observations returned from the viewfinder via the dataloader, and not the extracted features from the sensor module. TODO M refactor this so that all sensory processing is done in the sensor module.
If we aren’t on the object, try first systematically orienting left around a point, then orienting down, and finally random orientations along the surface of a fixed sphere.
- Parameters:
raw_observation – The raw observation from the simulator.
view_sensor_id – The ID of the viewfinder sensor.
state – The current state of the motor system.
- Returns:
Action to take.
- vertical_distances(rotation_degrees: float) Tuple[float, float][source]#
Compute the down and forward distances to move to.
Compensate for a given rotation of a certain angle.
- Parameters:
rotation_degrees – The angle to rotate by
- Returns:
The down distance to move move_forward_distance: The forward distance to move
- Return type:
move_down_distance
- class SurfacePolicyCurvatureInformed(alpha, pc_alpha, max_pc_bias_steps, min_general_steps, min_heading_steps, **kwargs)[source]#
Bases:
SurfacePolicyPolicy class for a more intelligent surface-agent.
Includes additional functions for moving along an object based on the direction of principle curvature (PC).
A general summary of the policy is that the agent will start following PC directions as soon as these are well defined. This will initially be the minimal curvature; it will follow these for as long as they are defined, and until reaching a certain number of steps (max_pc_bias_steps), before then changing to follow maximal curvature. This process continues to alternate, as long as the PC directions are well defined.
If PC are not meaningfully defined (which is often the case), then the agent uses standard momentum to take a step in a direction similar to the previous step (weighted by the alpha parameter); it will do this for a minimum number of steps (min_general_steps) before it will consider using PC information again.
If the agent is taking a step that will bring it to a previously visited location according to its estimates, then it will attempt to correct for this and choose another direction; after finding a new heading, the agent performs a minimum number of steps (min_heading_steps) before it will check again for a conflicting heading.
The main other event that can occur is that PCs are defined, but these are predominantly in the z-direction (relative to the agent); in that case, the PC-defined heading is ignored on that step, and a standard, momentum-based step is taken; such z-defined PC directions tend to occur on e.g. the rim of cups and are presumably due to issues with the agents re-orientation steps vs. noisiness of the PCs defined by the surface
TODO update this method to accept more general notions of directions-of-variance (rather than strictly principal curvature), such that it can be applied in more abstract spaces
- attempt_conflict_resolution(vec_copy)[source]#
Try to define direction vector that avoids revisiting previous locations.
- avoid_revisiting_locations(state: MotorSystemState, conflict_divisor=3, max_steps=100)[source]#
Avoid revisiting locations.
Check if the new proposed location direction is already pointing to somewhere we’ve visited before; if not, we can use the initially proposed movement.
If there is a conflict, we select a new heading that avoids this; this is achieved by iteratively searching for a heading that does not conflict with any previously visited locations.
- Parameters:
conflict_divisor – The amount pi is divided by to determine that a current heading will be too close to a previously visited location; this is an initial value that will be dynamically adjusted. Defaults to 3.
max_steps – Maximum iterations of the search to perform to try to find a non-conflicting heading. Defaults to 100.
state (MotorSystemState) – The current state of the motor system.
Note that while the policy might have “unrealistic” access to information about it’s location in the environment, this could easily be replaced by relative locations based on the first sensation
Finally, note that in many situations, revisiting locations can be a good thing (e.g. re-anchoring given noisy path-integration), so we may want to activate /inactivate this as necessary (TODO)
TODO separate out avoid_revisiting_locations as its own mixin so that it can be used more broadly
- check_for_flipped_pc()[source]#
Check for arbitrarily flipped PC direction.
Do a quick check to see if the previous PC heading has been arbitrarily flipped, in which case, flip it back. With any luck, this will allow us to automatically pass avoid_revisiting_locations and thereby continue using PCs where relevant.
- check_for_preference_change()[source]#
Flip the preference for the min or max PC after a certain number of steps.
This way, we can more quickly explore different “parts” of an object, rather than just persistently following e.g. the rim of a cup. By default, there is always an initial bias for the smallest principal curvature.
TODO can eventually combine with a hypothesis-testing policy that encourages exploration of unvisited parts of the object, rather than relying on this simple counter-heuristic
- conflict_check(rotated_locs, ii)[source]#
Check for conflict in the current heading.
Target location needs to be similar and we need to have a similar point normal to discount the current proposed heading; if point normals are significantly different, then we are likely on a different surface, in which case passing by nearby previous points is no longer problematic.
Note that when executing failed hypothesis-testing jumps, we can have multiple instances of the same location in our history; this will result in get_angle_beefed_up returning infinity, i.e. we don’t worry about avoiding our current location
- Returns:
True if there is a conflict, False otherwise.
- determine_pc_for_use()[source]#
Determine the principal curvature to use for our heading.
Use magnitude (ignoring negatives), as well as the current direction preference.
- Returns:
Principal curvature to use.
- pc_moving_average()[source]#
Calculate a moving average of the principal curvature direction.
The moving average should be consistent even on curved surfaces as the directions will be in the reference frame of the agent (which has rotated itself to align with the point normal)
- perform_pc_guided_step(state: MotorSystemState)[source]#
Inform steps to take using defined directions of principal curvature.
Use the defined directions of principal curvature to inform (ideally a series) of steps along the appropriate direction.
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
direction of the action
- Return type:
VectorXYZ
- perform_standard_tang_step(state: MotorSystemState)[source]#
Perform a standard tangential step across the object.
This is in contrast to, for example, being guided by principal curvatures.
Note this is still more “intelligent” than the tangential step of the baseline surface-agent policy, because it also attempts to avoid revisiting old locations
- Parameters:
state (MotorSystemState) – The current state of the motor system.
- Returns:
direction of the action
- Return type:
VectorXYZ
- reset_pc_buffers()[source]#
Reset counters and other variables.
We’ve just left a series of PC-defined trajectories (i.e. entered a region of undefined PCs), or had to select a new heading in order to avoid revisiting old locations. As such, appropriately reset counters and other variables.
Note we do not reset tangential_angle or the vector, such that this information can still be used by e.g. momentum on the next step to keep us going generally forward
- tangential_direction(state: MotorSystemState) Tuple[float, float, float][source]#
Set the direction of action to be a direction 0 - 2pi.
- This controls the move_tangential action
start at 0 (go up in the reference frame of the agent, i.e. based on
where it is facing), with the actual orientation determined via either principal curvature, or a random step weighted by momentum
Tangential movements are the primary means of progressively exploring an object’s surface
- Parameters:
state – The current state of the motor system.
- Returns:
direction of the action
- Return type:
VectorXYZ
- update_action_details()[source]#
Store informaton for later logging.
This stores information that details elements of the policy or observations relevant to policy decisions.
E.g. if model-free policy has been unable to find a path that avoids revisiting old locations, an LM might use this information to inform a particular action (TODO not yet implemented, and NOTE that any modelling should ultimately be located in the learning module(s), not in motor systems)
- update_tangential_reps(vec_form=None, angle_form=None)[source]#
Update the angle and vector representation of a tangential heading.
Angle and vector representations are stored as self.tangential_angle and self.tangential_vec, respectively.
Ensures the two representations are always consistent. Further ensures that, because these movements are tangential, it will be defined in the plane, relative to the agent (i.e. movement along x and y only), and any inadvertant movement along the z-axis relative to the agent will be eliminated. User should supply either the vector form or the (Euler) angle form in radians that will define the new representations.
- enforce_pi_bounds(theta)[source]#
Enforce an orientation to be bounded between - pi and + pi.
- Returns:
Angle in radians.
- get_perc_on_obj(rgba_obs)[source]#
Get the percentage of pixels in the observation that land on the object.
This uses the RGBA image which means it only works with one object in an empty void. With multiple objects this would need to be solved differently.
- Parameters:
rgba_obs – RGBA image observation.
- Returns:
Percentage of pixels on the object.
- Return type:
perc_on_obj
- get_perc_on_obj_semantic(semantic_obs, semantic_id=0)[source]#
Get the percentage of pixels in the observation that land on the target object.
If a semantic ID is provided, then only pixels on the target object are counted; otherwise, pixels on any object are counted.
This uses the semantic image, where each pixel is associated with a semantic ID that is unique for each object, and always >0.
- Parameters:
semantic_obs – Semantic image observation.
semantic_id – Semantic ID of the target object.
- Returns:
Percentage of pixels on the object.
- Return type:
perc_on_obj
- projected_angle_from_vec(vector)[source]#
Determine the rotation about a z-axis (pointing “up”).
Note that because of the convention for moving along the y when theta=0, the typical order of arguments is swapped from calculating the standard atan2 (https://en.wikipedia.org/wiki/Atan2).
- Returns:
Angle in radians.
- projected_vec_from_angle(angle)[source]#
Determine the vector in the plane defined by an orientation around the z-axis.
Takes angle in radians, bound between -pi : pi.
This continues the convention started in the original surface-agent policy that a theta of 0 should correspond to a movement of 1 in the y direction, and 0 in the x direction, rather than vice-versa; this convention is implemented by the np.pi/2 offsets.
- Returns:
Vector in the plane defined by an orientation around the z-axis.
- read_action_file(file: str) List[Action][source]#
Load a file with one action per line.
- Parameters:
file – name of file to load
- Returns:
list of actions
- Return type:
List[Action]
tbp.monty.frameworks.models.motor_system#
- class MotorSystem(policy: MotorPolicy, state: MotorSystemState | None = None)[source]#
Bases:
objectThe basic motor system implementation.
tbp.monty.frameworks.models.motor_system_state#
- class AgentState(*args, **kwargs)[source]#
Bases:
dictThe proprioceptive state of an agent.
TODO: Change into dataclass
- sensors: Dict[str, SensorState]#
The proprioceptive state of the agent’s sensors.
- class MotorSystemState[source]#
-
The state of the motor system.
- TODO: Currently, ProprioceptiveState can be cast to MotorSystemState since
MotorSystemState is a generic dictionary. In the future, make ProprioceptiveState a param on MotorSystemState to more clearly distinguish between the two.
- convert_motor_state()[source]#
Convert the motor state into something that can be pickled/saved to JSON.
i.e. substitute vector and quaternion objects; note e.g. copy.deepcopy does not work.
TODO ?clean this up with a recursive algorithm, or use BufferEncoder in buffer.py
- Returns:
Copy of the motor state.
- Return type:
(dict)
- class ProprioceptiveState[source]#
Bases:
Dict[str,AgentState]The proprioceptive state of the motor system.
TODO: Change into dataclass
tbp.monty.frameworks.models.no_reset_evidence_matching#
- class MontyForNoResetEvidenceGraphMatching(*args, **kwargs)[source]#
Bases:
MontyForEvidenceGraphMatchingMonty class for unsupervised inference without explicit episode resets.
This variant of MontyForEvidenceGraphMatching is designed for unsupervised inference experiments where objects may change dynamically without any reset signal. Unlike standard experiments, this class avoids resetting Monty’s internal state (e.g., hypothesis space, evidence scores) between episodes.
This setup better reflects real-world conditions, where object boundaries are ambiguous and no supervisory signal is available to indicate when a new object appears. Only minimal state — such as step counters and termination flags — is reset to prevent buffers from accumulating across objects. Additionally, Monty is currently forced to switch to Matching state. Evaluation of unsupervised inference is performed over a fixed number of matching steps per object.
Intended for evaluation-only runs using pre-trained models, with Monty remaining in the matching phase throughout.
tbp.monty.frameworks.models.object_model#
- exception GridTooSmallError[source]#
Bases:
ExceptionException raised when grid is too small to fit all observations.
- class GraphObjectModel(object_id)[source]#
Bases:
ObjectModelObject model class that represents object as graphs.
- build_model(locations, features, k_n, graph_delta_thresholds)[source]#
Build graph from locations and features sorted into grids.
- update_model(locations, features, location_rel_model, object_location_rel_body, object_rotation, object_scale=1)[source]#
Add new locations and features into grids and rebuild graph.
- property edge_attr#
- property edge_index#
- property feature_ids_in_graph#
- property feature_mapping#
- property norm#
- property num_nodes#
- property pos#
- property x#
- class GridObjectModel(object_id, max_nodes, max_size, num_voxels_per_dim)[source]#
Bases:
GraphObjectModelModel of an object and all its functions.
This model has the same basic functionality as the NumpyGraph models used in older LM versions. On top of that we now have a grid representation of the object that constraints the model size and resultion. Additionally, this model class implements a lot of functionality that was previously implemented in the graph_utils.py file.
- TODO: General cleanups that require more changes in other code
remove node_ids from input_channels and have as graph attribute
remove .norm as attribute and store as feature instead?
- build_model(locations, features)[source]#
Build graph from locations and features sorted into grids.
- find_nearest_neighbors(search_locations, num_neighbors, return_distance=False)[source]#
Find nearest neighbors in graph for list of search locations.
Note
This is currently using kd tree search. In the future we may consider doing this directly by indexing the grids. However, an initial implementation of this does not seem to be faster than the kd tree search (~5-10x slower). However one must consider that search directly in the grid would remove the cost of building the tree. TODO: Investigate this further.
- Returns:
If return_distance is True, return distances. Otherwise, return indices of nearest neighbors.
tbp.monty.frameworks.models.sensor_modules#
- class DetailedLoggingSM(sensor_module_id, save_raw_obs, pc1_is_pc2_threshold=10, point_normal_method='TLS', weight_curvature=True)[source]#
Bases:
SensorModuleBaseSensor module that keeps track of raw observations for logging.
- extract_and_add_features(features, obs_3d, rgba_feat, depth_feat, center_id, center_row_col, sensor_frame_data, world_camera)[source]#
Extract features specified in self.features from sensor patch.
Returns the features in the patch, and True if the point-normal or principal curvature directions were ill-defined.
- Returns:
The features in the patch. morphological_features: ? invalid_signals: True if the point-normal or principal curvature directions
were ill-defined.
- Return type:
features
- observations_to_comunication_protocol(data, on_object_only=True)[source]#
Turn raw observations into instance of State class following CMP.
- Parameters:
data – Raw observations.
on_object_only –
If False, do the following: - If the center of the image is not on the object, but some other part
of the object is in the image, continue with feature extraction
- Get the point normal for the whole image, not just the parts of the
image that include an object.
- Returns:
Features and morphological features.
- Return type:
- class FeatureChangeSM(sensor_module_id, features, delta_thresholds, surf_agent_sm=False, save_raw_obs=False, noise_params=None)[source]#
Bases:
HabitatDistantPatchSM,NoiseMixinSensor Module that turns Habitat camera obs into features at locations.
Takes in camera rgba and depth input and calculates locations from this. It also extracts features which are currently: on_object, rgba, point_normal, curvature.
- class HabitatDistantPatchSM(sensor_module_id, features, save_raw_obs=False, pc1_is_pc2_threshold=10, noise_params=None, process_all_obs=False)[source]#
Bases:
DetailedLoggingSM,NoiseMixinSensor Module that turns Habitat camera obs into features at locations.
Takes in camera rgba and depth input and calculates locations from this. It also extracts features which are currently: on_object, rgba, point_normal, curvature.
- class HabitatSurfacePatchSM(sensor_module_id, features, save_raw_obs=False, noise_params=None)[source]#
Bases:
HabitatDistantPatchSMHabitatDistantPatchSM that continues feature extraction when patch not on object.
Identical to HabitatDistantPatchSM except that feature extraction continues even if the center of the sensor patch is not on the object. TODO: remove and replace with surf_agent_sm=True.
- class NoiseMixin(noise_params, **kwargs)[source]#
Bases:
object- add_noise_to_sensor_data(sensor_data)[source]#
Add noise to features specified in noise_params.
- Noise params should have structure {“features”:
{“feature_keys”: noise_amount, …},
“locations”: noise_amount}
noise_amount specifies the standard deviation of the gaussian noise sampled for real valued features. For boolian features it specifies the probability that the boolean flips. If we are dealing with normed vectors (point_normal or curvature_directions) the noise is applied by rotating the vector given a sampled rotation. Otherwise noise is just added onto the perceived feature value.
- Parameters:
sensor_data – Sensor data to add noise to.
- Returns:
Sensor data with noise added.
tbp.monty.frameworks.models.states#
- class GoalState(location: numpy.ndarray | None, morphological_features: Dict[str, Any] | None, non_morphological_features: Dict[str, Any] | None, confidence: float, use_state: bool, sender_id: str, sender_type: str, goal_tolerances: Dict[str, Any] | None, info: Dict[str, Any] | None = None)[source]#
Bases:
StateSpecialization of
Statefor goal states with null (None) values allowed.Specialized form of state that still adheres to the cortical messaging protocol, but can have null (None) values associated with the location and morphological features.
Used by goal-state generators (GSGs) to communicate goal states to other GSGs, and to motor actuators.
The state variables generally have the same meaning as for the base
Stateclass, and they represent the target values for the receiving system. Thus if a goal-state specifies a particular object ID (non-morphological feature) in a particular pose (location and morphological features), then the receiving system should attempt to achieve that state.Note however that for the goal-state, the confidence corresponds to the conviction with which a GSG believes that the current goal-state should be acted upon. Float bound in [0.0, 1.0].
- class State(location, morphological_features, non_morphological_features, confidence, use_state, sender_id, sender_type)[source]#
Bases:
objectState class used as message packages passed in Monty using CMP.
The cortical messaging protocol (CMP) is used to pass messages between Monty components and makes sure we can easily set up arbitrary configurations of them. This class makes it easier to define the CMP in one place and defines the content and structure of messages passed between Monty components. It also contains some helper funtions to access and modify the message content.
States are represented in this format but can be interpreted by the receiver in different ways:
Observed states: states output py sensor modules Hypothesized states: states output by learning modules Goal states: motor output of learning modules
- location#
3D vector representing the location of the state
- morphological_features#
dictionary of morphological features. Should include pose_vectors of shape (3,3) and pose_fully_defined (bool).
- non_morphological_features#
dictionary of non-morphological features.
- confidence#
confidence in the state. In range [0,1].
- use_state#
boolean indicating whether the state should be used or not.
- sender_id#
string identifying the sender of the state.
- sender_type#
string identifying the type of sender. Can be “SM” or “LM”.
- get_curvature_directions()[source]#
Return the curvature direction vectors.
- Raises:
ValueError – If self.sender_type is not SM
- get_nth_pose_vector(pose_vector_index)[source]#
Return the nth pose vector.
When self.sender_type == “SM”, the first pose vector is the point normal and the second and third are the curvature directions. When self.sender_type == “LM”, the pose vectors correspond to the rotation of the object relative to the model learned of it.
- get_on_object()[source]#
Return whether we think we are on the object or not.
This is currently used in the policy to stay on the object.
- get_point_normal()[source]#
Return the point normal vector.
- Raises:
ValueError – If self.sender_type is not SM