tbp.monty.frameworks.models.evidence_matching#

tbp.monty.frameworks.models.evidence_matching.graph_memory#

class EvidenceGraphMemory(max_nodes_per_graph, max_graph_size, num_model_voxels_per_dim, *args, **kwargs)[source]#

Bases: GraphMemory

Custom GraphMemory that stores GridObjectModel instead of GraphObjectModel.

get_initial_hypotheses()[source]#

get_rotation_features_at_all_nodes(graph_id, input_channel)[source]#

Get rotation features from all N nodes. shape=(N, 3, 3).

Returns:: The rotation features from all N nodes. shape=(N, 3, 3).

tbp.monty.frameworks.models.evidence_matching.hypotheses#

class ChannelHypotheses(evidence: numpy.ndarray, locations: numpy.ndarray, poses: numpy.ndarray, input_channel: str)[source]#

Bases: Hypotheses

A set of hypotheses for a single input channel.

input_channel: str#

class Hypotheses(evidence: numpy.ndarray, locations: numpy.ndarray, poses: numpy.ndarray)[source]#

Bases: object

Set of hypotheses consisting of evidence, locations, and poses.

The three arrays are expected to have the same shape. Each index corresponds to a hypothesis.

evidence: numpy.ndarray#

locations: numpy.ndarray#

poses: numpy.ndarray#

tbp.monty.frameworks.models.evidence_matching.hypotheses_displacer#

class DefaultHypothesesDisplacer(feature_weights: dict, graph_memory: EvidenceGraphMemory, max_match_distance: float, tolerances: dict, use_features_for_matching: dict[str, bool], feature_evidence_calculator: Type[FeatureEvidenceCalculator] = <class 'tbp.monty.frameworks.models.evidence_matching.feature_evidence.calculator.DefaultFeatureEvidenceCalculator'>, feature_evidence_increment: int = 1, max_nneighbors: int = 3, past_weight: float = 1, present_weight: float = 1)[source]#

Bases: object

displace_hypotheses_and_compute_evidence(channel_displacement: numpy.ndarray, channel_features: dict, evidence_update_threshold: float, graph_id: str, possible_hypotheses: ChannelHypotheses, total_hypotheses_count: int) → ChannelHypotheses[source]#

class HypothesesDisplacer(*args, **kwargs)[source]#

Bases: Protocol

displace_hypotheses_and_compute_evidence(channel_displacement: numpy.ndarray, channel_features: dict, evidence_update_threshold: float, graph_id: str, possible_hypotheses: ChannelHypotheses, total_hypotheses_count: int) → ChannelHypotheses[source]#

Updates evidence by comparing features after applying sensed displacement.

This function applies the sensor displacement to the existing hypothesis and uses the result as search locations for comparing the sensed features. This comparison is used to update the evidence scores of the existing hypotheses. The hypotheses locations are updated to the new locations (i.e., after displacement)

Parameters:

channel_displacement (np.ndarray) – Channel-specific sensor displacement.
channel_features (dict) – Channel-specific input features.
evidence_update_threshold (float) – Evidence update threshold.
graph_id (str) – The ID of the current graph
possible_hypotheses (ChannelHypotheses) – Channel-specific possible hypotheses.
total_hypotheses_count (int) – Total number of hypotheses in the graph.

Returns:

Displaced hypotheses with computed evidence.

Return type:

ChannelHypotheses

tbp.monty.frameworks.models.evidence_matching.hypotheses_updater#

class DefaultHypothesesUpdater(feature_weights: dict, graph_memory: EvidenceGraphMemory, max_match_distance: float, tolerances: dict, feature_evidence_calculator: type[FeatureEvidenceCalculator] = <class 'tbp.monty.frameworks.models.evidence_matching.feature_evidence.calculator.DefaultFeatureEvidenceCalculator'>, feature_evidence_increment: int = 1, features_for_matching_selector: type[FeaturesForMatchingSelector] = <class 'tbp.monty.frameworks.models.evidence_matching.features_for_matching.selector.DefaultFeaturesForMatchingSelector'>, initial_possible_poses: Literal[('uniform', 'informed')] | list[Rotation] = 'informed', max_nneighbors: int = 3, past_weight: float = 1, present_weight: float = 1, umbilical_num_poses: int = 8)[source]#

Bases: object

update_hypotheses(hypotheses: Hypotheses, features: dict, displacements: dict | None, graph_id: str, mapper: ChannelMapper, evidence_update_threshold: float) → list[ChannelHypotheses][source]#

Update hypotheses based on sensor displacement and sensed features.

Updates existing hypothesis space or initializes a new hypothesis space if one does not exist (i.e., at the beginning of the episode). Updating the hypothesis space includes displacing the hypotheses possible locations, as well as updating their evidence scores. This process is repeated for each input channel in the graph.

Parameters:

hypotheses (Hypotheses) – Hypotheses for all input channels in the graph_id
features (dict) – Input features
displacements (dict or None) – Given displacements
graph_id (str) – Identifier of the graph being updated
mapper (ChannelMapper) – Mapper for the graph_id to extract data from evidence, locations, and poses based on the input channel
evidence_update_threshold (float) – Evidence update threshold.

Returns:

The list of hypotheses updates to be applied to: each input channel.

Return type:

list[ChannelHypotheses]

class HypothesesUpdater(*args, **kwargs)[source]#

Bases: Protocol

Update hypotheses based on sensor displacement and sensed features.

Parameters:

hypotheses (Hypotheses) – Hypotheses for all input channels for the graph_id
features (dict) – Input features
displacements (dict or None) – Given displacements
graph_id (str) – Identifier of the graph being updated
mapper (ChannelMapper) – Mapper for the graph_id to extract data from evidence, locations, and poses based on the input channel
evidence_update_threshold (float) – Evidence update threshold

Returns:

The list of channel hypotheses updates to be: applied.

Return type:

list[ChannelHypotheses]

all_usable_input_channels(features: dict, all_input_channels: list[str]) → list[str][source]#

Determine all usable input channels.

Parameters:

features (dict) – Input features.
all_input_channels (list[str]) – All input channels that are stored in the graph.

Returns:

All input channels that are usable for matching.

Return type:

list[str]

tbp.monty.frameworks.models.evidence_matching.learning_module#

class EvidenceGraphLM(max_match_distance, tolerances: dict, feature_weights: dict, feature_evidence_increment=1, evidence_threshold_config: float | str = 'all', vote_evidence_threshold=0.8, past_weight=1, present_weight=1, vote_weight=1, object_evidence_threshold=1, x_percent_threshold=10, path_similarity_threshold=0.1, pose_similarity_threshold=0.35, required_symmetry_evidence=5, graph_delta_thresholds=None, max_graph_size=0.3, max_nodes_per_graph=2000, num_model_voxels_per_dim=50, use_multithreading=True, gsg_class=<class 'tbp.monty.frameworks.models.goal_state_generation.EvidenceGoalStateGenerator'>, gsg_args=None, hypotheses_updater_class: type[HypothesesUpdater] = <class 'tbp.monty.frameworks.models.evidence_matching.hypotheses_updater.DefaultHypothesesUpdater'>, hypotheses_updater_args: dict | None = None, *args, **kwargs)[source]#

Bases: GraphLM

Learning module that accumulates evidence for objects and poses.

Matching Attributes:

max_match_distance: Maximum distance of a tested and stored location to: be matched.
tolerances: How much can each observed feature deviate from the stored: features to still be considered a match.
feature_weights: How much should each feature be weighted when calculating: the evidence update for hypotheses. Weights are stored in a dictionary with keys corresponding to features (same as keys in tolerances)
feature_evidence_increment: Feature evidence (between 0 and 1) is multiplied: by this value before being added to the overall evidence of a hypothesis. This factor is only multiplied with the feature evidence (not the pose evidence as opposed to the present_weight).
evidence_threshold_config (float | str): How to decide which hypotheses: should be updated. When this parameter is either ‘[int]%’ or ‘x_percent_threshold’, then this parameter is applied to the evidence for the Most Likely Hypothesis (MLH) to determine a minimum evidence threshold in order for other hypotheses to be updated. Any hypotheses falling below the resulting evidence threshold do not get updated. The other options set a fixed threshold that does not take MLH evidence into account. In [int, float, ‘[int]%’, ‘mean’, ‘median’, ‘all’, ‘x_percent_threshold’]. Defaults to ‘all’.
vote_evidence_threshold: Only send votes that have a scaled evidence above: this threshold. Vote evidences are in the range of [-1, 1] so the threshold should not be outside this range.
past_weight: How much should the evidence accumulated so far be weighted: when combined with the evidence from the most recent observation.
present_weight: How much should the current evidence be weighted when added: to the previous evidence. If past_weight and present_weight add up to 1, the evidence is bounded and can’t grow infinitely. NOTE: right now this doesn’t give as good performance as with unbounded evidence since we don’t keep a full history of what we saw. With a more efficient policy and better parameters that may be possible to use though and could help when moving from one object to another and to generally make setting thresholds etc. more intuitive.
vote_weight: Vote evidence (between -1 and 1) in multiplied by this value: when being added to the overall evidence of a hypothesis. If past and current_weight add up to 1, it is use as weight in np.average to keep the evidence in a fixed range.

Terminal Condition Attributes:

object_evidence_threshold: Minimum required evidence for an object to be

recognized. We additionally check that the evidence for this object is significantly higher than for all other objects.

x_percent_threshold: Used in two places:

All objects whose highest evidence is greater than the most likely
objects evidence - x_percent of the most like objects evidence are considered possible matches. That means to only have one possible match, no other object can have more evidence than the candidate match’s evidence - x percent of it.
Within one object, possible poses are considered possible if their
evidence is larger than the most likely pose of this object - x percent of this poses evidence.

# TODO: should we use a separate threshold for within and between objects? If this value is larger, the model is usually more robust to noise and reaches a better performance but also requires a lot more steps to reach a terminal condition, especially if there are many similar object in the data set.

path_similarity_threshold: How similar do paths have to be to be

considered the same in the terminal condition check.

pose_similarity_threshold: difference between two poses to be considered

unique when checking for the terminal condition (in radians).

required_symmetry_evidence: number of steps with unchanged possible poses

to classify an object as symmetric and go into terminal condition.

Model Attributes:

graph_delta_thresholds: Thresholds used to compare nodes in the graphs being: learned, and thereby whether to include a new point or not. By default, we only consider the distance between points, using a threshold of 0.001 (determined in remove_close_points). Can also specify thresholds based on e.g. point-normal angle difference, or principal curvature magnitude difference.
max_graph_size: Maximum size of a graph in meters. Any observations that fall: out of this range will be discarded/used for building a new model. This constraints the size of models that an LM can learn and enforces learning models of sub-components of objects.
max_nodes_per_graph: Maximum number of nodes in a graph. This will be k when: picking the k-winner voxels to add their content into the graph used for matching.
num_model_voxels_per_dim: Number of voxels per dimension in the model grid.: This constraints the spatial resolution that the model can represent. max_graph_size/num_model_voxels_per_dim = how much space is lumped into one voxel. All locations that fall into the same voxel will be averaged and represented as one value. num_model_voxels_per_dim should not be too large since the memory requirements grow cubically with this number.

gsg_class: The type of goal-state-generator to associate with the LM. gsg_args: Dictionary of configuration parameters for the GSG. hypotheses_updater_class: The type of hypotheses updater to associate with the

LM.

hypotheses_updater_args: Dictionary of configuration parameters for the: hypotheses updater.

Debugging Attributes:

use_multithreading: Whether to calculate evidence updates for different: objects in parallel using multithreading. This can be done since the updates to different objects are completely independent of each other. In general it is recommended to use this but it can be useful to turn it off for debugging purposes.

collect_stats_to_save()[source]#

Get all stats that this LM should store in the buffer for logging.

Returns:: The stats dictionary.

get_all_evidences()[source]#: Return evidence for each pose on each graph (pointer).

get_current_mlh()[source]#

Return the current most likely hypothesis of the learning module.

Returns:: graph_id, location, rotation, scale, evidence
Return type:: dict with keys

get_evidence_for_each_graph()[source]#: Return maximum evidence count for a pose on each graph.

get_mlh_for_object(object_id)[source]#

Get mlh for a specific object ID.

Note

When trying to retrieve the MLH for the current most likely object and not any other object, it is better to use self.current_mlh

Returns:: The most likely hypothesis for the object ID.

get_output()[source]#

Return the most likely hypothesis in same format as LM input.

The input to an LM at the moment is a dict of features at a location. The output therefor has the same format to keep the messaging protocol consistent and make it easy to stack multiple LMs on top of each other.

If the evidence for mlh is < object_evidence_threshold, interesting_features == False

get_possible_hypothesis_ids(object_id)[source]#

get_possible_matches()[source]#: Return graph ids with significantly higher evidence than median.

get_possible_poses(as_euler=True)[source]#

Return possible poses for each object (for logging).

Here this list doesn’t get narrowed down. This is not really used for evidence matching since we threshold in other places.

get_top_two_mlh_ids()[source]#

Retrieve the two most likely object IDs for this LM.

Returns:: The two most likely object IDs.

get_top_two_pose_hypotheses_for_graph_id(graph_id)[source]#: Return top two hypotheses for a given graph_id.

get_unique_pose_if_available(object_id)[source]#

Get the most likely pose of an object if narrowed down.

If there is not one unique possible pose or symmetry detected, return None

Returns:: The pose and scale if a unique pose is available, otherwise None.

receive_votes(vote_data)[source]#

Get evidence count votes and use to update own evidence counts.

Weighted by distance to votes and their evidence. TODO: also take into account rotation vote

vote_data contains:: pos_location_votes: shape=(N, 3) pos_rotation_votes: shape=(N, 3, 3) pose_evidences: shape=(N,)

reset()[source]#: Reset evidence count and other variables.

send_out_vote()[source]#

Send out hypotheses and the evidence for them.

Votes are a dict and contain the following:

pose_hypotheses: locations (V, 3) and rotations (V, 3, 3) pose_evidence: Evidence (V) for each location-rotation pair in the

pose hypotheses. Scaled into range [-1, 1] where 1 is the hypothesis with the largest evidence in this LM and -1 the one with the smallest evidence. When thresholded, pose_evidence will be in range [self.vote_evidence_threshold, 1]

sensed_pose_rel_body: sensed location and rotation of the input to this: LM. Rotation is represented by the pose vectors (point normal and curvature directions) for the SMs. For input from LMs it is also represented as 3 unit vectors, these are calculated from the estimated rotation of the most likely object. This pose is used to calculate the displacement between two voting LMs and to translate the votes between their reference frames. Shape=(4,3).

Where V is the number of votes (V=number of hypotheses if not thresholded) If none of the hypotheses of an object are > vote_evidence_threshold, this object will not send out a vote.

Returns:: possible_states: The possible states. sensed_pose_rel_body: The sensed pose relative to the body.
Return type:: None or dict

set_detected_object(terminal_state)[source]#

Set the current graph ID.

If we didn’t recognize the object this will be new_object{n} where n is len(graph_memory) + 1. Otherwise it is the id of the graph that we recognized. If we timed out it is None and we will not update the graph memory.

tbp.monty.frameworks.models.evidence_matching.model#

class MontyForEvidenceGraphMatching(*args, **kwargs)[source]#

Bases: MontyForGraphMatching

Monty model for evidence based graphs.

Customize voting and union of possible matches.

switch_to_exploratory_step()[source]#

Switch to exploratory step.

Also, set mlh evidence high enough to generate output during exploration.

tbp.monty.frameworks.models.evidence_matching.resampling_hypotheses_updater#

class ResamplingHypothesesUpdater(feature_weights: dict, graph_memory: EvidenceGraphMemory, max_match_distance: float, tolerances: dict, feature_evidence_calculator: Type[FeatureEvidenceCalculator] = <class 'tbp.monty.frameworks.models.evidence_matching.feature_evidence.calculator.DefaultFeatureEvidenceCalculator'>, feature_evidence_increment: int = 1, features_for_matching_selector: Type[FeaturesForMatchingSelector] = <class 'tbp.monty.frameworks.models.evidence_matching.features_for_matching.selector.DefaultFeaturesForMatchingSelector'>, hypotheses_count_multiplier: float = 1.0, hypotheses_existing_to_new_ratio: float = 0.0, initial_possible_poses: Literal[('uniform', 'informed')] | list[Rotation] = 'informed', max_nneighbors: int = 3, past_weight: float = 1, present_weight: float = 1, umbilical_num_poses: int = 8)[source]#

Bases: object

Hypotheses updater that resamples hypotheses at every step.

This updater enables updating of the hypothesis space by resampling and rebuilding the hypothesis space at every step. We resample hypotheses from the existing hypothesis space, as well as new hypotheses informed by the sensed pose.

The resampling process is governed by two main parameters:

hypotheses_count_multiplier: scales the total number of hypotheses every step.
hypotheses_existing_to_new_ratio: controls the proportion of existing vs.
informed hypotheses during resampling.

To reproduce the behavior of DefaultHypothesesUpdater sampling a fixed number of hypotheses only at the beginning of the episode, you can set hypotheses_count_multiplier=1.0 and hypotheses_existing_to_new_ratio=0.0.

Update hypotheses based on sensor displacement and sensed features.

Parameters:

hypotheses (Hypotheses) – Hypotheses for all input channels in the graph_id
features (dict) – Input features
displacements (dict or None) – Given displacements
graph_id (str) – Identifier of the graph being updated
mapper (ChannelMapper) – Mapper for the graph_id to extract data from evidence, locations, and poses based on the input channel
evidence_update_threshold (float) – Evidence update threshold.

Returns:

The list of hypotheses updates to be applied to: each input channel.

Return type:

list[ChannelHypotheses]