tbp.monty.frameworks.models.salience

tbp.monty.frameworks.models.salience.motor_policy

exception GoalCollocatedWithSensor

Bases: RuntimeError

Raised when a goal is collocated with a sensor.

class LookAtGoal(agent_id: AgentID, sensor_id: SensorID)

Bases: MotorPolicy

A policy that looks at a target.

This class assumes a system similar to a 2-DOF gimbal in which the “outer” part can yaw left/right about the y-axis and the “inner” part can pitch up/down about the x-axis. This setup is typical of our distant agent in which the agent turns left/right and sensor mounted to it looks up/down.

Note that this code only uses TurnLeft and LookUp. Turning right or looking down are performed using negative degrees with TurnLeft and LookUp, respectively.

__init__(agent_id: AgentID, sensor_id: SensorID)

Initialize the look at policy.

Parameters:

• agent_id (NewType()(AgentID, str)) – The agent ID

• sensor_id (NewType()(SensorID, str)) – The sensor ID

• suppress_runtime_errors – Whether to suppress runtime errors. Runtime errors can be raised when goal is None or invalid. When in an experimental mode, we want to raise runtime errors by default. When in a production mode, we want to suppress runtime errors by default. Currently, we run a lot of experiments, so the current default is to raise runtime errors.

load_state_dict(state_dict: dict[str, Any]) → None

Take a state dict as an argument and set state for policy.

pre_episode(motor_system: MotorSystem) → None

Pre episode hook.

Parameters:

motor_system (MotorSystem) – The motor system.

Return type:

None

state_dict() → dict[str, Any]

Return a serializable dict with everything needed to save/load policy.

tbp.monty.frameworks.models.salience.on_object_observation

class OnObjectObservation(center_location: np.ndarray | None, locations: np.ndarray, salience: np.ndarray) → None

Bases: object

__init__(center_location: np.ndarray | None, locations: np.ndarray, salience: np.ndarray) → None

center_location: np.ndarray | None

locations: np.ndarray

salience: np.ndarray

on_object_observation(observation: SensorObservation, salience_map: numpy.ndarray) → OnObjectObservation

Convert all raw observation data into image format.

This function reformats the arrays in a raw observations dictionary so that they’re all indexable by image row and column indices. It also splits the semantic_3d array into 3D locations and an on-object/surface indicator array.

Parameters:

• observation (SensorObservation) – A sensor observation.

• salience_map (ndarray) – A salience map.

Return type:

OnObjectObservation

Returns:

The grid/matrix formatted (unraveled) on-object salience and location data, along with the location corresponding to the central pixel.

tbp.monty.frameworks.models.salience.return_inhibitor

class DecayField(kernel_factory: DecayKernelFactory | None = None)

Bases: object

Manages a collection of decay kernels.

__init__(kernel_factory: DecayKernelFactory | None = None)

add(location: numpy.ndarray) → None

Add a kernel to the field.

Return type:

None

compute_weights(points: numpy.ndarray) → numpy.ndarray

Return type:

ndarray

reset() → None

Return type:

None

step() → None

Step each kernel to increment its counter, and keep only non-expired ones.

Return type:

None

class DecayKernel(location: np.ndarray, tau_t: float = 10.0, tau_s: float = 0.01, spatial_cutoff: float | None = 0.02, w_t_min: float = 0.1)

Bases: object

Decay kernel represents a previously visited location.

Returns the product of time- and space-dependent exponentials.

__init__(location: np.ndarray, tau_t: float = 10.0, tau_s: float = 0.01, spatial_cutoff: float | None = 0.02, w_t_min: float = 0.1)

step() → bool

Increment the step counter, and check if the kernel is expired.

Return type:

bool

Returns:

True if the kernel is expired, False otherwise.

w_s(points: numpy.ndarray) → numpy.ndarray

Compute the distance-dependent weight.

The weight is computed as exp(-z / lam), where z is the distance between the kernel’s center and the given point(s), and lam is equal to tau_s / log(2).

Parameters:

points (ndarray) – An (num_points, 3) array of points.

Return type:

ndarray

Returns:

The weight, bounded to [0, 1]. Has shape (num_points,).

w_t() → float

Compute the time-dependent weight at the current step.

The weight is computed as exp(-t / lam), where t is the number of steps since the kernel was created, and lam is equal to tau_t / log(2).

Return type:

float

Returns:

The weight, bounded to [0, 1].

class DecayKernelFactory(tau_t: float = 10.0, tau_s: float = 0.01, spatial_cutoff: float | None = 0.02, w_t_min: float = 0.1)

Bases: object

__init__(tau_t: float = 10.0, tau_s: float = 0.01, spatial_cutoff: float | None = 0.02, w_t_min: float = 0.1)

class ReturnInhibitor(decay_field: DecayField | None = None)

Bases: object

__init__(decay_field: DecayField | None = None)

reset() → None

Return type:

None

tbp.monty.frameworks.models.salience.sensor_module

class SalienceSM(sensor_module_id: str, save_raw_obs: bool = False, salience_strategy: SalienceStrategy | None = None, return_inhibitor: ReturnInhibitor | None = None, snapshot_telemetry: SnapshotTelemetry | None = None) → None

Bases: SensorModule

__init__(sensor_module_id: str, save_raw_obs: bool = False, salience_strategy: SalienceStrategy | None = None, return_inhibitor: ReturnInhibitor | None = None, snapshot_telemetry: SnapshotTelemetry | None = None) → None

pre_episode() → None

This method is called before each episode.

Return type:

None

propose_goals() → list[Goal]

Return the goals proposed by this Sensor Module.

state_dict()

Return a serializable dict with this sensor module’s state.

Includes everything needed to save/load this sensor module.

step(ctx: RuntimeContext, observation: SensorObservation, motor_only_step: bool = False) → None

Generate goal for the current step.

If motor_only_step is True, this method will return without using the salience strategy, stepping the return inhibitor, or modifying self._goals in any way.

Parameters:

• ctx (RuntimeContext) – The runtime context.

• observation (SensorObservation) – Sensor observation.

• motor_only_step (bool) – Whether the current step is a motor-only step.

Return type:

None

update_state(agent: AgentState)

Update information about the sensor’s location and rotation.

property sensor_module_id: str

tbp.monty.frameworks.models.salience.strategies

class SalienceStrategy(*args, **kwargs)

Bases: Protocol

__init__(*args, **kwargs)

class UniformSalienceStrategy(*args, **kwargs)

Bases: SalienceStrategy