tbp.monty.simulators.habitat#

tbp.monty.simulators.habitat.actions#

class SetAgentPitch(*args: Any, **kwargs: Any)#

Bases: SceneNodeControl

Custom habitat-sim action used to set the agent absolute pitch rotation.

Note that unless otherwise changed, the sensor maintains identity orientation w.r.t the agent, so in effect, this will also adjust the pitch of the sensor w.r.t the environment.

This difference in behavior is controlled by the body_action boolean set in the decorator.

ActuationSpec amount contains the new absolute pitch rotation in degrees.

TODO add unit test to habitat_sim_test.py

class SetAgentPose(*args: Any, **kwargs: Any)#

Bases: SceneNodeControl

Custom habitat-sim action used to set the agent pose.

This action sets the agent body absolute location (xyz coordinate), and absolute rotation (i.e. relative to the identity rotation, which is defined by the axes of xyz in Habitat) in the environment.

ActuationSpec amount contains location and rotation (latter as a numpy: quaternion)

class SetSensorPitch(*args: Any, **kwargs: Any)#

Bases: SceneNodeControl

Custom habitat-sim action used to set the sensor absolute pitch rotation.

Note this does not update the pitch of the agent (imagine e.g. the “body” associated with the eye remaining in place, but the eye moving).

ActuationSpec amount contains the new absolute pitch rotation in degrees.

class SetSensorPose(*args: Any, **kwargs: Any)#

Bases: SceneNodeControl

Custom habitat-sim action used to set the sensor pose.

This action sets the sensor’s absolute location (xyz coordinate), and rotation relative to the agent.

Note that body_action=False, resulting in a sensor (rather than agent) update.

ActuationSpec amount contains location and rotation (latter as a numpy: quaternion)

class SetSensorRotation(*args: Any, **kwargs: Any)#

Bases: SceneNodeControl

Custom habitat-sim action used to set the sensor rotation.

This action sets the sensor’s absolute rotation relative to the agent.

Note body_action=False, resulting in a sensor (rather than agent) update

ActuationSpec amount contains rotation (as a numpy quaternion)

class SetYaw(*args: Any, **kwargs: Any)#

Bases: SceneNodeControl

Custom habitat-sim action used to set the agent body absolute yaw rotation.

ActuationSpec amount contains the new absolute yaw rotation in degrees.

tbp.monty.simulators.habitat.actuator#

class HabitatActuator[source]#

Bases: Actuator, HabitatActuatorRequirements

Habitat implementation of Actuator.

HabitatActuator is responsible for executing actions in the Habitat simulation.

It is a separate class to encapsulate the actuation logic in one place. This class is expected to be mixed into HabitatSim and expects HabitatActuatorRequirements to be met.

action_name(action: Action) → str[source]#

Returns Monty’s Habitat action naming convention.

The action name is prefixed by the agent ID.

actuate(action: Action, parameterizer: Type[HabitatParameterizer])[source]#

Transition from the Monty to the Habitat sim domain and execute the action.

Parameters:

action – Monty action to execute by the agent specified in the action.
parameterizer – Parameterizer to use to set action parameters within Habitat sim prior to executing the action.

Raises:

ValueError – If the action name is invalid

actuate_look_down(action: LookDown) → None[source]#

actuate_look_up(action: LookUp) → None[source]#

actuate_move_forward(action: MoveForward) → None[source]#

actuate_move_tangentially(action: MoveTangentially) → None[source]#

actuate_orient_horizontal(action: OrientHorizontal) → None[source]#

actuate_orient_vertical(action: OrientVertical) → None[source]#

actuate_set_agent_pitch(action: SetAgentPitch) → None[source]#

actuate_set_agent_pose(action: SetAgentPose) → None[source]#

actuate_set_sensor_pitch(action: SetSensorPitch) → None[source]#

actuate_set_sensor_pose(action: SetSensorPose) → None[source]#

actuate_set_sensor_rotation(action: SetSensorRotation) → None[source]#

actuate_set_yaw(action: SetYaw) → None[source]#

actuate_turn_left(action: TurnLeft) → None[source]#

actuate_turn_right(action: TurnRight) → None[source]#

class HabitatActuatorRequirements[source]#

Bases: ABC

HabitatActuator requires these to be available when mixed in.

abstract get_agent(agent_id: str) → habitat_sim.Agent[source]#

class HabitatParameterizer[source]#

Bases: ABC

abstract static parameterize(params: Any, action: Action) → None[source]#

Copies relevant parameters from action to params.

Habitat does not expose an API for passing parameters to actions. This is a work around for this limitation by artisanally setting specific action parameters directly in Habitat sim.

tbp.monty.simulators.habitat.agents#

class HabitatAgent(agent_id: str, position: Tuple[float, float, float] = (0.0, 1.5, 0.0), rotation: Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), height: float = 0.0)[source]#

Bases: object

Habitat agent wrapper.

Agents are used to define moveable bodies in the environment. Every habitat agent will inherit from this class.

agent_id#: Unique ID of this agent in env. Observations returned by environment will be mapped to this id. {"agent_id": {"sensor": [...]}}. Actions provided by this sensor module will be prefixed by this id. i.e. “agent_id.move_forward”

position#: Module initial position in meters. Default (0, 1.5, 0)

rotation#: Module initial rotation quaternion. Default (1, 0, 0, 0)

height#: Module height in meters. Default 0.0

get_spec()[source]#

Returns a habitat-sim agent configuration.

Returns:: habitat_sim.agent.AgentConfiguration spec create from this sensor module configuration.

initialize(simulator)[source]#

Initialize habitat-sim agent runtime state.

This method must be called to update the agent and sensors runtime instance. This is necessary because some of the configuration attributes requires access to the instanciated node.

Parameters:: simulator – Instantiated HabitatSim instance

process_observations(agent_obs)[source]#

Callback processing raw habitat agent observations to Monty-compatible ones.

Parameters:: agent_obs – Agent raw habitat-sim observations
Returns:: The processed observations grouped by sensor_id
Return type:: dict

class MultiSensorAgent(agent_id: str, sensor_ids: Tuple[str], position: Tuple[float, float, float] = (0.0, 1.5, 0.0), rotation: Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: str = 'distant_agent', resolutions: Tuple[Tuple[int, int]] = ((16, 16),), positions: Tuple[Tuple[float, float, float]] = ((0.0, 0.0, 0.0),), rotations: Tuple[Tuple[float, float, float, float]] = ((1.0, 0.0, 0.0, 0.0),), zooms: Tuple[float] = (1.0,), semantics: Tuple[bool] = (False,))[source]#

Bases: HabitatAgent, ActionSpaceMixin

Minimal version of a HabitatAgent with multiple RGBD sensors mounted.

The RGBD sensors are mounted to the same movable object (like two go-pros mounted to a helmet) with the following pre-defined actions:

“agent_id.move_forward”: Move camera forward using translation_step

“agent_id.turn_left”: Turn camera left rotation_step

“agent_id.turn_right”: Turn camera right rotation_step

“agent_id.look_up”: Turn the camera up rotation_step

“agent_id.look_down”: Turn the camera down rotation_step

“agent_id”.set_yaw” : Set camera agent absolute yaw value

“agent_id”.set_sensor_pitch” : Set camera sensor absolute pitch value

“agent_id”.set_agent_pitch” : Set camera agent absolute pitch value

Each camera will return the following observations:

“sensor_ids[i].rgba”: Color information for every pixel (x, y, 4)

“sensor_ids[i].depth”: Depth information for every pixel (x, y, 1)

“sensor_ids[i].semantic”: Optional object semantic information for every pixel
(x, y, 1)

where i is an integer indexing the list of sensor_ids.

Note

The parameters resolutions, rotations and so on effectively specify both the number of sensors, and the sensor parameters. For N sensors, specify a list of N resolutions, and so on. All lists must be the same length. By default, a list of length one will be provided. Therefore, do not leave an argument blank if you wish to run a simulation with N > 1 sensors.

Note

The parameters translation_step and rotation_step are set to 0 by default. All action amounts should be specified by the MotorSystem.

agent_id#: Actions provided by this camera will be prefixed by this id. Default “camera”

sensor_ids#: List of ids for each sensor. Actions are prefixed with agent id, but observations are prefixed with sensor id.

resolutions#: List of camera resolutions (width, height). Defaut = (16, 16)

positions#: List of camera initial absolute positions in meters, relative to the agent.

rotations#: List of camera rotations (quaternion). Default (1, 0, 0, 0)

zooms#: List of camera zoom multipliers. Use >1 to increase, 0<factor<1 to decrease. Default 1.0

semantics#: List of booleans deciding if each RGBD sensor also gets a semantic sensor with it. Default = False

height#: Height of the mount itself in meters. Default 0.0 (but position of the agent will be 1.5 meters in “height” dimension)

rotation_step#: Rotation step in degrees used by the turn_* and look_* actions. Default 0 degrees

translation_step#: Translation step is meters used by the move_* actions. Default: 0m

action_space_type#: Decides between three action spaces. “distant_agent” actions saccade like a ball-in-socket joint, viewing an object from a distance “surface_agent” actions orient to an object surface and move tangentially along it “absolute_only” actions are movements in absolute world coordinates only

get_spec()[source]#

Returns a habitat-sim agent configuration.

Returns:: habitat_sim.agent.AgentConfiguration spec create from this sensor module configuration.

initialize(simulator)[source]#

Initialize agent runtime state.

This method must be called by HabitatSim to update the agent and sensors runtime instance. This is necessary because some of the configuration attributes requires access to the instanciated scene node.

Parameters:: simulator – Instantiated HabitatSim instance

class SingleSensorAgent(agent_id: str, sensor_id: str, agent_position: Tuple[float, float, float] = (0.0, 1.5, 0.0), sensor_position: Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, resolution: Tuple[int, int] = (16, 16), zoom: float = 1.0, semantic: bool = False, rotation_step: float = 0.0, translation_step: float = 0.0, action_space: Tuple | None = None, action_space_type: str = 'distant_agent')[source]#

Bases: HabitatAgent, ActionSpaceMixin

Minimal version of a HabitatAgent.

This is the special case of MultiSensorAgent when there is at most 1 RGBD and 1 semantic sensor. Thus, the arguments are single values instead of lists.

get_spec()[source]#

Returns a habitat-sim agent configuration.

Returns:: habitat_sim.agent.AgentConfiguration spec create from this sensor module configuration.

initialize(simulator)[source]#

Initialize agent runtime state.

Parameters:: simulator – Instantiated HabitatSim instance

tbp.monty.simulators.habitat.sensors#

class RGBDSensorConfig(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), resolution: ~typing.Tuple[int, int] = (64, 64), zoom: float = 1.0)[source]#

Bases: SensorConfig

RGBD Camera sensor configuration.

Use this class to configure two different Habitat sensors simultaneously, one for RGBA observations and another for depth. Both sensors will use the same pose, resolution and zoom factor. RGB observations are named “rgba”, depth observations are named “depth”.

sensor_id#

Sensor ID unique within the sensor module. If given then observations made by this sensor will be prefixed by this id. i.e. “sensor_id.rgba”

Type:: str

resolution#

Camera resolution (width, height). Default (64, 64)

Type:: Tuple[int, int]

position#

Sensor position relative to HabitatAgent. Default (0, 0, 0)

Type:: Tuple[float, float, float]

rotation#

Sensor rotation quaternion. Default (1, 0, 0, 0)

Type:: Tuple[float, float, float, float]

zoom#

Camera zoom multiplier. Use >1 to increase, 0<factor<1 to decrease. Default 1.0, no zoom

Type:: float

get_specs() → List[habitat_sim.sensor.SensorSpec][source]#: Returns List of Habitat sensor specs to be passed to habitat-sim.

resolution: Tuple[int, int] = (64, 64)#

zoom: float = 1.0#

class SemanticSensorConfig(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), resolution: ~typing.Tuple[int, int] = (64, 64), zoom: float = 1.0)[source]#

Bases: SensorConfig

Semantic object sensor configuration.

Use this class to configure a sensor to observe known objects in the scene returning their semantic IDs (ground truth) at each XY position. Semantic observations are named “semantic”.

sensor_id#

Optional sensor ID unique within the sensor module. If given then observations made by this sensor will be prefixed by this id. i.e.”sensor_id.semantic”

Type:: str

resolution#

Camera resolution (width, height). Default (64, 64)

Type:: Tuple[int, int]

position#

Sensor position relative to HabitatAgent. Default (0, 0, 0)

Type:: Tuple[float, float, float]

rotation#

Sensor rotation quaternion. Default (1, 0, 0, 0)

Type:: Tuple[float, float, float, float]

zoom#

Camera zoom multiplier. Use >1 to increase, 0<factor<1 to decrease. Default 1.0, no zoom

Type:: float

get_specs() → List[habitat_sim.sensor.SensorSpec][source]#: Returns List of Habitat sensor specs to be passed to habitat-sim.

resolution: Tuple[int, int] = (64, 64)#

zoom: float = 1.0#

class SensorConfig(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0))[source]#

Bases: object

Sensor configuration.

Every habitat sensor will inherit from this class.

sensor_id#

Optional sensorID unique within the sensor module. If given then observations made by this sensor will be prefixed by this id. i.e. “sensor_id.data”

Type:: str

position#

Sensor position relative to HabitatAgent. Default (0, 0, 0)

Type:: Tuple[float, float, float]

rotation#

Sensor rotation quaternion. Default (1, 0, 0, 0)

Type:: Tuple[float, float, float, float]

get_specs() → List[habitat_sim.sensor.SensorSpec][source]#: Returns List of Habitat sensor specs to be passed to habitat-sim.

process_observations(sensor_obs)[source]#

Callback used to process habitat raw sensor observations.

Parameters:: sensor_obs – Sensor raw habitat-sim observations
Returns:: The processed observations grouped by agent_id
Return type:: dict

position: Tuple[float, float, float] = (0.0, 0.0, 0.0)#

rotation: Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0)#

sensor_id: str#

tbp.monty.simulators.habitat.simulator#

AI Habitat simulator interface for Monty.

See also

facebookresearch/habitat-sim

class HabitatSim(agents: List[HabitatAgent], data_path: str | None = None, scene_id: str | None = None, seed: int = 42, enable_physics: bool = True)[source]#

Bases: HabitatActuator

Habitat-sim interface for tbp.monty.

This class wraps habitat-sim simulator for tbp.monty. It aims to hide habitat-sim internals simplifying experiments configuration within the tbp.monty framework.

Example:

camera = SingleSensorAgent(
    agent_id="camera",
    sensor_id="camera_id",
    resolution=(64, 64),
)
with HabitatSim(agents=[camera]) as sim:
    sim.add_object(name="coneSolid", position=(0.0, 1.5, -0.2))
    obs = sim.get_observations()

plot_image(obs["camera"]["camera_id"]["rgba"])
plot_image(obs["camera"]["camera_id"]["depth"])

agents#: List of HabitatAgents to place in the simulator

data_path#: Habitat data path location, usually the same path used by habitat_sim.utils.environments_download

scene_id#: Scene to use or None for empty environment.

seed#: Simulator seed to use

enable_physics#: Whether or not to enable the physics engine. Default False

add_object(name: str, position: Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), scale: Tuple[float, float, float] = (1.0, 1.0, 1.0), semantic_id: str | None = None, enable_physics: bool | None = False, object_to_avoid=False, primary_target_bb=None)[source]#

Add new object to simulated environment.

Parameters:

name – Registered object name. It could be any of habitat-sim primitive objects or any configured habitat object. For a list of primitive objects see PRIMITIVE_OBJECT_TYPES
position – Object initial absolute position
rotation – Object rotation quaternion. Default (1, 0, 0, 0)
scale – Object scale. Default (1, 1, 1)
semantic_id – Optional override object semantic ID
enable_physics – Whether or not to enable physics on this objects
object_to_avoid – If True, run collision checks to ensure the object is not colliding with any other objects in the scene, and otherwise move it
primary_target_bb – If not None, the bounding box of the primary target object; passed when we’re adding multiple objects, such that we ensure that the added object obscures the initial view of the primary target object (which avoiding collision alone cannot guarantee); defined by a list of the min and max corners

Returns:

The newly added object

Return type:

RigidObject

apply_action(action: Action) → Dict[str, Dict][source]#

Execute given action in the environment.

Parameters:: action – The action to execute
Returns:: A dictionary with the observations grouped by agent_id
Raises:: ValueError – If the action name is invalid

check_viewpoint_collision(primary_obj_bb, new_obj_bb, overlap_threshold=0.75)[source]#

Check if the object being added overlaps in the x-axis with target.

The object overlapping the primary target object risks obstructing the initial view of the agent at the start of the experiment.

Recall that +z is out of the page, where the agent starts facing in the -z direction at the beginning of the episode; +y is the up vector, and +x is the right-ward direction

Parameters:

primary_obj_bb – the bounding box of the primary target in the scene
new_obj_bb – the bounding box fo the new object being added
overlap_threshold – The threshold for overlap. Defaults to 0.75.

Returns:

True if the overlap is greater than overlap_threshold; 1.0 corresponds: to total overlap (the primary target is potentially not visible)

Return type:

bool

close()[source]#: Close simulator and release resources.

find_non_colliding_positions(new_object, start_position, start_orientation, primary_obj_bb, max_distance=1, step_size=5e-05)[source]#

Find a position for the object being added.

The criteria are such that the object does not: i) have a physical collision with other objects (i.e. collision meshes intersect) ii) “collide” with the initial view of the primary target object, i.e. obscure the ability of the agent to start on the primary target at the beginning of an experiment

Parameters:

new_object – The object being added
start_position – The starting position of the new object
start_orientation – The initial orientation of the new object
primary_obj_bb – Bounding box of the primary target object (list of two defining corners)
max_distance – The maximum distance to attempt moving the new object
step_size – The step size for moving the new object

Returns:

The newly added object (position updated)

Raises:

RuntimeError – If failed to find a non-colliding position

get_action_space()[source]#

Returns a set with all available actions.

All available actions are those registered by all agents in the environment.

get_agent(agent_id)[source]#: Return habitat agent instance.

get_num_objects()[source]#: Return the number of instantiated objects.

get_observations()[source]#

Get sensor observations.

Returns:

A dictionary with all sensor observations grouped by sensor module.

For example:

{

“agent1”: {

“sensor1”: {: “rgba”: [….], “depth”: [….],

}, “agent2”: {

”sensor2”:
”rgba”: [….], “depth”: [….],

:

}

Return type:

dict

get_states()[source]#

Get agent and sensor states (position, rotation, etc..).

Returns:

A dictionary with the agent pose in world coordinates and any other

agent specific state as well as every sensor pose relative to the agent as well as any sensor specific state that is not returned by get_observations().

For example:

{

“camera”: {

“position”: [2.125, 1.5, -5.278], “rotation”: [0.707107, 0.0, 0.0.707107, 0.0], “sensors” : {

”rgba”: {
“position”: [0.0, 1.5, 0.0], “rotation”: [1.0, 0.0, 0.0, 0.0],

}, “depth”: {

”position”: [0.0, 1.5, 0.0], “rotation”: [1.0, 0.0, 0.0, 0.0],

}, :

}

}, :

}

Return type:

dict

initialize_agent(agent_id, agent_state)[source]#

Update agent runtime state.

Usually called first thing to update agent initial pose.

Parameters:

agent_id – Agent id of the agent to be updated
agent_state – Agent state to update to

non_conflicting_vector()[source]#

Find a non-conflicting vector.

A non-conflicting vector avoids sampling directions that will be just in front of or behind a target object.

Returns:: The non-conflicting vector
Return type:: np.array

process_observations(obs)[source]#

Habitat returns observations grouped by agent_index.

Initially, we group observations by agent_id instead and call all agents to further process the observations.

Parameters:: obs – The observations to process
Returns:: The processed observations grouped by agent_id
Return type:: dict

remove_all_objects()[source]#: Remove all objects from simulated environment.

reset()[source]#

PRIMITIVE_OBJECT_TYPES = {'capsule3DSolid': 101, 'coneSolid': 102, 'cubeSolid': 103, 'cylinderSolid': 104, 'icosphereSolid': 105, 'uvSphereSolid': 106}#: Maps habitat-sim pre-configure primitive object types to semantic IDs