tbp.monty.simulators.habitat#

tbp.monty.simulators.habitat.actions#

class SetAgentPitch(*args: Any, **kwargs: Any) → None#

Bases: SceneNodeControl

Custom habitat-sim action used to set the agent’s absolute pitch rotation.

Note that unless otherwise changed, the sensor maintains its identity orientation with respect to the agent, so this will also adjust the pitch of the sensor with respect to the environment.

This difference in behavior is controlled by the body_action boolean set in the decorator.

The ActuationSpec amount contains the new absolute pitch rotation in degrees.

TODO: add a unit test to habitat_sim_test.py

class SetAgentPose(*args: Any, **kwargs: Any) → None#

Bases: SceneNodeControl

Custom habitat-sim action used to set the agent pose.

This action sets the agent body’s absolute location (xyz coordinate) and absolute rotation (i.e. relative to the identity rotation, which is defined by the axes of xyz in Habitat) within the environment.

The ActuationSpec amount contains location and rotation (latter as a numpy quaternion)

class SetSensorPitch(*args: Any, **kwargs: Any) → None#

Bases: SceneNodeControl

Custom habitat-sim action used to set the sensor’s absolute pitch rotation.

Note that this does not update the pitch of the agent (imagine e.g. the “body” associated with the eye remaining in place, but the eye moving).

The ActuationSpec amount contains the new absolute pitch rotation in degrees.

class SetSensorPose(*args: Any, **kwargs: Any) → None#

Bases: SceneNodeControl

Custom habitat-sim action used to set the sensor pose.

This action sets the sensor’s absolute location (xyz coordinate), and rotation relative to the agent.

Note that body_action=False, which results in a sensor (rather than agent) update.

The ActuationSpec amount contains location and rotation (latter as a numpy quaternion)

class SetSensorRotation(*args: Any, **kwargs: Any) → None#

Bases: SceneNodeControl

Custom habitat-sim action used to set the sensor rotation.

This action sets the sensor’s absolute rotation relative to the agent.

Note that body_action=False, which results in a sensor (rather than agent) update.

The ActuationSpec amount contains rotation (as a numpy quaternion)

class SetYaw(*args: Any, **kwargs: Any) → None#

Bases: SceneNodeControl

Custom habitat-sim action used to set the agent body’s absolute yaw rotation.

The ActuationSpec amount contains the new absolute yaw rotation in degrees.

tbp.monty.simulators.habitat.actuator#

class HabitatActuator(*args, **kwargs)[source]#

Bases: HabitatActuatorRequirements

Habitat implementation of an Actuator.

HabitatActuator is responsible for executing actions in the Habitat simulation.

It is a separate class to encapsulate the actuation logic in one place. This class is expected to be mixed into HabitatSim and expects HabitatActuatorRequirements to be met.

Note

Habitat does not expose an API for passing parameters to actions. Each actuate method works around this limitation by artisanally setting specific action parameters directly in habitat-sim.

action_name(action: Action) → str[source]#

Return Monty’s Habitat action naming convention.

The action name is prefixed by the agent ID.

Return type:: str

actuate_look_down(action: LookDown) → None[source]#

Return type:: None

actuate_look_up(action: LookUp) → None[source]#

Return type:: None

actuate_move_forward(action: MoveForward) → None[source]#

Return type:: None

actuate_move_tangentially(action: MoveTangentially) → None[source]#

Return type:: None

actuate_orient_horizontal(action: OrientHorizontal) → None[source]#

Return type:: None

actuate_orient_vertical(action: OrientVertical) → None[source]#

Return type:: None

actuate_set_agent_pitch(action: SetAgentPitch) → None[source]#

Return type:: None

actuate_set_agent_pose(action: SetAgentPose) → None[source]#

Return type:: None

actuate_set_sensor_pitch(action: SetSensorPitch) → None[source]#

Return type:: None

actuate_set_sensor_pose(action: SetSensorPose) → None[source]#

Return type:: None

actuate_set_sensor_rotation(action: SetSensorRotation) → None[source]#

Return type:: None

actuate_set_yaw(action: SetYaw) → None[source]#

Return type:: None

actuate_turn_left(action: TurnLeft) → None[source]#

Return type:: None

actuate_turn_right(action: TurnRight) → None[source]#

Return type:: None

to_habitat(action: Action) → tuple[Agent, ActuationSpec, str][source]#

Transition from the Monty to the habitat-sim domain.

Parameters:

action – Monty action to execute by the agent specified in the action.

Returns:

The Habitat agent to execute the action, the Habitat action parameters to set prior to executing the action, and the Habitat action name to execute.

Raises:

InvalidActionName – If the action name is invalid.
NoActionParameters – If the action has no parameters.

class HabitatActuatorRequirements(*args, **kwargs)[source]#

Bases: Protocol

HabitatActuator requires these to be available when mixed in.

__init__(*args, **kwargs)#

get_agent(agent_id: AgentID) → Agent | None[source]#

Return type:: Agent | None

tbp.monty.simulators.habitat.agents#

class HabitatAgent(agent_id: AgentID | None, position: Vector3 = (0.0, 1.5, 0.0), rotation: Quaternion = (1.0, 0.0, 0.0, 0.0), height: float = 0.0)[source]#

Bases: object

Habitat agent wrapper.

Agents are used to define movable bodies in the environment. Every Habitat agent will inherit from this class.

agent_id#: Unique ID of this agent in the environment. Observations returned by the environment will be mapped to this ID. {"agent_id": {"sensor": [...]}}. Actions provided by this sensor module will be prefixed by this ID, i.e. “agent_id.move_forward”

position#: Module initial position in meters. Defaults to (0, 1.5, 0).

rotation#: Module initial rotation quaternion. Defaults to (1, 0, 0, 0).

height#: Module height in meters. Defaults to 0.0.

__init__(agent_id: AgentID | None, position: Vector3 = (0.0, 1.5, 0.0), rotation: Quaternion = (1.0, 0.0, 0.0, 0.0), height: float = 0.0)[source]#

get_spec() → habitat_sim.agent.AgentConfiguration[source]#

Return a habitat-sim agent configuration.

Return type:: AgentConfiguration
Returns:: Spec created from this sensor module configuration.

initialize(simulator)[source]#

Initialize the habitat-sim agent’s runtime state.

This method must be called to update the agent and sensors runtime instance because some of the configuration attributes require access to the instantiated node.

Parameters:: simulator – Instantiated HabitatSim instance

process_observations(agent_obs) → AgentObservations[source]#

Callback that processes raw Habitat agent observations.

Converts raw observations into Monty-compatible ones.

Parameters:: agent_obs – Raw habitat-sim observations from the agent
Return type:: AgentObservations
Returns:: The processed observations grouped by sensor_id

class MultiSensorAgent(agent_id: AgentID | None, sensor_ids: tuple[str], position: Vector3 = (0.0, 1.5, 0.0), rotation: Quaternion = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: ActionSpaceName = 'distant_agent', resolutions: tuple[Size] = ((16, 16),), positions: tuple[Vector3] = ((0.0, 0.0, 0.0),), rotations: tuple[Quaternion] = ((1.0, 0.0, 0.0, 0.0),), zooms: tuple[float] = (1.0,), semantics: tuple[bool] = (False,))[source]#

Bases: HabitatAgent

Minimal version of a HabitatAgent with multiple RGBD sensors mounted.

The RGBD sensors are mounted to the same movable object (like two go-pros mounted to a helmet) with the following pre-defined actions:

“agent_id.move_forward”: Move camera forward using translation_step

“agent_id.turn_left”: Turn camera left rotation_step

“agent_id.turn_right”: Turn camera right rotation_step

“agent_id.look_up”: Turn the camera up rotation_step

“agent_id.look_down”: Turn the camera down rotation_step

“agent_id.set_yaw”: Set the camera agent’s absolute yaw value

“agent_id.set_sensor_pitch”: Set the camera sensor’s absolute pitch value

“agent_id.set_agent_pitch”: Set the camera agent’s absolute pitch value

Each camera will return the following observations:

“sensor_ids[i].rgba”: Color information for every pixel (x, y, 4)

“sensor_ids[i].depth”: Depth information for every pixel (x, y, 1)

“sensor_ids[i].semantic”: Optional object semantic information for every pixel
(x, y, 1)

where i is an integer indexing the list of sensor_ids.

Note

The parameters resolutions, rotations, and so on effectively specify both the number of sensors, and the sensor parameters. For N sensors, specify a list of N resolutions, and so on. All lists must be the same length. By default, a list of length one will be provided. Therefore, do not leave an argument blank if you wish to run a simulation with N > 1 sensors.

Note

The parameters translation_step and rotation_step are set to 0 by default. All action amounts should be specified by the MotorSystem.

agent_id#: Actions provided by this camera will be prefixed by this ID. Defaults to “camera”.

sensor_ids#: List of IDs for each sensor. Actions are prefixed with agent ID, but observations are prefixed with sensor ID.

resolutions#: List of camera resolutions (width, height). Defaults to (16, 16).

positions#: List of camera initial absolute positions in meters, relative to the agent.

rotations#: List of camera rotations (quaternion). Defaults to (1, 0, 0, 0).

zooms#: List of camera zoom multipliers. Use >1 to increase and 0 < factor < 1 to decrease. Defaults to 1.0.

semantics#: List of booleans determining if each RGBD sensor also gets a semantic sensor with it. Defaults to False.

height#: Height of the mount itself in meters. Defaults to 0.0 (but position of the agent will be 1.5 meters in the “height” dimension).

rotation_step#: Rotation step in degrees used by the turn_* and look_* actions. Defaults to 0 degrees.

translation_step#: Translation length in meters used by the move_* actions. Defaults to 0 m.

action_space_type#: Decides between three action spaces: “distant_agent” actions saccade like a ball-in-socket joint, viewing an object from a distance. “surface_agent” actions orient to an object surface and move tangentially along it. “absolute_only” actions are movements in absolute world coordinates only.

__init__(agent_id: AgentID | None, sensor_ids: tuple[str], position: Vector3 = (0.0, 1.5, 0.0), rotation: Quaternion = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: ActionSpaceName = 'distant_agent', resolutions: tuple[Size] = ((16, 16),), positions: tuple[Vector3] = ((0.0, 0.0, 0.0),), rotations: tuple[Quaternion] = ((1.0, 0.0, 0.0, 0.0),), zooms: tuple[float] = (1.0,), semantics: tuple[bool] = (False,))[source]#

get_spec()[source]#

Return a habitat-sim agent configuration.

Returns:: Spec created from this sensor module configuration.

initialize(simulator)[source]#

Initialize agent runtime state.

This method must be called by HabitatSim to update the agent and sensors runtime instance. This is necessary because some of the configuration attributes require access to the instantiated scene node.

Parameters:: simulator – Instantiated HabitatSim instance

class SingleSensorAgent(agent_id: AgentID | None, sensor_id: SensorID, agent_position: Vector3 = (0.0, 1.5, 0.0), sensor_position: Vector3 = (0.0, 0.0, 0.0), rotation: Quaternion = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, resolution: Size = (16, 16), zoom: float = 1.0, semantic: bool = False, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: ActionSpaceName = 'distant_agent')[source]#

Bases: HabitatAgent

Minimal version of a HabitatAgent.

This is the special case of MultiSensorAgent when there is at most 1 RGBD and 1 semantic sensor. Thus, the arguments are single values instead of lists.

__init__(agent_id: AgentID | None, sensor_id: SensorID, agent_position: Vector3 = (0.0, 1.5, 0.0), sensor_position: Vector3 = (0.0, 0.0, 0.0), rotation: Quaternion = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, resolution: Size = (16, 16), zoom: float = 1.0, semantic: bool = False, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: ActionSpaceName = 'distant_agent')[source]#

Initialize agent runtime state.

Note that, like the multi-sensor agent, the position of the agent is treated separately from the position of the sensor (which is relative to the agent).

get_spec()[source]#

Return a habitat-sim agent configuration.

Returns:: Spec created from this sensor module configuration.

initialize(simulator)[source]#

Initialize agent runtime state.

Parameters:: simulator – Instantiated HabitatSim instance

tbp.monty.simulators.habitat.environment#

class AgentConfig(agent_type: type[HabitatAgent], agent_args: dict | type[HabitatAgentArgs]) → None[source]#

Bases: object

Agent configuration used by HabitatEnvironment.

__init__(agent_type, agent_args)#

agent_args: dict | type[HabitatAgentArgs]#

agent_type: type[HabitatAgent]#

Bases: SimulatedObjectEnvironment

habitat-sim environment compatible with Monty.

agents#: List of AgentConfig to place in the scene.

objects#: Optional list of ObjectConfig to place in the scene.

scene_id#: Scene to use or None for empty environment.

seed#: Simulator seed to use.

data_path#: Path to the dataset.

__init__(agents: dict | AgentConfig, objects: list[dict | ObjectConfig] | None = None, scene_id: str | None = None, seed: int = 42, data_path: str | Path | None = None)[source]#

add_object(name: str, position: VectorXYZ = (0.0, 0.0, 0.0), rotation: QuaternionWXYZ = (1.0, 0.0, 0.0, 0.0), scale: VectorXYZ = (1.0, 1.0, 1.0), semantic_id: SemanticID | None = None, primary_target_object: ObjectID | None = None) → ObjectID[source]#

Add an object to the environment.

Parameters:

name (str) – The name of the object to add.
position (VectorXYZ) – The initial absolute position of the object.
rotation (QuaternionWXYZ) – The initial rotation WXYZ quaternion of the object. Defaults to (1,0,0,0).
scale (VectorXYZ) – The scale of the object to add. Defaults to (1,1,1).
semantic_id (SemanticID | None) – Optional override for the object semantic ID. Defaults to None.
primary_target_object (ObjectID | None) – The ID of the primary target object. If not None, the added object will be positioned so that it does not obscure the initial view of the primary target object (which avoiding collision alone cannot guarantee). Used when adding multiple objects. Defaults to None.

Return type:

ObjectID

Returns:

The ID of the added object.

close() → None[source]#

Close the environment and release all resources.

Any call to any other environment method may raise an exception.

Return type:: None

remove_all_objects() → None[source]#

Remove all objects from the environment.

Return type:: None

TODO: This remove_all_objects interface is elevated from: HabitatSim.remove_all_objects and is quite specific to HabitatSim implementation. We should consider refactoring this to be more generic.

reset() → tuple[Observations, ProprioceptiveState][source]#

Reset the environment to its initial state.

Returns:: The environment’s initial observations and proprioceptive state.

step(actions: Sequence[Action]) → tuple[Observations, ProprioceptiveState][source]#

Apply the given actions to the environment.

Parameters:: actions – The actions to apply to the environment.
Returns:: The current observations and proprioceptive state.

Note

If the actions are an empty sequence, the current observations are returned.

class MultiSensorAgentArgs(agent_id: AgentID | None, sensor_ids: tuple[str], position: tuple = (0.0, 1.5, 0.0), rotation: tuple = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: str = 'distant_agent', resolutions: tuple = ((16, 16),), positions: tuple = ((0.0, 0.0, 0.0),), rotations: tuple = ((1.0, 0.0, 0.0, 0.0),), zooms: tuple = (1.0,), semantics: tuple = (False,)) → None#

Bases: HabitatAgentArgs

__init__(agent_id: AgentID | None, sensor_ids: tuple[str], position: tuple = (0.0, 1.5, 0.0), rotation: tuple = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: str = 'distant_agent', resolutions: tuple = ((16, 16),), positions: tuple = ((0.0, 0.0, 0.0),), rotations: tuple = ((1.0, 0.0, 0.0, 0.0),), zooms: tuple = (1.0,), semantics: tuple = (False,)) → None#

action_space_type: str = 'distant_agent'#

agent_id: AgentID | None#

height: float = 0.0#

position: tuple = (0.0, 1.5, 0.0)#

positions: tuple = ((0.0, 0.0, 0.0),)#

resolutions: tuple = ((16, 16),)#

rotation: tuple = (1.0, 0.0, 0.0, 0.0)#

rotation_step: float = 0.0#

rotations: tuple = ((1.0, 0.0, 0.0, 0.0),)#

semantics: tuple = (False,)#

sensor_ids: tuple[str]#

translation_step: float = 0.0#

zooms: tuple = (1.0,)#

class ObjectConfig(name: str, position: tuple = (0.0, 0.0, 0.0), rotation: tuple = (1.0, 0.0, 0.0, 0.0), scale: tuple = (1.0, 1.0, 1.0), semantic_id: None = None, primary_target_object: None = None) → None#

Bases: object

__init__(name: str, position: tuple = (0.0, 0.0, 0.0), rotation: tuple = (1.0, 0.0, 0.0, 0.0), scale: tuple = (1.0, 1.0, 1.0), semantic_id: None = None, primary_target_object: None = None) → None#

name: str#

position: tuple = (0.0, 0.0, 0.0)#

primary_target_object: None = None#

rotation: tuple = (1.0, 0.0, 0.0, 0.0)#

scale: tuple = (1.0, 1.0, 1.0)#

semantic_id: None = None#

class SingleSensorAgentArgs(agent_id: AgentID | None, sensor_id: SensorID, agent_position: tuple = (0.0, 1.5, 0.0), sensor_position: tuple = (0.0, 0.0, 0.0), rotation: tuple = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, resolution: tuple = (16, 16), zoom: float = 1.0, semantic: bool = False, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: str = 'distant_agent') → None#

Bases: HabitatAgentArgs

__init__(agent_id: AgentID | None, sensor_id: SensorID, agent_position: tuple = (0.0, 1.5, 0.0), sensor_position: tuple = (0.0, 0.0, 0.0), rotation: tuple = (1.0, 0.0, 0.0, 0.0), height: float = 0.0, resolution: tuple = (16, 16), zoom: float = 1.0, semantic: bool = False, rotation_step: float = 0.0, translation_step: float = 0.0, action_space_type: str = 'distant_agent') → None#

action_space_type: str = 'distant_agent'#

agent_id: AgentID | None#

agent_position: tuple = (0.0, 1.5, 0.0)#

height: float = 0.0#

resolution: tuple = (16, 16)#

rotation: tuple = (1.0, 0.0, 0.0, 0.0)#

rotation_step: float = 0.0#

semantic: bool = False#

sensor_id: SensorID#

sensor_position: tuple = (0.0, 0.0, 0.0)#

translation_step: float = 0.0#

zoom: float = 1.0#

tbp.monty.simulators.habitat.sensors#

class RGBDSensorConfig(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), resolution: ~typing.Tuple[int, int] = (64, 64), zoom: float = 1.0) → None[source]#

Bases: SensorConfig

RGBD camera sensor configuration.

Use this class to configure two different Habitat sensors simultaneously, one for RGBA observations and another for depth. Both sensors will use the same pose, resolution and zoom factor. RGB observations are named “rgba”, depth observations are named “depth”.

sensor_id#: Sensor ID unique within the sensor module. If given, then observations made by this sensor will be prefixed by this ID. i.e. “sensor_id.rgba”

resolution#: Camera resolution (width, height). Defaults to (64, 64).

position#: Sensor position relative to HabitatAgent. Defaults to (0, 0, 0).

rotation#: Sensor rotation quaternion. Defaults to (1, 0, 0, 0).

zoom#: Camera zoom multiplier. Use >1 to increase and 0 < factor < 1 to decrease. Defaults to 1.0 (no zoom).

__init__(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), resolution: ~typing.Tuple[int, int] = (64, 64), zoom: float = 1.0) → None#

get_specs() → list[SensorSpec][source]#: Return a list of Habitat sensor specs to pass to habitat-sim.

resolution: Tuple[int, int] = (64, 64)#

zoom: float = 1.0#

class SemanticSensorConfig(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), resolution: ~typing.Tuple[int, int] = (64, 64), zoom: float = 1.0) → None[source]#

Bases: SensorConfig

Semantic object sensor configuration.

Use this class to configure a sensor to observe known objects in the scene, returning their semantic IDs (ground truth) at each XY position. Semantic observations are named “semantic”.

sensor_id#: Optional sensor ID unique within the sensor module. If given, then observations made by this sensor will be prefixed by this ID. i.e. “sensor_id.semantic”

resolution#: Camera resolution (width, height). Defaults to (64, 64).

position#: Sensor position relative to HabitatAgent. Defaults to (0, 0, 0).

rotation#: Sensor rotation quaternion. Defaults to (1, 0, 0, 0).

zoom#: Camera zoom multiplier. Use >1 to increase and 0 < factor < 1 to decrease. Defaults to 1.0 (no zoom).

__init__(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), resolution: ~typing.Tuple[int, int] = (64, 64), zoom: float = 1.0) → None#

get_specs() → list[SensorSpec][source]#: Return a list of Habitat sensor specs to pass to habitat-sim.

resolution: Tuple[int, int] = (64, 64)#

zoom: float = 1.0#

class SensorConfig(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0)) → None[source]#

Bases: object

Sensor configuration.

Every Habitat sensor will inherit from this class.

sensor_id#: Optional sensor ID unique within the sensor module. If given, then observations made by this sensor will be prefixed by this ID, i.e. “sensor_id.data”

position#: Sensor position relative to HabitatAgent. Defaults to (0, 0, 0).

rotation#: Sensor rotation quaternion. Defaults to (1, 0, 0, 0).

__init__(sensor_id: str = <factory>, position: ~typing.Tuple[float, float, float] = (0.0, 0.0, 0.0), rotation: ~typing.Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0)) → None#

get_specs() → list[SensorSpec][source]#: Return a list of Habitat sensor specs to pass to habitat-sim.

process_observations(sensor_obs: SensorObservation) → SensorObservation[source]#

Callback used to process Habitat raw sensor observations.

Parameters:: sensor_obs (SensorObservation) – Raw habitat-sim observations from this sensor
Return type:: SensorObservation
Returns:: The processed observations grouped by agent_id

position: Tuple[float, float, float] = (0.0, 0.0, 0.0)#

rotation: Tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0)#

sensor_id: str#

tbp.monty.simulators.habitat.simulator#

AI Habitat simulator interface for Monty.

See also

facebookresearch/habitat-sim

class HabitatSim(agents: list[HabitatAgent], data_path: str | Path | None = None, scene_id: str | None = None, seed: int = 42)[source]#

Bases: HabitatActuator, Simulator

habitat-sim interface for tbp.monty.

This class wraps habitat-sim simulator for tbp.monty. It aims to hide habitat-sim internals, simplifying experiment configuration within the tbp.monty framework.

Example:

camera = SingleSensorAgent(
    agent_id=AgentID("camera"),
    sensor_id="camera_id",
    resolution=(64, 64),
)
with HabitatSim(agents=[camera]) as sim:
    sim.add_object(name="coneSolid", position=(0.0, 1.5, -0.2))
    obs = sim.observations

plot_image(obs["camera"]["camera_id"]["rgba"])
plot_image(obs["camera"]["camera_id"]["depth"])

agents#: List of HabitatAgent instances to place in the simulator.

data_path#: Habitat data path location, usually the same path used by habitat_sim.utils.environments_download.

scene_id#: Scene to use or None for empty environment.

seed#: Simulator seed to use.

__init__(agents, data_path=None, scene_id=None, seed=42)[source]#

Add new object to simulated environment.

Parameters:

name (str) – Registered object name. It can be any habitat-sim primitive object or any configured habitat object. See PRIMITIVE_OBJECT_TYPES for a list of primitive objects.
position (VectorXYZ) – Object initial absolute position.
rotation (QuaternionWXYZ) – Object rotation quaternion. Defaults to (1, 0, 0, 0).
scale (VectorXYZ) – Object scale. Defaults to (1, 1, 1).
semantic_id (SemanticID | None) – Optional override object semantic ID. Defaults to None.
primary_target_object (ObjectID | None) – ID of the primary target object. If not None, the added object will be positioned so that it does not obscure the initial view of the primary target object (which avoiding collision alone cannot guarantee). Used when adding multiple objects. Defaults to None.

Return type:

ObjectInfo

Returns:

The added object’s information.

check_viewpoint_collision(primary_obj_bb, new_obj_bb, overlap_threshold=0.75) → bool[source]#

Check if the object being added overlaps in the x-axis with target.

The object overlapping the primary target object risks obstructing the initial view of the agent at the start of the experiment.

Recall that +z is out of the page, where the agent starts facing in the -z direction at the beginning of the episode; +y is the up vector, and +x is the right-ward direction.

Parameters:

primary_obj_bb – the bounding box of the primary target in the scene
new_obj_bb – the bounding box of the new object being added
overlap_threshold – The threshold for overlap. Defaults to 0.75.

Return type:

bool

Returns:

True if the overlap is greater than overlap_threshold; 1.0 corresponds to total overlap (the primary target is potentially not visible)

close() → None[source]#

Close simulator and release resources.

Return type:: None

find_non_colliding_positions(new_object, start_position, start_orientation, primary_obj_bb, max_distance=1, step_size=5e-05)[source]#

Find a position for the object being added.

The criteria are such that the object does not: i) have a physical collision with other objects (i.e., collision meshes intersect) ii) “collide” with the initial view of the primary target object, i.e., obscure the ability of the agent to start on the primary target at the beginning of an experiment

Parameters:

new_object – The object being added
start_position – The starting position of the new object
start_orientation – The initial orientation of the new object
primary_obj_bb – Bounding box of the primary target object (list of two defining corners)
max_distance – The maximum distance to attempt moving the new object
step_size – The step size for moving the new object

Returns:

The newly added object (position updated)

Raises:

RuntimeError – If failed to find a non-colliding position

get_agent(agent_id: AgentID) → habitat_sim.Agent[source]#

Return habitat agent instance.

Return type:: Agent

initialize_agent(agent_id: AgentID, agent_state) → None[source]#

Update the agent’s runtime state.

Usually called first thing to update the agent’s initial pose.

Parameters:

agent_id (NewType()(AgentID, str)) – Agent ID of the agent to be updated.
agent_state – Agent state to update to.

Return type:

None

non_conflicting_vector() → numpy.ndarray[source]#

Find a non-conflicting vector.

A non-conflicting vector avoids sampling directions that will be just in front of or behind a target object.

Return type:: ndarray
Returns:: The non-conflicting vector

process_observations(habitat_obs: dict[int, ObservationDict]) → Observations[source]#

Habitat returns observations grouped by agent_index.

Initially, we group observations by agent_id instead and call all agents to further process the observations.

Parameters:: habitat_obs – The observations from HabitatSim to process.
Returns:: Observations grouped by agent_id.

remove_all_objects() → None[source]#

Remove all objects from the simulated environment.

Return type:: None

reset() → tuple[Observations, ProprioceptiveState][source]#

Reset the simulator.

Returns:: The initial observations from the simulator and proprioceptive state.

step(actions: Sequence[Action]) → tuple[Observations, ProprioceptiveState][source]#

Execute given actions in the environment.

Parameters:

actions – The actions to execute

Returns:

The observations and proprioceptive state.

Raises:

TypeError – If the action type is invalid
ValueError – If the action name is invalid

property action_space: set[str]#: Return the action space.

property num_objects#: Return the number of instantiated objects.

property observations: Observations#

Retrieve and process observations from the simulator.

Returns:: All observations.

property states: ProprioceptiveState#

Returns proprioceptive state of the agents and sensors.

Note: A Monty RGBD sensor is represented by separate RGBA and depth sensors in Habitat (and a separate semantic sensor, if enabled). Their positions and rotations are identical (they belong to the same body), so we arbitrarily return the position and rotation from whichever sensor we see first.

PRIMITIVE_OBJECT_TYPES = {'capsule3DSolid': 101, 'coneSolid': 102, 'cubeSolid': 103, 'cylinderSolid': 104, 'icosphereSolid': 105, 'uvSphereSolid': 106}#: Maps habitat-sim pre-configured primitive object types to semantic IDs