|
| 1 | +--- |
| 2 | +outline: [2, 4] |
| 3 | +--- |
| 4 | + |
| 5 | +# Chapter 14: Environment Raycast |
| 6 | + |
| 7 | +The IWSDK provides an environment raycast system that enables AR applications to detect real-world surfaces and place virtual content on them. This chapter covers implementing hit-testing in your WebXR applications. |
| 8 | + |
| 9 | +## What You'll Build |
| 10 | + |
| 11 | +By the end of this chapter, you'll be able to: |
| 12 | + |
| 13 | +- Set up environment raycasting for AR hit-testing |
| 14 | +- Create objects that follow controller raycasts to real-world surfaces |
| 15 | +- Place virtual content on detected surfaces with a trigger press |
| 16 | +- Configure different ray sources (controllers, gaze, screen touch) |
| 17 | +- Align objects to surface normals automatically |
| 18 | + |
| 19 | +## Overview |
| 20 | + |
| 21 | +The environment raycast system leverages WebXR's hit-test API to cast rays against real-world geometry and position virtual objects at hit points. Unlike Scene Understanding (Chapter 11), environment raycasting **does not require room scanning** - it works immediately using real-time surface detection. |
| 22 | + |
| 23 | +### Environment Raycast vs Scene Understanding |
| 24 | + |
| 25 | +| Feature | Environment Raycast | Scene Understanding | |
| 26 | +| ------------------------- | ------------------- | ------------------- | |
| 27 | +| Room scanning required | No | Yes | |
| 28 | +| Works immediately | Yes | After scanning | |
| 29 | +| Provides surface position | Yes | Yes | |
| 30 | +| Provides surface normal | Yes | Yes | |
| 31 | +| Detects planes/meshes | No | Yes | |
| 32 | +| Semantic labels | No | Yes | |
| 33 | +| Best for | Instant placement | Rich scene data | |
| 34 | + |
| 35 | +**Use Environment Raycast when** you want immediate tap-to-place or controller-based placement without waiting for room scanning. |
| 36 | + |
| 37 | +**Use Scene Understanding when** you need detailed information about detected surfaces like semantic labels (table, wall, floor) or mesh geometry. |
| 38 | + |
| 39 | +### Key Components |
| 40 | + |
| 41 | +- **`EnvironmentRaycastSystem`** - Core system that manages WebXR hit-test sources |
| 42 | +- **`EnvironmentRaycastTarget`** - Component that positions an entity at raycast hit points |
| 43 | +- **`RaycastSpace`** - Enum for ray source selection (Left, Right, Viewer, Screen) |
| 44 | + |
| 45 | +## Quick Start |
| 46 | + |
| 47 | +Here's a minimal example to get environment raycasting working: |
| 48 | + |
| 49 | +```typescript |
| 50 | +import { |
| 51 | + World, |
| 52 | + SessionMode, |
| 53 | + EnvironmentRaycastTarget, |
| 54 | + RaycastSpace, |
| 55 | +} from '@iwsdk/core'; |
| 56 | + |
| 57 | +World.create(document.getElementById('scene-container'), { |
| 58 | + xr: { |
| 59 | + sessionMode: SessionMode.ImmersiveAR, |
| 60 | + features: { |
| 61 | + hitTest: { required: true }, // Enable WebXR hit-test |
| 62 | + }, |
| 63 | + }, |
| 64 | + features: { |
| 65 | + environmentRaycast: true, // Enable EnvironmentRaycastSystem |
| 66 | + }, |
| 67 | +}).then((world) => { |
| 68 | + // Create a reticle that follows the raycast |
| 69 | + const reticleMesh = createReticleMesh(); // Your reticle geometry |
| 70 | + const reticle = world.createTransformEntity(reticleMesh); |
| 71 | + |
| 72 | + reticle.addComponent(EnvironmentRaycastTarget, { |
| 73 | + space: RaycastSpace.Right, // Use right controller |
| 74 | + maxDistance: 10, // Maximum raycast distance in meters |
| 75 | + }); |
| 76 | + |
| 77 | + // The reticle now automatically: |
| 78 | + // - Moves to where the controller points at real surfaces |
| 79 | + // - Orients to match the surface normal |
| 80 | + // - Hides when there's no hit |
| 81 | +}); |
| 82 | +``` |
| 83 | + |
| 84 | +## System Setup |
| 85 | + |
| 86 | +### Step 1: Enable WebXR Hit-Test Feature |
| 87 | + |
| 88 | +```typescript |
| 89 | +World.create(container, { |
| 90 | + xr: { |
| 91 | + sessionMode: SessionMode.ImmersiveAR, |
| 92 | + features: { |
| 93 | + hitTest: { required: true }, // Required for environment raycast |
| 94 | + }, |
| 95 | + }, |
| 96 | + features: { |
| 97 | + environmentRaycast: true, |
| 98 | + }, |
| 99 | +}); |
| 100 | +``` |
| 101 | + |
| 102 | +**Important**: The `hitTest` WebXR feature must be enabled for environment raycasting to work. |
| 103 | + |
| 104 | +### Step 2: Create a Raycast Target Entity |
| 105 | + |
| 106 | +```typescript |
| 107 | +const previewMesh = createPreviewMesh(); |
| 108 | +const previewEntity = world.createTransformEntity(previewMesh); |
| 109 | + |
| 110 | +previewEntity.addComponent(EnvironmentRaycastTarget, { |
| 111 | + space: RaycastSpace.Right, |
| 112 | + maxDistance: 10, |
| 113 | +}); |
| 114 | +``` |
| 115 | + |
| 116 | +The entity's Object3D will automatically: |
| 117 | + |
| 118 | +- **Position** at the raycast hit point |
| 119 | +- **Orient** to match the surface normal |
| 120 | +- **Hide** when there's no hit (visibility set to false) |
| 121 | + |
| 122 | +## Understanding the Components |
| 123 | + |
| 124 | +### EnvironmentRaycastTarget |
| 125 | + |
| 126 | +Makes an entity follow environment raycast hit points. |
| 127 | + |
| 128 | +#### Input Properties |
| 129 | + |
| 130 | +- **`space`** - Ray source: `RaycastSpace.Left`, `Right`, `Viewer`, or `Screen` (default: `Right`) |
| 131 | +- **`maxDistance`** - Maximum raycast distance in meters (default: `100`) |
| 132 | +- **`offsetPosition`** - Offset from ray origin (default: `undefined`) |
| 133 | +- **`offsetQuaternion`** - Rotation offset for ray direction (default: `undefined`) |
| 134 | + |
| 135 | +#### Output Properties (Read-only) |
| 136 | + |
| 137 | +- **`xrHitTestResult`** - The underlying `XRHitTestResult` when there's a hit, `undefined` otherwise |
| 138 | +- **`inputSource`** - For Screen mode: the `XRInputSource` that triggered the hit |
| 139 | + |
| 140 | +```typescript |
| 141 | +// Check if there's a valid hit |
| 142 | +const xrResult = entity.getValue(EnvironmentRaycastTarget, 'xrHitTestResult'); |
| 143 | +if (xrResult) { |
| 144 | + console.log('Hit detected at:', entity.object3D.position); |
| 145 | +} |
| 146 | +``` |
| 147 | + |
| 148 | +### RaycastSpace Options |
| 149 | + |
| 150 | +Choose the ray source based on your use case: |
| 151 | + |
| 152 | +| Space | Description | Best For | |
| 153 | +| --------------------- | ----------------------------- | -------------------------- | |
| 154 | +| `RaycastSpace.Right` | Right controller's target ray | Quest controller placement | |
| 155 | +| `RaycastSpace.Left` | Left controller's target ray | Left-handed users | |
| 156 | +| `RaycastSpace.Viewer` | Head/gaze direction | Gaze-based placement | |
| 157 | +| `RaycastSpace.Screen` | Screen touch (phone AR) | Tap-to-place on phones | |
| 158 | + |
| 159 | +## Common Patterns |
| 160 | + |
| 161 | +### Preview + Place Pattern |
| 162 | + |
| 163 | +The most common pattern: show a preview that follows the raycast, then spawn a permanent object on trigger press. |
| 164 | + |
| 165 | +```typescript |
| 166 | +import { |
| 167 | + AssetManager, |
| 168 | + createSystem, |
| 169 | + EnvironmentRaycastTarget, |
| 170 | + RaycastSpace, |
| 171 | +} from '@iwsdk/core'; |
| 172 | + |
| 173 | +class PlacementSystem extends createSystem({ |
| 174 | + targets: { required: [EnvironmentRaycastTarget] }, |
| 175 | +}) { |
| 176 | + private previewEntity: Entity | null = null; |
| 177 | + |
| 178 | + init() { |
| 179 | + // Create preview object |
| 180 | + const previewMesh = AssetManager.getGLTF('myObject').scene.clone(); |
| 181 | + this.previewEntity = this.world.createTransformEntity(previewMesh); |
| 182 | + this.previewEntity.addComponent(EnvironmentRaycastTarget, { |
| 183 | + space: RaycastSpace.Right, |
| 184 | + maxDistance: 10, |
| 185 | + }); |
| 186 | + } |
| 187 | + |
| 188 | + update() { |
| 189 | + const triggerPressed = this.input.gamepads.right?.getSelectStart(); |
| 190 | + |
| 191 | + if (triggerPressed && this.previewEntity) { |
| 192 | + const xrResult = this.previewEntity.getValue( |
| 193 | + EnvironmentRaycastTarget, |
| 194 | + 'xrHitTestResult', |
| 195 | + ); |
| 196 | + |
| 197 | + // Only place if we have a valid hit and preview is visible |
| 198 | + if (xrResult && this.previewEntity.object3D?.visible) { |
| 199 | + this.spawnObject( |
| 200 | + this.previewEntity.object3D.position.clone(), |
| 201 | + this.previewEntity.object3D.quaternion.clone(), |
| 202 | + ); |
| 203 | + } |
| 204 | + } |
| 205 | + } |
| 206 | + |
| 207 | + private spawnObject(position: Vector3, quaternion: Quaternion) { |
| 208 | + const mesh = AssetManager.getGLTF('myObject').scene.clone(); |
| 209 | + mesh.position.copy(position); |
| 210 | + mesh.quaternion.copy(quaternion); |
| 211 | + this.scene.add(mesh); |
| 212 | + this.world.createTransformEntity(mesh); |
| 213 | + } |
| 214 | +} |
| 215 | +``` |
| 216 | + |
| 217 | +### Phone AR Tap-to-Place |
| 218 | + |
| 219 | +For phone-based AR experiences, use `RaycastSpace.Screen` to detect where the user taps: |
| 220 | + |
| 221 | +```typescript |
| 222 | +const reticle = world.createTransformEntity(reticleMesh); |
| 223 | +reticle.addComponent(EnvironmentRaycastTarget, { |
| 224 | + space: RaycastSpace.Screen, // Tracks screen touch |
| 225 | + maxDistance: 10, |
| 226 | +}); |
| 227 | + |
| 228 | +// In your system, check for the input source |
| 229 | +class TapPlaceSystem extends createSystem({ |
| 230 | + targets: { required: [EnvironmentRaycastTarget] }, |
| 231 | +}) { |
| 232 | + update() { |
| 233 | + this.queries.targets.entities.forEach((entity) => { |
| 234 | + const inputSource = entity.getValue( |
| 235 | + EnvironmentRaycastTarget, |
| 236 | + 'inputSource', |
| 237 | + ); |
| 238 | + const xrResult = entity.getValue( |
| 239 | + EnvironmentRaycastTarget, |
| 240 | + 'xrHitTestResult', |
| 241 | + ); |
| 242 | + |
| 243 | + // inputSource is set when user is touching the screen |
| 244 | + if (inputSource && xrResult) { |
| 245 | + // Place object at touch point |
| 246 | + this.placeObject(entity.object3D.position.clone()); |
| 247 | + } |
| 248 | + }); |
| 249 | + } |
| 250 | +} |
| 251 | +``` |
| 252 | + |
| 253 | +### Gaze-Based Placement |
| 254 | + |
| 255 | +For hands-free placement using head gaze: |
| 256 | + |
| 257 | +```typescript |
| 258 | +const reticle = world.createTransformEntity(reticleMesh); |
| 259 | +reticle.addComponent(EnvironmentRaycastTarget, { |
| 260 | + space: RaycastSpace.Viewer, // Uses head/gaze direction |
| 261 | + maxDistance: 5, |
| 262 | +}); |
| 263 | +``` |
| 264 | + |
| 265 | +### Multiple Raycast Sources |
| 266 | + |
| 267 | +You can have multiple entities with different ray sources: |
| 268 | + |
| 269 | +```typescript |
| 270 | +// Left hand reticle |
| 271 | +const leftReticle = world.createTransformEntity(leftMesh); |
| 272 | +leftReticle.addComponent(EnvironmentRaycastTarget, { |
| 273 | + space: RaycastSpace.Left, |
| 274 | +}); |
| 275 | + |
| 276 | +// Right hand reticle |
| 277 | +const rightReticle = world.createTransformEntity(rightMesh); |
| 278 | +rightReticle.addComponent(EnvironmentRaycastTarget, { |
| 279 | + space: RaycastSpace.Right, |
| 280 | +}); |
| 281 | +``` |
| 282 | + |
| 283 | +## Surface Alignment |
| 284 | + |
| 285 | +The system automatically orients objects to match the surface normal. This means: |
| 286 | + |
| 287 | +- Objects placed on floors stand upright |
| 288 | +- Objects placed on walls align to the wall surface |
| 289 | +- Objects placed on angled surfaces match that angle |
| 290 | + |
| 291 | +The orientation is applied using the hit result's pose, which includes the surface normal direction. |
| 292 | + |
| 293 | +## Troubleshooting |
| 294 | + |
| 295 | +### Common Issues |
| 296 | + |
| 297 | +**Raycast not working:** |
| 298 | + |
| 299 | +- Verify `hitTest: { required: true }` in XR features |
| 300 | +- Verify `environmentRaycast: true` in World features |
| 301 | +- Check that you're in an AR session (not VR) |
| 302 | +- Ensure the device supports WebXR hit-test |
| 303 | + |
| 304 | +**Preview object not visible:** |
| 305 | + |
| 306 | +- The object is hidden when there's no hit - point at a surface |
| 307 | +- Check `maxDistance` isn't too small |
| 308 | +- Verify the object's mesh and material are correct |
| 309 | + |
| 310 | +**Wrong controller:** |
| 311 | + |
| 312 | +- Check the `space` property matches your intended controller |
| 313 | +- For left-handed users, use `RaycastSpace.Left` |
| 314 | + |
| 315 | +**Object not aligned to surface:** |
| 316 | + |
| 317 | +- The system uses the hit result's pose for orientation |
| 318 | +- Check that you're copying both position and quaternion when spawning |
| 319 | + |
| 320 | +### Debug Tips |
| 321 | + |
| 322 | +```typescript |
| 323 | +class DebugSystem extends createSystem({ |
| 324 | + targets: { required: [EnvironmentRaycastTarget] }, |
| 325 | +}) { |
| 326 | + update() { |
| 327 | + this.queries.targets.entities.forEach((entity) => { |
| 328 | + const xrResult = entity.getValue( |
| 329 | + EnvironmentRaycastTarget, |
| 330 | + 'xrHitTestResult', |
| 331 | + ); |
| 332 | + const visible = entity.object3D?.visible; |
| 333 | + |
| 334 | + console.log({ |
| 335 | + hasHit: !!xrResult, |
| 336 | + visible, |
| 337 | + position: entity.object3D?.position, |
| 338 | + }); |
| 339 | + }); |
| 340 | + } |
| 341 | +} |
| 342 | +``` |
| 343 | + |
| 344 | +## Performance Considerations |
| 345 | + |
| 346 | +1. **Limit raycast targets** - Each `EnvironmentRaycastTarget` creates a WebXR hit-test source |
| 347 | +2. **Use appropriate maxDistance** - Shorter distances may perform slightly better |
| 348 | +3. **Clean up unused entities** - Remove raycast components when not needed |
| 349 | + |
| 350 | +## Best Practices |
| 351 | + |
| 352 | +1. Always check for valid hit before placing objects |
| 353 | +2. Provide visual feedback (preview object) before placement |
| 354 | +3. Use appropriate `RaycastSpace` for your input method |
| 355 | +4. Handle the case when no surface is detected (object hidden) |
| 356 | +5. Test on target devices as hit-test capabilities vary |
| 357 | + |
| 358 | +## Example Projects |
| 359 | + |
| 360 | +Check out the complete implementation in the SDK: |
| 361 | + |
| 362 | +- **`examples/environment-raycast`** - AR plant placement with controller-based raycasting |
| 363 | + |
| 364 | +```bash |
| 365 | +cd immersive-web-sdk |
| 366 | +pnpm install |
| 367 | +pnpm run build:tgz |
| 368 | +cd examples/environment-raycast |
| 369 | +npm install |
| 370 | +npm run dev |
| 371 | +``` |
| 372 | + |
| 373 | +## What's Next |
| 374 | + |
| 375 | +You now know how to detect real-world surfaces and place virtual content in AR! Combined with Scene Understanding (Chapter 11), you have powerful tools for building AR experiences. |
| 376 | + |
| 377 | +Consider exploring: |
| 378 | + |
| 379 | +- Combining environment raycast with physics for realistic object placement |
| 380 | +- Using anchors (from Scene Understanding) to persist placed objects |
| 381 | +- Building interactive AR furniture placement apps |
0 commit comments