Skip to content

Commit 4cd9ab4

Browse files
felixtrzmeta-codesync[bot]
authored andcommitted
feat(examples): add environment-raycast example demonstrating AR hit-test
Summary: - Add new environment-raycast example for AR hit-testing - Demonstrate EnvironmentRaycastTarget component for continuous raycasting - Show plant placement on controller trigger press using hit result position/orientation - Include plant 3D model asset (plantSansevieria GLTF with textures) - Configure AR session with hitTest: { required: true } and environmentRaycast: true - Add welcome panel UI with instructions Reviewed By: zjm-meta Differential Revision: D88604735 Privacy Context Container: L1334777 fbshipit-source-id: b7318b5cfa7a7b77e5eb8e7b711e7b80e41952da
1 parent 4a4e1c7 commit 4cd9ab4

15 files changed

Lines changed: 1086 additions & 0 deletions
Lines changed: 381 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,381 @@
1+
---
2+
outline: [2, 4]
3+
---
4+
5+
# Chapter 14: Environment Raycast
6+
7+
The IWSDK provides an environment raycast system that enables AR applications to detect real-world surfaces and place virtual content on them. This chapter covers implementing hit-testing in your WebXR applications.
8+
9+
## What You'll Build
10+
11+
By the end of this chapter, you'll be able to:
12+
13+
- Set up environment raycasting for AR hit-testing
14+
- Create objects that follow controller raycasts to real-world surfaces
15+
- Place virtual content on detected surfaces with a trigger press
16+
- Configure different ray sources (controllers, gaze, screen touch)
17+
- Align objects to surface normals automatically
18+
19+
## Overview
20+
21+
The environment raycast system leverages WebXR's hit-test API to cast rays against real-world geometry and position virtual objects at hit points. Unlike Scene Understanding (Chapter 11), environment raycasting **does not require room scanning** - it works immediately using real-time surface detection.
22+
23+
### Environment Raycast vs Scene Understanding
24+
25+
| Feature | Environment Raycast | Scene Understanding |
26+
| ------------------------- | ------------------- | ------------------- |
27+
| Room scanning required | No | Yes |
28+
| Works immediately | Yes | After scanning |
29+
| Provides surface position | Yes | Yes |
30+
| Provides surface normal | Yes | Yes |
31+
| Detects planes/meshes | No | Yes |
32+
| Semantic labels | No | Yes |
33+
| Best for | Instant placement | Rich scene data |
34+
35+
**Use Environment Raycast when** you want immediate tap-to-place or controller-based placement without waiting for room scanning.
36+
37+
**Use Scene Understanding when** you need detailed information about detected surfaces like semantic labels (table, wall, floor) or mesh geometry.
38+
39+
### Key Components
40+
41+
- **`EnvironmentRaycastSystem`** - Core system that manages WebXR hit-test sources
42+
- **`EnvironmentRaycastTarget`** - Component that positions an entity at raycast hit points
43+
- **`RaycastSpace`** - Enum for ray source selection (Left, Right, Viewer, Screen)
44+
45+
## Quick Start
46+
47+
Here's a minimal example to get environment raycasting working:
48+
49+
```typescript
50+
import {
51+
World,
52+
SessionMode,
53+
EnvironmentRaycastTarget,
54+
RaycastSpace,
55+
} from '@iwsdk/core';
56+
57+
World.create(document.getElementById('scene-container'), {
58+
xr: {
59+
sessionMode: SessionMode.ImmersiveAR,
60+
features: {
61+
hitTest: { required: true }, // Enable WebXR hit-test
62+
},
63+
},
64+
features: {
65+
environmentRaycast: true, // Enable EnvironmentRaycastSystem
66+
},
67+
}).then((world) => {
68+
// Create a reticle that follows the raycast
69+
const reticleMesh = createReticleMesh(); // Your reticle geometry
70+
const reticle = world.createTransformEntity(reticleMesh);
71+
72+
reticle.addComponent(EnvironmentRaycastTarget, {
73+
space: RaycastSpace.Right, // Use right controller
74+
maxDistance: 10, // Maximum raycast distance in meters
75+
});
76+
77+
// The reticle now automatically:
78+
// - Moves to where the controller points at real surfaces
79+
// - Orients to match the surface normal
80+
// - Hides when there's no hit
81+
});
82+
```
83+
84+
## System Setup
85+
86+
### Step 1: Enable WebXR Hit-Test Feature
87+
88+
```typescript
89+
World.create(container, {
90+
xr: {
91+
sessionMode: SessionMode.ImmersiveAR,
92+
features: {
93+
hitTest: { required: true }, // Required for environment raycast
94+
},
95+
},
96+
features: {
97+
environmentRaycast: true,
98+
},
99+
});
100+
```
101+
102+
**Important**: The `hitTest` WebXR feature must be enabled for environment raycasting to work.
103+
104+
### Step 2: Create a Raycast Target Entity
105+
106+
```typescript
107+
const previewMesh = createPreviewMesh();
108+
const previewEntity = world.createTransformEntity(previewMesh);
109+
110+
previewEntity.addComponent(EnvironmentRaycastTarget, {
111+
space: RaycastSpace.Right,
112+
maxDistance: 10,
113+
});
114+
```
115+
116+
The entity's Object3D will automatically:
117+
118+
- **Position** at the raycast hit point
119+
- **Orient** to match the surface normal
120+
- **Hide** when there's no hit (visibility set to false)
121+
122+
## Understanding the Components
123+
124+
### EnvironmentRaycastTarget
125+
126+
Makes an entity follow environment raycast hit points.
127+
128+
#### Input Properties
129+
130+
- **`space`** - Ray source: `RaycastSpace.Left`, `Right`, `Viewer`, or `Screen` (default: `Right`)
131+
- **`maxDistance`** - Maximum raycast distance in meters (default: `100`)
132+
- **`offsetPosition`** - Offset from ray origin (default: `undefined`)
133+
- **`offsetQuaternion`** - Rotation offset for ray direction (default: `undefined`)
134+
135+
#### Output Properties (Read-only)
136+
137+
- **`xrHitTestResult`** - The underlying `XRHitTestResult` when there's a hit, `undefined` otherwise
138+
- **`inputSource`** - For Screen mode: the `XRInputSource` that triggered the hit
139+
140+
```typescript
141+
// Check if there's a valid hit
142+
const xrResult = entity.getValue(EnvironmentRaycastTarget, 'xrHitTestResult');
143+
if (xrResult) {
144+
console.log('Hit detected at:', entity.object3D.position);
145+
}
146+
```
147+
148+
### RaycastSpace Options
149+
150+
Choose the ray source based on your use case:
151+
152+
| Space | Description | Best For |
153+
| --------------------- | ----------------------------- | -------------------------- |
154+
| `RaycastSpace.Right` | Right controller's target ray | Quest controller placement |
155+
| `RaycastSpace.Left` | Left controller's target ray | Left-handed users |
156+
| `RaycastSpace.Viewer` | Head/gaze direction | Gaze-based placement |
157+
| `RaycastSpace.Screen` | Screen touch (phone AR) | Tap-to-place on phones |
158+
159+
## Common Patterns
160+
161+
### Preview + Place Pattern
162+
163+
The most common pattern: show a preview that follows the raycast, then spawn a permanent object on trigger press.
164+
165+
```typescript
166+
import {
167+
AssetManager,
168+
createSystem,
169+
EnvironmentRaycastTarget,
170+
RaycastSpace,
171+
} from '@iwsdk/core';
172+
173+
class PlacementSystem extends createSystem({
174+
targets: { required: [EnvironmentRaycastTarget] },
175+
}) {
176+
private previewEntity: Entity | null = null;
177+
178+
init() {
179+
// Create preview object
180+
const previewMesh = AssetManager.getGLTF('myObject').scene.clone();
181+
this.previewEntity = this.world.createTransformEntity(previewMesh);
182+
this.previewEntity.addComponent(EnvironmentRaycastTarget, {
183+
space: RaycastSpace.Right,
184+
maxDistance: 10,
185+
});
186+
}
187+
188+
update() {
189+
const triggerPressed = this.input.gamepads.right?.getSelectStart();
190+
191+
if (triggerPressed && this.previewEntity) {
192+
const xrResult = this.previewEntity.getValue(
193+
EnvironmentRaycastTarget,
194+
'xrHitTestResult',
195+
);
196+
197+
// Only place if we have a valid hit and preview is visible
198+
if (xrResult && this.previewEntity.object3D?.visible) {
199+
this.spawnObject(
200+
this.previewEntity.object3D.position.clone(),
201+
this.previewEntity.object3D.quaternion.clone(),
202+
);
203+
}
204+
}
205+
}
206+
207+
private spawnObject(position: Vector3, quaternion: Quaternion) {
208+
const mesh = AssetManager.getGLTF('myObject').scene.clone();
209+
mesh.position.copy(position);
210+
mesh.quaternion.copy(quaternion);
211+
this.scene.add(mesh);
212+
this.world.createTransformEntity(mesh);
213+
}
214+
}
215+
```
216+
217+
### Phone AR Tap-to-Place
218+
219+
For phone-based AR experiences, use `RaycastSpace.Screen` to detect where the user taps:
220+
221+
```typescript
222+
const reticle = world.createTransformEntity(reticleMesh);
223+
reticle.addComponent(EnvironmentRaycastTarget, {
224+
space: RaycastSpace.Screen, // Tracks screen touch
225+
maxDistance: 10,
226+
});
227+
228+
// In your system, check for the input source
229+
class TapPlaceSystem extends createSystem({
230+
targets: { required: [EnvironmentRaycastTarget] },
231+
}) {
232+
update() {
233+
this.queries.targets.entities.forEach((entity) => {
234+
const inputSource = entity.getValue(
235+
EnvironmentRaycastTarget,
236+
'inputSource',
237+
);
238+
const xrResult = entity.getValue(
239+
EnvironmentRaycastTarget,
240+
'xrHitTestResult',
241+
);
242+
243+
// inputSource is set when user is touching the screen
244+
if (inputSource && xrResult) {
245+
// Place object at touch point
246+
this.placeObject(entity.object3D.position.clone());
247+
}
248+
});
249+
}
250+
}
251+
```
252+
253+
### Gaze-Based Placement
254+
255+
For hands-free placement using head gaze:
256+
257+
```typescript
258+
const reticle = world.createTransformEntity(reticleMesh);
259+
reticle.addComponent(EnvironmentRaycastTarget, {
260+
space: RaycastSpace.Viewer, // Uses head/gaze direction
261+
maxDistance: 5,
262+
});
263+
```
264+
265+
### Multiple Raycast Sources
266+
267+
You can have multiple entities with different ray sources:
268+
269+
```typescript
270+
// Left hand reticle
271+
const leftReticle = world.createTransformEntity(leftMesh);
272+
leftReticle.addComponent(EnvironmentRaycastTarget, {
273+
space: RaycastSpace.Left,
274+
});
275+
276+
// Right hand reticle
277+
const rightReticle = world.createTransformEntity(rightMesh);
278+
rightReticle.addComponent(EnvironmentRaycastTarget, {
279+
space: RaycastSpace.Right,
280+
});
281+
```
282+
283+
## Surface Alignment
284+
285+
The system automatically orients objects to match the surface normal. This means:
286+
287+
- Objects placed on floors stand upright
288+
- Objects placed on walls align to the wall surface
289+
- Objects placed on angled surfaces match that angle
290+
291+
The orientation is applied using the hit result's pose, which includes the surface normal direction.
292+
293+
## Troubleshooting
294+
295+
### Common Issues
296+
297+
**Raycast not working:**
298+
299+
- Verify `hitTest: { required: true }` in XR features
300+
- Verify `environmentRaycast: true` in World features
301+
- Check that you're in an AR session (not VR)
302+
- Ensure the device supports WebXR hit-test
303+
304+
**Preview object not visible:**
305+
306+
- The object is hidden when there's no hit - point at a surface
307+
- Check `maxDistance` isn't too small
308+
- Verify the object's mesh and material are correct
309+
310+
**Wrong controller:**
311+
312+
- Check the `space` property matches your intended controller
313+
- For left-handed users, use `RaycastSpace.Left`
314+
315+
**Object not aligned to surface:**
316+
317+
- The system uses the hit result's pose for orientation
318+
- Check that you're copying both position and quaternion when spawning
319+
320+
### Debug Tips
321+
322+
```typescript
323+
class DebugSystem extends createSystem({
324+
targets: { required: [EnvironmentRaycastTarget] },
325+
}) {
326+
update() {
327+
this.queries.targets.entities.forEach((entity) => {
328+
const xrResult = entity.getValue(
329+
EnvironmentRaycastTarget,
330+
'xrHitTestResult',
331+
);
332+
const visible = entity.object3D?.visible;
333+
334+
console.log({
335+
hasHit: !!xrResult,
336+
visible,
337+
position: entity.object3D?.position,
338+
});
339+
});
340+
}
341+
}
342+
```
343+
344+
## Performance Considerations
345+
346+
1. **Limit raycast targets** - Each `EnvironmentRaycastTarget` creates a WebXR hit-test source
347+
2. **Use appropriate maxDistance** - Shorter distances may perform slightly better
348+
3. **Clean up unused entities** - Remove raycast components when not needed
349+
350+
## Best Practices
351+
352+
1. Always check for valid hit before placing objects
353+
2. Provide visual feedback (preview object) before placement
354+
3. Use appropriate `RaycastSpace` for your input method
355+
4. Handle the case when no surface is detected (object hidden)
356+
5. Test on target devices as hit-test capabilities vary
357+
358+
## Example Projects
359+
360+
Check out the complete implementation in the SDK:
361+
362+
- **`examples/environment-raycast`** - AR plant placement with controller-based raycasting
363+
364+
```bash
365+
cd immersive-web-sdk
366+
pnpm install
367+
pnpm run build:tgz
368+
cd examples/environment-raycast
369+
npm install
370+
npm run dev
371+
```
372+
373+
## What's Next
374+
375+
You now know how to detect real-world surfaces and place virtual content in AR! Combined with Scene Understanding (Chapter 11), you have powerful tools for building AR experiences.
376+
377+
Consider exploring:
378+
379+
- Combining environment raycast with physics for realistic object placement
380+
- Using anchors (from Scene Understanding) to persist placed objects
381+
- Building interactive AR furniture placement apps

0 commit comments

Comments
 (0)