Youtube: link

This is my submission for the AI Interactive Program – Track: Scope.

Project Overview / Goals

The goal of this project is to explore how AI generation can be integrated into interactive audiovisual experiences—not as a separate effect, but organically woven into the 3D environment, gameplay, and narrative.

The Ninth Door is a 3D puzzle-style game built with Three.js and Scope, where progression unlocks mechanics that gradually alter how the space is perceived. Through player interaction, different prompts and VACE parameters are modified in real time, causing the environment to be reimagined rather than simply changed. The same corridor is experienced through multiple perceptual layers.  

GITHUB: https://github.com/jgoyret/the-ninth-door-threejscope

Technical Architecture

Core Engine: React Three Fiber (Three.js)

AI Visuals:  R3F Canvas → WebRTC stream → Scope API → WebRTC return stream → HTML video element

Real-time Updates:  Prompt and VACE parameter changes sent via WebRTC data channel, triggered by player interactions

State Management:  Zustand stores for door sequence, canvas visibility and connection status

Video with Technical explication of mechanics and scope parameters update  (sorry for my english): 

The Three Canvas System

The game manages three canvases simultaneously:

1. Three.js Canvas – The main game render: corridor, doors, orbs, player

2. Depth Canvas – A grayscale render representing scene depth (closer objects = lighter, farther = darker) 

3. AI Output Canvas – The video stream returned by Scope

How the Depth Canvas is Generated?

The depth canvas uses Three.js [DepthTexture]  to capture the Z-buffer of the scene. A post-process shader converts these depth values to a grayscale image where closer objects appear lighter and farther objects appear darker. The result is copied to a standard 2D canvas that serves as the WebRTC stream source.

At any moment, only one canvas is visible to the player, but all three are always active. The game dynamically decides:

- Which canvas to show the player (`visibleCanvas`)

- Which canvas to send to Scope (`streamSource`)

- What VACE parameters to use

Game Concept

The Ninth Door is a first-person puzzle game set inside a corridor with nine doors. Each door can only be opened by following a specific sequence. When a door is opened, the player absorbs an energy visualized through an orb, and their perception of reality shifts.

As the player advances through these perceptual states, the corridor is continuously reimagined by Scope in real time. The experience culminates behind the ninth door, where the player enters a dream state and gains the ability to actively guide where the dream goes.   

Mechanics

The Absorption Flow (Doors 1-8)

Each door follows this technical flow:

1. Door Opens → A prompt specific to that door is sent to Scope and vace scale is set to 0.45

2. Orb Delivery → Player carries the orb to the Orb collector.

 On delivery: `vace_ref_images` is sent (the orb screenshot image) and Visible canvas switches from Three.js to AI Output Canvas and stream source is Depth Canvas.

3.Absorption → Player sees Scope transform the depth scene based on the door's prompt

4.Exit → Player presses E to exit:-`vace_context_scale` is updated (0.8) to make transition smoother and 5 second blend transitions back to Three.js.

Door Sequence

The doors cannot be opened arbitrarily. Progression follows a specific sequence derived from vortex-based mathematics: **1–5–7–8–4–2–3–6–9**

This organizes progression as a cyclical flow rather than a linear path, with 3 and 6 acting as opposing poles, and 9 functioning as a threshold where the rules change.

Entering the Dream

Door 9 works differently. Instead of an absorption that ends and returns you to the corridor, it takes you to chose what do you want to dream guiding the AI with your own prompts.

When door 9 opens:

- The visible canvas switches to Depth (not AI Output)

- Stream source is Depth

- A specific prompt is sent with higher `vace_context_scale`

When the player looks toward the end of the corridor, they trigger "Enter the Dream"—the AI Output canvas becomes visible, the Three.js corridor disappears, and the player is fully immersed in Scope's vision.

Guiding the Dream

After 10 seconds in the dream, a UI appears allowing the player to write their own prompts.

The player can:

- Describe where they want the dream to go

- Adjust the **intensity** (`vace_context_scale`) with a slider from 0 to 1

VACE Usage

- vace_context_scale: Controls how much the visual input influences the output. Lower values (0.4) give the AI more creative freedom; higher values (0.8-1.0) maintain more fidelity to the original structure.

- vace_ref_images: Reference images that influence the output aesthetic. Used here for the orb during absorption.

- Depth as input: Provides spatial structure without visual details, enabling dramatic reinterpretation while keeping spatial coherence.  

Setup

• Web deployment in Vercel
• RunPod for running Scope

Notes

- The main challenge is the delay between Three.js input and Scope output. Some latency is unavoidable, but the absorption mechanic works around this—the transition moments give Scope time to process while feeling intentional to the player.

- The game runs at 1280×720, but Scope output is 512px (16:9), stretched to fill the canvas.

Thank you for reading !