Case study
Tricorder: AI Agents for Mixed-Reality Medical Simulation
Built an internal multi-agent research prototype at Booz Allen to explore medical triage support in deep-space and austere environments, combining mixed reality, voice, and vision.
Agent-system development · Internal research prototype at Booz Allen Hamilton
I built Tricorder's multi-agent system at Booz Allen in collaboration with two AI researchers and an experienced pediatric surgeon. The concept explored how AI agents might support medical triage for astronauts on deep-space missions and people in austere environments. The Unity/HoloLens prototype used a simulated patient, conversational agents, and visual input to explore that idea through an interactive assessment scenario.
- Ownership: The custom agent framework, interactive headset onboarding, event-driven interactions, and model-provider abstraction.
- Technical scope: Onboard speech recognition, agent function calling, hosted vision inference, and a local text-only model.
- Delivery: An internal research demonstration of simulated assessment conversations, with switchable local and hosted model paths.
Technical Overview
HoloLens recognized speech onboard and passed the resulting text to the agent system. The hosted model path used GPT-4 with vision, receiving headset captures as well as conversational context. A local text-only model also ran as another model option; vision processing remained on the hosted path.
The agent framework coordinated the patient and triage roles, with function calling supporting the simulated assessment workflow. Model-generated interpretations and follow-up prompts returned through spoken playback and the spatial interface.
Simplified prototype architecture. The dashed medical-reference retrieval branch represents work in progress.
How the Project Started
Conversations with NASA flight surgeons helped spark the idea for Tricorder. The question was how AI agents might support medical triage when access to specialist care is limited, such as during a deep-space mission or in an austere environment.
Tricorder explored that question through simulation. A participant wearing HoloLens could observe a virtual patient, ask questions, and relay observations to an agent. The prototype let us explore how a participant could gather information and how the system responded as the conversation developed.
Building the Multi-Agent System
I developed a custom Unity framework around transcription events, agent responses, and changes to the simulation's assessment state. Agent creation used a factory pattern, with provider selection separated from each agent's role and behavior.
From inside the headset, a participant could switch between a local endpoint backed by a text-only model running in a Docker container and a remote Azure instance serving GPT-4. The purpose was to evaluate medical question answering across local and frontier models through the same application.
The patient agent described symptoms without naming the underlying condition. The triage agent received the participant's observations and produced follow-up questions and candidate interpretations. Keeping those roles separate created a conversation in which the participant had to gather and communicate information.
I used agent function calling to support that simulated assessment process. The framework connected model interaction to application behavior, while event-driven updates kept the conversation and the spatial interface coordinated.
Designing the Headset Experience
I also built interactive onboarding inside the headset to introduce the experience. The Unity application used HoloLens and MRTK for spatial interaction and device input, bringing the participant's speech, visual observations, and agent responses into one interface.
The simulated patient was a static photogrammetry scan. In one vision experiment, the patient's texture was modified to depict a rash. A mixed-reality capture containing the rendered patient was then sent to the hosted vision model.
The interaction followed a short loop:
- Observe: View the virtual patient and gather information from the patient agent.
- Describe: Speak observations; onboard recognition converts speech to text for the agent system.
- Add visual context: Supply a headset capture to the hosted vision model.
- Respond and continue: Review the triage agent's candidate interpretations and follow-up prompts in audio and the spatial interface.
Design and Engineering Choices
| Choice | Purpose |
|---|---|
| Separate patient and triage agents | Give the simulation distinct symptom-reporting and assessment roles. |
| Connect speech events and function calling to the application | Coordinate the agent interaction with the simulated assessment workflow. |
| Expose endpoint switching inside the headset | Support medical question-answering evaluation across a Docker-hosted local text model and GPT-4 on Azure. |
| Use a virtual patient and headset capture | Combine spoken observations and visual context in a repeatable scenario. |
Medical-Reference Retrieval in Development
The work also included developing embeddings and retrieval for medical reference resources. The goal was to ground responses in relevant material and reduce hallucinations. The retrieval pipeline was still in development, and fully offline packaging remained future work.
What the Prototype Established
Tricorder brought multi-agent interaction, voice, vision, and mixed reality into an internal demonstration. My contribution was translating the research team's ideas into an interactive system for exploring and evaluating simulated assessment conversations. Clinical validation remained outside the prototype's scope.