AI Voice Assistant API
The AI Voice Assistant feature enables seamless integration with OpenAI's Realtime API for low-latency conversational experiences, allowing to stream real-time audio responses from AI. When activated, it prompts users engaged with your 3D experience to grant microphone access, enabling them to communicate directly with the AI via voice and receive spoken AI responses.
Paired with the AI conversational functionality, you can influence your 3D scene with AI by using variables like the Voice Spectrum or using the
Creating a new AI Voice Assistant API
Navigate to the Variables & Data Panel on the right sidebar of the Spline editor (accessible when nothing is selected in the editor).
Click on the “+” icon in the top right corner.
Click on AI Voice Assistant.
Configuring an AI Voice Assistant
API Key: Enter your OpenAI API key. Learn how to get your API key.
Note: To use OpenAI’s API, you’ll need to set up a paid OpenAI account and choose a pay-as-you-go pricing plan or subscribe to one of the monthly packages.
Voice: Choose the AI voice.
Instructions
You can define as many guidelines as you’d like for the AI to follow. By default, a set of instructions will already be provided, but you can either modify or replace them entirely to suit your needs.
This approach unlocks a wide range of possibilities. For instance, you can use a variable change event to trigger specific actions when the AI updates certain variables, allowing for dynamic responses tailored to various contexts.
Personality
In the personality section, you can define the AI’s initial character and influence how it expresses itself and communicates. For example, you could instruct it to be upbeat and genuine, or convey confusion and nervousness, depending on your needs.
Events
In the events section, you can create new AI Assistant Trigger events. Each event includes a description, and when a user request matches that description, the AI will recognize it and execute the associated actions.
Learn more about the
Note: All written instructions (Instructions, Personality and Trigger descriptions) support non-English languages. You can also speak with the AI in different languages and will respond in the same language. Only languages supported by OpenAI.
AI Voice Spectrum Variable
The AI Voice Assistant API also includes a voice spectrum variable, which in simple terms means the sound of the voice. This value can be linked to other variables and consequently to object properties to manipulate 3D objects in your scene.
To achieve this, you can use the
Pairing Voice Spectrum with Morph Targets
When importing an object with
Learn more about related topics: