Local voice control without the cloud

How a local voice assistant works

Every voice assistant does four jobs in a row. Cloud assistants do them on a company’s servers; a local one does them on a computer at home.

1. Wake word. A small device listens for a phrase and wakes up only when it hears it. This detection runs on the device itself.
2. Speech-to-text. Your voice becomes text.
3. Intent. Software decides what the sentence means: “turn off the kitchen lights.”
4. Text-to-speech. A spoken answer comes back.

Home Assistant calls its version Assist. Its documentation says Assist can run fully on your own hardware, so your voice commands stay private, and that it can also use a large language model for more conversational requests if you choose. That second option is separate: if you connect a cloud model, your text goes to that provider, so read its privacy terms or stay with the local setup.

What hardware you need

Home Assistant’s page lists purpose-built voice hardware from the project, custom satellites built with ESPHome on inexpensive ESP32 microcontrollers and an experimental option that runs on any x64 or ARM64 device able to handle the audio. Compatibility changes, so read the page for your Home Assistant version.

Speed: the number that decides whether you will like it

The documentation gives timings that matter more than any spec sheet. On a Raspberry Pi 4, the Whisper speech engine takes around 8 seconds to process a voice command, while on an Intel NUC-class mini PC it finishes in under a second. A lighter engine called Speech-to-Phrase transcribes in under a second even on a Raspberry Pi 4 or the project’s own preinstalled hub, with a smaller vocabulary. Piper, the text-to-speech engine, generates about 1.6 seconds of voice per second on a Raspberry Pi with medium-quality models.

In practice, an 8-second delay feels broken. If your server is small, use Speech-to-Phrase for quick device commands, or move to a stronger mini PC if you want free-form speech. If you use Home Assistant’s paid cloud, it includes speech-to-text and text-to-speech processing, which is faster but not local.

What a local assistant will not do well

• It understands fewer phrasings. Home Assistant says Assist may not understand some sentences, and you can add custom sentences. Expect to adjust names: “living room lamp” works if that is what you called it.
• Language quality varies. Support differs by language and by speech engine, and the project says it is still a work in progress. Check your language in the engine’s list before buying anything.
• No built-in knowledge. There is no web search, shopping list sync with a retailer, music service or news from a cloud, unless you add integrations.
• Quality depends on microphones. A room with a TV on or a kitchen fan is harder than a quiet bedroom. A microphone array helps more than a bigger server.

Privacy: what stays home and what does not

In a fully local setup, audio is processed at home. Check these points to be sure:

1. Choose local speech-to-text and text-to-speech in your Assist settings, not the cloud versions.
2. Use a satellite with a physical mute switch, and leave it muted when you want certainty.
3. Prefer a satellite that detects the wake word on the device itself, so audio travels to your server only after the wake word; check the satellite’s documentation.
4. Do not connect a cloud conversation agent unless you intend to.
5. Put the server and satellites on your home network behind your router; follow the FTC’s basics of unique passwords and updates.

Compare this with a typical cloud smart speaker: see Smart speaker privacy settings for the controls you do have if you stay with one.

A sensible way to start

1. Start with the phone: use Assist from the companion app to test your device names and sentences.
2. Pick the lighter engine first and time your commands. If it is slow, upgrade the server before buying satellites.
3. Add one satellite in the room you use most.
4. Keep your existing speaker for music and timers until the local one handles those jobs.

The full parts list and the rest of the system are in the Home Assistant starter kit and the local-only smart home kit. If you have not yet installed Home Assistant, begin with Is Home Assistant worth it?.