Home assistant has a built in voice assistant function that can be as simple or robust as you need it to be. The whole thing can be setup fully locally and mine runs easily on an old micro-pc I got for $100. I had it running on a Pi3b originally but the STT and TTS would take 10+ seconds to process, which was too long.
Out of the box it controls local devices, does to-do lists, controls media, sets timers. Setting reminders doesn't work out of the box, but can be setup with some great community templates. Services that require web content like "tell me the news" or "what's the weather in Seattle" need to be either setup with custom commands that have access to the info you want, or need to go through an LLM.
Luckily, the past few months have seen the open home foundation add integrations for LLM's, both local and web-based (chatgpt, gemini, etc) are possible, so you can have it run queries through models run on a local GPU. Though this is currently fairly bleeding edge and I haven't tried running a local LLM myself yet so I can't speak to it's complexity.
More on that here: https://www.home-assistant.io/blog/2024/06/07/ai-agents-for-the-smart-home/