I think you are confusing valid concern with valid point. None of the information in the "No" pamphlet is based on fact, only conjecture based on possibilities.
The concerns about the voice to parliament made by the "No" campaign is based on what could happen if we implemented a voice poorly, not what will happen if we added a few lines to the constitution making a voice mandatory.
While it's splitting hairs I believe that the No campaigns arguments don't really amount to much beyond scare mongering. The question asked (as I understand it) is should a voice to parliament be made a mandatory part of our constitution. With the government of the day deciding its scope.
The dreamer in me wants to think we can solve the problems of Australia without making mandatory what should be a given. The realist in me understands that sometimes you have to spell it out to stop those in power forgetting.
I would like to think the intent of the referendum is to show the majority of Australia wants to listen to what Indigenous Australians have to say on matters that concerns them. Flexibility in how that happens is key. By enshrining the voice in our constitution we are saying, "I'll listen even if I don't like what is said because it's important." With enough leeway that governments can change how they listen but not silence the voice altogether.
I'd be happy for additional referendums focused on the specifics but I'd also be fine with the elected government choosing what the voice of the day looks like.
Nothing will change overnight and I agree with the sentiment that we can make things better without a voice.
That said, if someone is hurting, you ask them how you can help. So it makes perfect sense to me to listen to what 250 Aboriginal and Torres Strait Islander Leaders have asked for and put the voice in the constitution.