This is strong evidence that Google just assumes that if you have the permission (and presumably network permission too) then of course you must be uploading the user’s contacts somewhere.
It sounds like there are a lot of other issues or potential bad faith with Google's process.
But this is an entirely reasonable stance to take. Merely touching the permission should be the bar to having extremely strong requirements in place to verify that you're not doing anything bad.