You don't forego any of those things when you cast.
Your remote still works, you're in the same media box UI, still only a click away from other apps.
Maybe slightly imperfect, sure. But it works great and I see no reason to act like it isn't a viable workable option.
What else can you complain about?