While in principle, I don't disagree. If you're impaired, you shouldn't drive. I lost a parent after they were hit by a drunk driver.
However, there are monstrously different amounts of impairment. You have reaction times and motor skills, decision making and judgement, awareness and attention.
Implying any type of impairment to be an unequivocal "no" to driving is short sighted, in my opinion. It's the easy argument to point at any mind-altering substance: caffeine, tobacco, or antidepressants could be classified an impaired driver.
It's also worth pointing out that even different emotions could dramatically alter driving performance. Not that we would ever think about restrictions on crying while driving.
I would actually love to find a nice big list of examples of this. Mostly because I want to know about stuff I've taken as truth