Ah...Yeah, this is definitely worse for recovering folks...I'd mainly thought of this from my perspective as a sober person, where it's a different sort of difficult. I've gotten the half-joking offers to try this or that & simply turn them down, but then sometimes it seems to make things a little awkward for those indulging when someone isn't and then I'm like, "...Er, right, yeah I guess why am I here if this is all we're doing (except me)"
I know there are groups/events I could join or attend that might connect me with people that are also more into sobriety or more mindful of their usage, but I haven't really found any that capture my interest yet.