Pointers do make more sense to me now than two decades ago, mostly owing to me being married to a computer scientist. But I always go back the fact that for the purposes of my first year programming course, pointers were (probably) unnecessary and thus confusing. I have a hard time understanding things if not given an immediate and tangible use case, and pointers didn't really help me when most of my programs used a bare few functions and some globally defined variables to solve simple physics problems.

EDIT: I'll also say that pointers alone weren't what sunk my interested in programming, they're just an easily identifiable concept that sticks out as "not making sense." At around the same time we had the lesson on pointers, our programs were also starting to reach a critical mass of complexity, and the amount of mental work I had to do to follow along became more than I was willing to put into it - it wasn't "fun" anymore. I only did well on my final project because a friend patiently sat in my dorm room for a few hours and talked me through each step of the program, and then fed me enough vocabulary to convince the TA that I knew what I was doing.