I totally feel you OP. I have a 500+ GB music collection I refuse to give up for streaming apps (which don't have all that music anyway). But I also acknowledge the vast majority of people install a couple apps, take a couple pics and videos, and that's about it, so I get why 128 is the default.
My solution was to cough up money for a DAP (specifically the Fiio M11S). I know it's outside the budget of most people and definitely a super niche purchase, but boy do I feel relieved knowing I can plug my 1 TB SD card in in it and have all my music whenever I want to (plus other neat features like an actual audio jack and no audio resampling). And it also means I'm no longer restricted in my phone choices which is wonderful.
Maybe it's something for you to consider over a NAS or other local server solutions that require more upkeep.
For Android DAPs you definitely want to look into the 500€+ range simply because cheaper ones use very low end SoCs (usually the snapdragon 4xx series) which is why you had that experience with the Sony one. Only look at devices with a snapdragon 6xx and above to be honest. Also note most will never get an OS update so what you get is what you will have forever.
I also tried a cheaper DAP at first too (the Hiby R5 I believe) but although the audio was fine the laggy app experience left a lot to be desired, so I'm very happy I paid an extra 200 to have a DAP that's not slow/laggy when navigating the interface.
For future reference you can bookmark this page that gets updated with newer DAP releases every once in a while.