Selfhosted

Honestly, don't bother with a dGPU and get a 12th or 13th gen Intel Core chip with QSV. Intel quietly tuned it up to the point where it's faster than nVidia's NVENC engine even in the latest gen plus you don't have mess around with the uncap streams hack and you're transcoding through system RAM not dGPU RAM, so far less likely that your stream limit will be artificially constrained by memory limitations.

To answer the question you asked though, the nVidia NVENC is the best solution on a dGPU. It's performance is largely the same across the same board generation, with one exception in the GTX 10X0 series. The absolute cheapest card you can lay your hands on that has an NVENC engine is the 1050TI.

The caveat is the 1070 and 1080 have two NVENC engines. It will double max number of streams in theory, however in reality you're memory bound on those cards and it's more like a 33% bump.

I know someone else said this but I would just get an i5 with a iGPU, I also have an 8000 series i5 and when it was new I did some load testing and it could do 5-10 transcodes at a time no problem.

ETA: Even a NUC with an iGPU is great, I have several friends doing that.

I have a 1660 from my old machine and it seems to handle transcoding multiple 4K streams easily. Pretty low power draw as well, I doubt there’s much need for anything more powerful unless your files are in av1 or something.

There is also an other approach: encode your media a priori into a format that you can play direct, and then you don't have to worry about transcoding performance. The advantage of this is that you can likely get better quality encodes.

I've used as little as a virtual machine running on my gaming computer to a full blown dedicated server. Really for transcoding as long as it has some cpu cores you can have a good dedicated older machine running plex fine. You don't need a gpu at all for a handful of streams. I think my max concurrent was 5 ever and I only upgraded for maintainability purposes.

I just got a used NUC and it works well and takes up little space.

I have it on my Synology NAS as well and I just let it preencode everything so that the stream doesn't need transcoding when clients request it. It consumes more space but the convenience is more important to me.

While I agree with everyone that Quicksync is the best option, a Quadro P400 is also a viable option. You can get one off eBay for around $40. It's only 2 gigs of RAM, so you're limited to 2 or 3 transcodes at a time. But it's cheap and may even work in your Synology depending on what model you have.