Have a look at alpine.
For the DE, something very light is needed and I would use Wayland for security AND performance.
Problem: apart from RaspberryPiOS I know no dedicated lightweight wayland DE.
They use Wayfire, but one of these will work too.
Then for the apps, good luck running a Browser at that low.
You will need only system packages, nothing else. Might try Bubblejail for sandboxing without using Flatpak (disk space, RAM). But that is in pretty early stages.
For your apps
- you mean Mullvad Browser not Mull. Screw that, use Librewolf
- you will not run a VM on that hardware. These are VM guest specs, not host. You can run Carburetor flatpak, or maybe a minimalist podman container with tor for proxying. User namespaces, bubblejail and seccomp are also secure.
- VLC is not small. Use Celluloid or just MPV or even better just ffplay. Celluloid/Haruna/Dragon is minimal and has wayland support
- rustdesk? Client or server? There is wayVNC and KDE and GNOME have their suites. But they need static IPs. Rustdesk Server has no wayland support
- deluge, ktorrent, qbittorrent doesnt matter, all light. But stick to one GUI toolkit.
I think Qt can work, pcmanfm-qt is nice.
LXQt 6.1 will have "full" Wayland support, but you need to configure stuff in config files of course.
I dont know a modern Wayland ready GTK alternative to GNOME.