This only happens when both network connection on the host are active.
I'm not a networking expert by any means but this seems like a pretty strong hint that it's a routing issue.
Check the routing tables on the host? I'd bet that the internet is only reachable on the LAN interface (again, not an expert but one of them has to take priority, right?). I'm guessing that disconnecting the LAN interface changes the routing to go through the WLAN interface instead.
You could possibly add a static route to work around this: https://libvirt.org/formatnetwork.html#static-routes