Linux

Not sure if Windows has that but I believe on macOS what happens is the app tries to record the screen, and if it fails macOS blocks the request and opens the security settings to enable the permission, and you have restart the whole application for the permission to take.

What's done for Wayland is the portal system: applications can use portals to request access to specific things like screen recording, the DE does what it needs to do and it starts feeding the data to the application through the portal. It's working fairly well, I haven't had issues with those in a while. The application just requests what it wants, and the DE prompts the user (or auto accept the request) optionally remembering the choice as well.

Generally the solution for X11 problems is to implement a modern API for it in either Wayland or as a portal. Which breaks old stuff, but once updated it works fine.

The main obstacle is getting Gnome to agree to the protocols.

I don't know about Wayland or MacOS, but In Windows, you can access quite a lot of information via WPF, UWP, WinUI, etc. This is to allow assistive tech to be able to do what they need to do, such as screen readers.

As long as you know how to search for window and control handles, you can read, store, and digest pretty much everything you as a person can see. No questions or elevation of privileges needed.

The caveat is that you'd have to have local access at a minimum.

I don't know how it works on a technical level, but:

On macOS the app can request permission. In case of screen reading, it can't just ask with a simple allow/deny prompt like with many other permissions (e.g. location), but most app requiring permission usually open the system settings app at the correct page (accessibility > screen readers or something). This page shows a list of all installed applications that specify that they have screen reader capabilities. The user can check a box next to the app's name to allow screen reading.

On Windows, a "classic" win32 application can essentially see anything running under the same user as itself. It can probably capture windows of applications running as another user (administrator), but afaik it can't send keystrokes to them. Appx apps generally have a permission system, but I'm not sure how screen readers are handled.

Others have written about how windows does it, but here's some more details.

A window which runs with higher privileges (even just elevated to admin but still with your same account) cannot be read by normal privileges. You can see this when you use a custom screenshot program with some privileged system utility, but it's key combo does not work when the higher privileged window is active (in the foreground, selected). The screenshot program could not access UI elements in the privileged window, and can't send messages to it, but it can still see it rendered and capture it.

There's also a feature called "secure desktop". This is a bit like opening a new desktop with it's separate "window namespace". It's distinct so much that it doesn't have the taskbar and start menu, and by default it would be blackness, but you don't notice it because the system takes a screenshot before opening it and sets that as background.
Admin utils rarely use this feature, as I know this is only used for the User Account Control window that appears when a program is asking for elevated permissions. This is where you type your password, or just accept or deny the elevation request.
The Keepass password manager can also make use of this feature for the unlock prompt, but it can't use it that effectively, because the new secure desktop can be found in some way by other programs if it was not created with elevated privileges. It writes about this in it's documentation.
Even though Linux nowadays has a password prompt dialog, it does not have anything similar to this secure desktop thing as I know.

Other than that, on windows (maybe linux too?) processes of the same user and privilege level can read each other's memory. Without elevation. It's quite complicated but it's always there.
And like with gdb and strace on linux, there are ways on windows too to analyze or modify at runtime how a process works.

Not an answer to your question, but a (perhas naive) question itself: are keyloggers impossible on Wayland?

Or QubesOS plz