Does moving webAuthn API from browser to OS improves security of registration process?
Usually, for all types of authentications, we trust the registration process and assume there is no attack is happening Like in the case of FIDO2 registration. However, as the registration process is built within the browser and can be compromised by a chrome extension it is an unrealistic assumption. Google's research shows that 1 out of 10 extensions that they publish is malicious and passes their filters. So, it's pretty straightforward for them to compromise the registration process and get the capability to access the user account for a longer duration from any machine. The attacker is able to do this because the registration process is built within the browser.
I think if we move the FIDO2 registration process in OS, we lessen the attack vector and the browser-related attacks cannot compromise a registration process, which makes the assumption of Trust during registration realistic. Is there any security or usability issue if we move the registration process to OS? The website initiates a call to a standalone application which communicates to the webserver and FIDO2 authenticator directly in the background and returns the session key after a successful registration.
I don't think that there is a security advantage and I'm not even clear how to move this process into the OS kernel in the first place. The kernel is about protecting local resources from each other, like making sure that one process cannot access memory from other processes. It has no idea about user space logic.
That does not mean though, that one cannot make use of the kernel features for better user space security. This is actually done with techniques like privilege separation, where parts of the application with different security requirements are separated from each other through different processes, sandboxes, users, containers etc. For example one process might process untrusted user input while not allowing any local file access, while another can do local file access but will not directly process user input. Both processes can then interact with each other through some clearly defined minimal API.
On top of that in case of WebAuthn, Smartcards etc yet another separation gets added: secret keys are kept away from the normal processing on a different hardware and one can only do cryptographic operations with these keys (like signing) but not have access to the secrets itself, neither from a user process nor from the kernel.