What happens if one certificate path is valid while the other one is not?
I have recently figured out that
Lets Encryptprovides "invalid" certificate chain in it's certificates that are pointing to the root certificate that has expired a month ago (they did it apparently to support old android devices that do not fully validate the root certificate validity).
Certificates obviously work for most of the use cases though because there is a newer CA root certificate that OS/browsers do trust. That means that we can see 2 certification paths like on this image:
What I am wondering is what is the algorithm/protocol to figure out if one can trust such certificates where one path is trusted, while the other one is not (there probably can be multiple ones as well). Is it enough that at least one is trusted, or maybe the shortest?
There is no universal algorithm because there is no such standard. Every certificate chaining engine (CCE) implementation uses its own algorithm to select the best chain.
However, most CCE implementations exclude invalid chains if at least one valid chain is found.
If multiple valid chains found, then CCE uses its own algorithm to select the best one. One can decide shortest, other can select with longest validity. There are multiple variables with different weights which can affect the resulting chain.