In RSA, can any of the two keys be used for both encryption and decryption?
I know RSA has the concept of "public" and "private" key and one key is used for "encryption" and another for "decryption", but I am confused if both keys can be used for both purposes.. I mean, what you encrypt with any of them, you can decrypt of the other.
There are many incorrect wordings/usages here.
RSA is merely the one public-key cryptosystem that can support both encryption and signature naively in the almost same type of operation.
What confuses the people is the Textbook RSA which is simply
c = m^e mod nfor encryption and
m= c^d mod nfor decryption, whereas
sign = hash(m)^d mod nfor signature and
hash(m)and checks that
hash(m) == sign^e mod n.
Both are insecure as you can see that you can multiply the ciphertexts, or .. ( this is a very long story; start from reading Twenty Years of Attacks on the RSA Cryptosystem for more attacks from Dan Boneh).
RSA encryption requires padding to be secure. PKCS#5v1.5 and Optimal Asymmetric Encryption Padding (OAEP) are the paddings that must be used. The former had lots of attacks due to incorrect implementations. The latter is much easier to implement. They are both proved to be secure so the only problem is the implementations.
RSA signature requires padding to be secure, too. This time the padding is a Probabilistic Signature Scheme and designed for signatures.
RSA sign!=RSA decryption
Don't use/say that RSA decryption is a signature. It is not. The signatures algorithm consists of two algorithms;
verify. During the sign operation, PSS be part of it.
For a detailed reading see from Cornell University page; RSA Signing is Not RSA Decryption
Don't use one RSA key for more than one purpose
Don't use the same RSA public-private key for more than one purpose. Use either for enc/dec or sign/verify. If you need both, you two different public/private key pairs where the modulus is distinct.
Public key is not for encryption
Public key cryptosystems are slow compared to private key cryptosystems. Therefore we prefer a hybrid cryptosystem where the public key is used for key exchange and the private key is used for encryption. Examples are;
- RSA-KEM for Key encapsulation Mechanism and AES-GCM for Data Encapsulation Mechanism
- DHKE for key establishment and ChaCha20-Poly1305 for data encapsulation mechanism
Where RSA is used today?
In today's security, RSA is mostly used for the signature. Although RSA-KEM is promising for key encapsulation there is no standard for that, we use Elliptic Curve Diffie-Hellman most of the time.