• I know this is usually frowned upon, but supposing you were to roll your own encryption cipher (completely on your own or with a small group of friends, with no peer review possible), what could you do to make your custom algorithm as safe as possible (or rather, to keep the danger of using it to a minimum)? Similarly, what would you do for something like a hash function to make the probability of it having some trivial collision or preimage attack as low as possible?

I understand that there would be no way to approach the security of algorithms like AES, and I have no plans to use custom algorithms for anything important, but I thought it would be informative to think of what some best practices would be for rolling your own cryptosystem if you were to do so.

In general, you may want to start by basing your cipher (mostly relevant for asymmetric ciphers) on a known hard problem. You may want to make mathematical proofs verifying your cipher has the desired properties and breaking it indeed requires solving the known hard problem. What kind of problems to choose was explained more on Crypto SE. By creating these mathematical proofs, you will make sure your cipher is not easily breakable directly.

The next step may be to look at side channels. Most notably timing attacks. If your cipher takes different amount of time for different inputs and keys, you may want to at least add code to either make it take the same amount of time always or just add random amount of time by active waiting.

There are many other side-channels and considerations a widely used cipher has to take into account, but this should get you started. Of course, making all these mathematical proofs is very difficult and even then, it does not give you any guarantee the cipher can't be broken. Even if the cipher is secure on its own, in combination with other cryptographic primitives, it may turn out to be easily breakable. That is why rolling your own crypto is such a big taboo. Getting it right is very very difficult and you will not find out you did something wrong until someone breaks it, or if you are unlucky, not even after that.

2

2

2

2

2

2

2

2

2

2

2

2

2

2

2