In the probable_prime() function we behave slightly different if the number
of bits we are interested in is <= BN_BITS2 (the num of bits in a BN_ULONG).
As part of the calculation we work out a size_limit as follows:
size_limit = (((BN_ULONG)1) << bits) - BN_get_word(rnd) - 1;
There is a problem though if bits == BN_BITS2. Shifting by that much causes
undefined behaviour. I did some tests. On my system BN_BITS2 == 64. So I
set bits to 64 and calculated the result of:
(((BN_ULONG)1) << bits)
I was expecting to get the result 0. I actually got 1! Strangely this...
(((BN_ULONG)0) << BN_BITS2)
...does equal 0! This means that, on my system at least, size_limit will be
off by 1 when bits == BN_BITS2.
This commit fixes the behaviour so that we always get consistent results.
Reviewed-by: Andy Polyakov <appro@openssl.org>
e4676e900f