Matt Caswell e4676e900f Fix probable_prime over large shift
In the probable_prime() function we behave slightly different if the number
of bits we are interested in is <= BN_BITS2 (the num of bits in a BN_ULONG).
As part of the calculation we work out a size_limit as follows:

    size_limit = (((BN_ULONG)1) << bits) - BN_get_word(rnd) - 1;

There is a problem though if bits == BN_BITS2. Shifting by that much causes
undefined behaviour. I did some tests. On my system BN_BITS2 == 64. So I
set bits to 64 and calculated the result of:

    (((BN_ULONG)1) << bits)

I was expecting to get the result 0. I actually got 1! Strangely this...

    (((BN_ULONG)0) << BN_BITS2)

...does equal 0! This means that, on my system at least, size_limit will be
off by 1 when bits == BN_BITS2.

This commit fixes the behaviour so that we always get consistent results.

Reviewed-by: Andy Polyakov <appro@openssl.org>
2015-03-17 13:41:49 +00:00
..
2015-03-13 21:10:13 -07:00
2015-03-16 15:54:19 +00:00
2015-03-17 13:41:49 +00:00
2015-02-13 13:50:36 +00:00
2015-02-24 10:11:36 +01:00
2015-03-05 09:09:57 +00:00
2015-03-12 09:22:56 +00:00
2015-03-12 09:23:42 +00:00
2015-03-05 09:09:57 +00:00
2015-03-15 15:56:24 +00:00
2015-03-15 15:56:24 +00:00
2015-02-06 16:49:17 -05:00
2015-03-17 13:39:53 +00:00
2015-03-05 09:09:57 +00:00
2015-03-12 14:12:17 +00:00
2015-02-06 16:49:17 -05:00
2015-02-24 10:11:36 +01:00
2015-03-16 15:54:19 +00:00
2015-03-16 15:54:19 +00:00
2015-03-05 09:09:57 +00:00
2015-03-12 09:26:14 +00:00
2015-03-17 13:39:53 +00:00
2015-03-05 09:09:57 +00:00
2015-03-12 14:12:17 +00:00
2015-03-16 15:54:19 +00:00
2015-03-12 14:12:17 +00:00
2015-01-23 15:38:41 +01:00
2015-03-05 18:20:06 +01:00
2015-01-27 17:44:12 -05:00
2015-02-09 12:53:36 +00:00
2015-01-22 09:20:09 +00:00
2015-01-23 15:38:41 +01:00
2015-01-23 19:09:01 +01:00
2015-01-22 09:20:09 +00:00
2015-01-22 09:20:09 +00:00
2015-01-22 09:20:09 +00:00
2015-01-23 11:58:26 -05:00
2015-01-22 09:20:09 +00:00
2015-03-12 14:12:17 +00:00
2014-06-04 08:34:18 +02:00
2015-02-06 14:52:40 -05:00
2015-01-22 09:20:09 +00:00