RT992: RSA_check_key should have a callback arg

The original RT request included a patch.  By the time
we got around to doing it, however, the callback scheme
had changed. So I wrote a new function RSA_check_key_ex()
that uses the BN_GENCB callback.  But thanks very much
to Vinet Sharma <vineet.sharma@gmail.com> for the
initial implementation.

Reviewed-by: Dr. Stephen Henson <steve@openssl.org>

crypto/rsa/rsa.h

@@ -325,6 +325,7 @@ int RSA_X931_derive_ex(RSA *rsa, BIGNUM *p1, BIGNUM *p2, BIGNUM *q1, BIGNUM *q2,
 int RSA_X931_generate_key_ex(RSA *rsa, int bits, const BIGNUM *e, BN_GENCB *cb);
 
 int	RSA_check_key(const RSA *);
+int	RSA_check_key_ex(const RSA *, BN_GENCB *cb);
 	/* next 4 return -1 on error */
 int	RSA_public_encrypt(int flen, const unsigned char *from,
 		unsigned char *to, RSA *rsa,int padding);

crypto/rsa/rsa_chk.c

@@ -54,6 +54,11 @@
 
 
 int RSA_check_key(const RSA *key)
+	{
+	return RSA_check_key_ex(key, NULL);
+	}
+
+int RSA_check_key_ex(const RSA *key, BN_GENCB *cb)
 	{
 	BIGNUM *i, *j, *k, *l, *m;
 	BN_CTX *ctx;
@@ -81,7 +86,7 @@ int RSA_check_key(const RSA *key)
 		}
 	
 	/* p prime? */
-	r = BN_is_prime_ex(key->p, BN_prime_checks, NULL, NULL);
+	r = BN_is_prime_ex(key->p, BN_prime_checks, NULL, cb);
 	if (r != 1)
 		{
 		ret = r;
@@ -91,7 +96,7 @@ int RSA_check_key(const RSA *key)
 		}
 	
 	/* q prime? */
-	r = BN_is_prime_ex(key->q, BN_prime_checks, NULL, NULL);
+	r = BN_is_prime_ex(key->q, BN_prime_checks, NULL, cb);
 	if (r != 1)
 		{
 		ret = r;

doc/crypto/BN_generate_prime.pod

@@ -104,10 +104,10 @@ programs should prefer the "new" style, whilst the "old" style is provided
 for backwards compatibility purposes.
 
 For "new" style callbacks a BN_GENCB structure should be initialised with a
-call to BN_GENCB_set, where B<gencb> is a B<BN_GENCB *>, B<callback> is of
+call to BN_GENCB_set(), where B<gencb> is a B<BN_GENCB *>, B<callback> is of
 type B<int (*callback)(int, int, BN_GENCB *)> and B<cb_arg> is a B<void *>.
 "Old" style callbacks are the same except they are initialised with a call
-to BN_GENCB_set_old and B<callback> is of type
+to BN_GENCB_set_old() and B<callback> is of type
 B<void (*callback)(int, int, void *)>.
 
 A callback is invoked through a call to B<BN_GENCB_call>. This will check

doc/crypto/RSA_check_key.pod

@@ -8,35 +8,42 @@ RSA_check_key - validate private RSA keys
 
  #include <openssl/rsa.h>
 
+ int RSA_check_key_ex(RSA *rsa, BN_GENCB *cb);
+
  int RSA_check_key(RSA *rsa);
 
 =head1 DESCRIPTION
 
-This function validates RSA keys. It checks that B<p> and B<q> are
+RSA_check_key_ex() function validates RSA keys.
+It checks that B<p> and B<q> are
 in fact prime, and that B<n = p*q>.
 
+It does not work on RSA public keys that have only the modulus
+and public exponent elements populated.
 It also checks that B<d*e = 1 mod (p-1*q-1)>,
 and that B<dmp1>, B<dmq1> and B<iqmp> are set correctly or are B<NULL>.
+It performs integrity checks on all
+the RSA key material, so the RSA key structure must contain all the private
+key data too.
+Therefore, it cannot be used with any arbitrary RSA key object,
+even if it is otherwise fit for regular RSA operation.
 
-As such, this function can not be used with any arbitrary RSA key object,
-even if it is otherwise fit for regular RSA operation. See B<NOTES> for more
-information.
+The B<cb> parameter is a callback that will be invoked in the same
+manner as L<BN_is_prime_ex(3)|BN_is_prime_ex(3)>.
+
+RSA_check_key() is equivalent to RSA_check_key_ex() with a NULL B<cb>.
 
 =head1 RETURN VALUE
 
-RSA_check_key() returns 1 if B<rsa> is a valid RSA key, and 0 otherwise.
--1 is returned if an error occurs while checking the key.
+RSA_check_key_ex() and RSA_check_key()
+return 1 if B<rsa> is a valid RSA key, and 0 otherwise.
+They return -1 if an error occurs while checking the key.
 
 If the key is invalid or an error occurred, the reason code can be
 obtained using L<ERR_get_error(3)|ERR_get_error(3)>.
 
 =head1 NOTES
 
-This function does not work on RSA public keys that have only the modulus
-and public exponent elements populated. It performs integrity checks on all
-the RSA key material, so the RSA key structure must contain all the private
-key data too.
-
 Unlike most other RSA functions, this function does B<not> work
 transparently with any underlying ENGINE implementation because it uses the
 key data in the RSA structure directly. An ENGINE implementation can
@@ -58,10 +65,13 @@ provide their own verifiers.
 
 =head1 SEE ALSO
 
-L<rsa(3)|rsa(3)>, L<ERR_get_error(3)|ERR_get_error(3)>
+L<BN_is_prime_ex(3)|BN_is_prime_ex(3)>,
+L<rsa(3)|rsa(3)>,
+L<ERR_get_error(3)|ERR_get_error(3)>
 
 =head1 HISTORY
 
 RSA_check_key() appeared in OpenSSL 0.9.4.
+RSA_check_key_ex() appeared after OpenSSL 1.0.2.
 
 =cut