EC_POINT_is_on_curve does not return a boolean

The function EC_POINT_is_on_curve does not return a boolean value.
It returns 1 if the point is on the curve, 0 if it is not, and -1
on error. Many usages within OpenSSL were incorrectly using this
function and therefore not correctly handling error conditions.

With thanks to the Open Crypto Audit Project for reporting this issue.

Reviewed-by: Kurt Roeckx <kurt@openssl.org>
(cherry picked from commit 68886be7e2)

crypto/ec/ec_lib.c

@@ -970,6 +970,13 @@ int EC_POINT_is_at_infinity(const EC_GROUP *group, const EC_POINT *point)
     return group->meth->is_at_infinity(group, point);
 }
 
+/*
+ * Check whether an EC_POINT is on the curve or not. Note that the return
+ * value for this function should NOT be treated as a boolean. Return values:
+ *  1: The point is on the curve
+ *  0: The point is not on the curve
+ * -1: An error occurred
+ */
 int EC_POINT_is_on_curve(const EC_GROUP *group, const EC_POINT *point,
                          BN_CTX *ctx)
 {