vpx/test
James Zern 984734436d Fix variance (signed integer) overflow
In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.

To fix this we force the multiplication to be unsigned.

The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.

For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).

This change is based on:
1698234 Missed some variance casts
fea3556 Fix variance overflow

Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-06 23:06:44 -08:00
..
acm_random.h Restyle code 2012-07-17 11:46:03 -07:00
boolcoder_test.cc fix test builds 2012-11-06 12:12:58 -08:00
dct16x16_test.cc fix test builds 2012-11-06 12:12:58 -08:00
fdct4x4_test.cc fix test builds 2012-11-06 12:12:58 -08:00
fdct8x8_test.cc fix test builds 2012-11-06 12:12:58 -08:00
idct8x8_test.cc fix test builds 2012-11-06 12:12:58 -08:00
test_libvpx.cc Restyle code 2012-07-17 11:46:03 -07:00
test-data.sha1 Port Yaowu's 4x4 fDCT test. 2012-06-28 15:48:46 -07:00
test.mk Fix variance (signed integer) overflow 2012-11-06 23:06:44 -08:00
variance_test.cc Fix variance (signed integer) overflow 2012-11-06 23:06:44 -08:00