984734436d
In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: |
||
---|---|---|
.. | ||
acm_random.h | ||
boolcoder_test.cc | ||
dct16x16_test.cc | ||
fdct4x4_test.cc | ||
fdct8x8_test.cc | ||
idct8x8_test.cc | ||
test_libvpx.cc | ||
test-data.sha1 | ||
test.mk | ||
variance_test.cc |