vpx/vp9
James Zern 984734436d Fix variance (signed integer) overflow
In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.

To fix this we force the multiplication to be unsigned.

The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.

For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).

This change is based on:
1698234 Missed some variance casts
fea3556 Fix variance overflow

Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-06 23:06:44 -08:00
..
common Fix variance (signed integer) overflow 2012-11-06 23:06:44 -08:00
decoder silent a lot of MSVC compiler warnings 2012-11-06 09:09:25 -08:00
encoder Fix variance (signed integer) overflow 2012-11-06 23:06:44 -08:00
exports_dec Rename vp8/ codec directory to vp9/. 2012-11-01 16:31:22 -07:00
exports_enc Rename vp8/ codec directory to vp9/. 2012-11-01 16:31:22 -07:00
vp9_common.mk group refmv experiment related functions 2012-11-06 15:54:47 -08:00
vp9_cx_iface.c silent a lot of MSVC compiler warnings 2012-11-06 09:09:25 -08:00
vp9_dx_iface.c vpx_scale: sync from master 2012-11-02 08:44:54 -07:00
vp9cx_arm.mk Rename vp8/ codec directory to vp9/. 2012-11-01 16:31:22 -07:00
vp9cx.mk Rename vp8/ codec directory to vp9/. 2012-11-01 16:31:22 -07:00
vp9dx_arm.mk Rename vp8/ codec directory to vp9/. 2012-11-01 16:31:22 -07:00
vp9dx.mk Rename vp8/ codec directory to vp9/. 2012-11-01 16:31:22 -07:00