fea3556e20
In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we cast the result of the multiplication as unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). PPC change is untested. Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265 |
||
---|---|---|
.. | ||
fastquantizeb_neon.asm | ||
picklpf_arm.c | ||
sad8_neon.asm | ||
sad16_neon.asm | ||
shortfdct_neon.asm | ||
subtract_neon.asm | ||
variance_neon.asm | ||
vp8_memcpy_neon.asm | ||
vp8_mse16x16_neon.asm | ||
vp8_shortwalsh4x4_neon.asm | ||
vp8_subpixelvariance8x8_neon.asm | ||
vp8_subpixelvariance16x16_neon.asm | ||
vp8_subpixelvariance16x16s_neon.asm |