vpx/vp8/encoder/arm/neon
Johann fea3556e20 Fix variance overflow
In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.

To fix this we cast the result of the multiplication as unsigned.

The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.

For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).

PPC change is untested.

Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265
2012-02-09 12:38:31 -08:00
..
fastquantizeb_neon.asm Change use of eob in the encoder 2011-11-03 16:08:09 +02:00
picklpf_arm.c Reduce partial frame copy in encoder's pick_filter_level_fast 2011-10-26 15:25:07 +03:00
sad8_neon.asm Use WebM in copyright notice for consistency 2010-09-09 10:01:21 -04:00
sad16_neon.asm Use WebM in copyright notice for consistency 2010-09-09 10:01:21 -04:00
shortfdct_neon.asm NEON FDCT updated to match current C code 2011-09-20 10:20:55 +03:00
subtract_neon.asm Added predictor stride argument(s) to subtract functions 2011-11-15 12:53:01 -05:00
variance_neon.asm Fix variance overflow 2012-02-09 12:38:31 -08:00
vp8_memcpy_neon.asm Reduce partial frame copy in encoder's pick_filter_level_fast 2011-10-26 15:25:07 +03:00
vp8_mse16x16_neon.asm remove one set of 16x16 variance funcations 2011-06-09 11:23:05 -07:00
vp8_shortwalsh4x4_neon.asm NEON walsh transform updated to match C 2011-09-19 10:15:33 +03:00
vp8_subpixelvariance8x8_neon.asm Fix variance overflow 2012-02-09 12:38:31 -08:00
vp8_subpixelvariance16x16_neon.asm Fix variance overflow 2012-02-09 12:38:31 -08:00
vp8_subpixelvariance16x16s_neon.asm Fix variance overflow 2012-02-09 12:38:31 -08:00