Johann fea3556e20 Fix variance overflow ... In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we cast the result of the multiplication as unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). PPC change is untested. Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265		2012-02-09 12:38:31 -08:00
..
armv5te	Fix encoder partitioned output on ARM	2011-11-23 12:29:43 +02:00
armv6	Fix variance overflow	2012-02-09 12:38:31 -08:00
neon	Fix variance overflow	2012-02-09 12:38:31 -08:00
boolhuff_arm.c	Fix encoder partitioned output on ARM	2011-11-23 12:29:43 +02:00
dct_arm.c	RTCD: add FDCT functions	2012-01-30 12:10:42 -08:00
quantize_arm.c	RTCD: add quantizer functions	2012-01-30 12:10:46 -08:00
variance_arm.c	RTCD: add variance functions	2012-01-30 12:08:30 -08:00