generic-library/vpx
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
fea3556e20
vpx/vp8/encoder
History
Johann fea3556e20 Fix variance overflow 
In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.

To fix this we cast the result of the multiplication as unsigned.

The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.

For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).

PPC change is untested.

Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265
2012-02-09 12:38:31 -08:00
..
arm Fix variance overflow 2012-02-09 12:38:31 -08:00
ppc Fix variance overflow 2012-02-09 12:38:31 -08:00
x86 Fix variance overflow 2012-02-09 12:38:31 -08:00
asm_enc_offsets.c Disconnect ARM tgt_isa from dsp extensions 2012-01-20 10:38:15 -08:00
bitstream.c Correct clamping in use of vp8_find_near_mvs() 2012-01-26 09:37:27 -08:00
bitstream.h Disconnect ARM tgt_isa from dsp extensions 2012-01-20 10:38:15 -08:00
block.h Moved ref_frame_cost from MACROBLOCKD to MACROBLOCK 2012-02-02 13:40:08 -05:00
boolhuff.c Fix: check cx_data buffer prior to write 2011-10-20 15:55:00 -04:00
boolhuff.h Fix: check cx_data buffer prior to write 2011-10-20 15:55:00 -04:00
dct.c Use WebM in copyright notice for consistency 2010-09-09 10:01:21 -04:00
defaultcoefcounts.h Faster vp8_default_coef_probs 2011-08-16 16:21:21 -04:00
encodeframe.c Moved ref_frame_cost from MACROBLOCKD to MACROBLOCK 2012-02-02 13:40:08 -05:00
encodeintra.c BLOCKD structure cleanup 2012-01-31 11:02:39 -05:00
encodeintra.h RTCD: add arnr functions 2012-01-30 12:10:48 -08:00
encodemb.c RTCD: add arnr functions 2012-01-30 12:10:48 -08:00
encodemb.h RTCD: add arnr functions 2012-01-30 12:10:48 -08:00
encodemv.c Fix encoder partitioned output on ARM 2011-11-23 12:29:43 +02:00
encodemv.h Use full-pixel MV in mvsadcost calculation 2011-04-01 16:41:58 -04:00
ethreading.c Removed frames_till_alt_ref_frame from MACROBLOCKD 2012-02-02 13:34:13 -05:00
firstpass.c BLOCKD structure cleanup 2012-01-31 11:02:39 -05:00
firstpass.h Move firstpass motion map to stats packet 2010-10-21 14:04:20 -04:00
lookahead.c squash some signed/unsigned comparison warnings 2011-12-21 13:49:19 -08:00
lookahead.h squash some signed/unsigned comparison warnings 2011-12-21 13:49:19 -08:00
mcomp.c BLOCKD structure cleanup 2012-01-31 11:02:39 -05:00
mcomp.h RTCD: add motion search functions 2012-01-30 12:10:47 -08:00
modecosts.c Fix relative include paths 2011-02-10 15:09:44 -05:00
modecosts.h Use WebM in copyright notice for consistency 2010-09-09 10:01:21 -04:00
mr_dissim.c Multiple-resolution encoder 2011-12-05 17:59:42 -05:00
mr_dissim.h Multiple-resolution encoder 2011-12-05 17:59:42 -05:00
onyx_if.c RTCD: finalize removal of old RTCD system 2012-01-30 12:10:48 -08:00
onyx_int.h RTCD: add arnr functions 2012-01-30 12:10:48 -08:00
pickinter.c Merge "Allow to skip highest-resolution encoding in multi-resolution encoder" 2012-02-06 13:58:11 -08:00
pickinter.h Merge mr_pick_inter_mode and pick_inter_mode 2011-12-16 18:02:29 -05:00
picklpf.c RTCD: finalize removal of old RTCD system 2012-01-30 12:10:48 -08:00
psnr.c Increase static linkage, remove unused functions 2011-03-17 20:53:47 -04:00
psnr.h Increase static linkage, remove unused functions 2011-03-17 20:53:47 -04:00
quantize.c RTCD: add quantizer functions 2012-01-30 12:10:46 -08:00
quantize.h RTCD: add quantizer functions 2012-01-30 12:10:46 -08:00
ratectrl.c Remove legacy integer types 2011-12-22 09:58:40 -08:00
ratectrl.h Move pick_frame_size() to ratectrl.c 2011-04-26 16:49:54 -04:00
rdopt.c Moved ref_frame_cost from MACROBLOCKD to MACROBLOCK 2012-02-02 13:40:08 -05:00
rdopt.h get_plane_pointers: use u/v planes consistently 2012-01-18 12:50:06 -08:00
sad_c.c Replace vpx_ports/config.h with vpx_config.h 2011-09-22 13:33:54 +03:00
segmentation.c Use WebM in copyright notice for consistency 2010-09-09 10:01:21 -04:00
segmentation.h Fix relative include paths 2011-02-10 15:09:44 -05:00
ssim.c RTCD: add variance functions 2012-01-30 12:08:30 -08:00
temporal_filter.c BLOCKD structure cleanup 2012-01-31 11:02:39 -05:00
tokenize.c tokenizer: use correct block type context in stuff1st_order_b 2011-12-13 11:58:56 -08:00
tokenize.h Use MAX_ENTROPY_TOKENS and ENTROPY_NODES more consistently 2011-06-28 17:03:55 -04:00
treewriter.c Use WebM in copyright notice for consistency 2010-09-09 10:01:21 -04:00
treewriter.h Fix relative include paths 2011-02-10 15:09:44 -05:00
variance_c.c Fix variance overflow 2012-02-09 12:38:31 -08:00
variance.h RTCD: add motion search functions 2012-01-30 12:10:47 -08:00