fea3556e20
In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we cast the result of the multiplication as unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). PPC change is untested. Change-Id: I1bad27ea0720067def6d71a6da5f789508cec265 |
||
|---|---|---|
| .. | ||
| common | ||
| decoder | ||
| encoder | ||
| exports_dec | ||
| exports_enc | ||
| vp8_common.mk | ||
| vp8_cx_iface.c | ||
| vp8_dx_iface.c | ||
| vp8cx_arm.mk | ||
| vp8cx.mk | ||
| vp8dx.mk | ||