generic-library/vpx
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Prevent negative cost for highbitdepth

Browse Source 
Adds proper scaling for highbitdepth in a rdopt cost.

Change-Id: I066694799a7f491b830945ef1c66eb202071c355
This commit is contained in:
Deb Mukherjee 2014-10-03 10:22:21 -07:00
parent ba172ff0a5
commit 431cdc33ee
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
vp9/encoder/vp9_rdopt.c
View File

@ -491,7 +491,9 @@ static void block_rd_txfm(int plane, int block, BLOCK_SIZE plane_bsize,
if (x->plane[plane].eobs[block]) {
int64_t dc_correct = coeff[0] * coeff[0] -
(coeff[0] - dqcoeff[0]) * (coeff[0] - dqcoeff[0]);
#if CONFIG_VP9_HIGHBITDEPTH
dc_correct >>= ((xd->bd - 8) * 2);
#endif
if (tx_size != TX_32X32)
dc_correct >>= 2;