generic-library/vpx
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Fix error in get_smooth_intra_threshold() for 10/12 bit.

Browse Source 
This function seems to scale the threshold for testing an
SSE value in the wrong direction for 10 and 12 bit inputs.

Also for a true SSE the scalings should probably be << 4 and 8

Change-Id: Iba8047b3f70d04aa46d9688a824f3d49c1c58e90
This commit is contained in:
paulwilkins
2016-06-30 13:31:39 +01:00
parent d004c64013
commit f9a3d08f1b
1 changed files with 2 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
vp9/encoder/vp9_firstpass.c
View File

@@ -532,10 +532,10 @@ static int get_smooth_intra_threshold(VP9_COMMON *cm) {
ret_val = SMOOTH_INTRA_THRESH;
break;
case VPX_BITS_10:
ret_val = SMOOTH_INTRA_THRESH >> 2;
ret_val = SMOOTH_INTRA_THRESH << 4;
break;
case VPX_BITS_12:
ret_val = SMOOTH_INTRA_THRESH >> 4;
ret_val = SMOOTH_INTRA_THRESH << 8;
break;
default:
assert(0 && "cm->bit_depth should be VPX_BITS_8, "