Use saturated addition in SSSE3 of 32x32 quant
The 32x32 forward transform can potentially reach peak coefficient value close to 32700, while the rounding factor can go upto 610. This could cause overflow issue in the SSSE3 implementation of 32x32 quantization process. This commit resolves this issue by replacing the addition operations with saturated addition operations in 32x32 block quantization. Change-Id: Id6b98996458e16c5b6241338ca113c332bef6e70
This commit is contained in:
@@ -135,6 +135,7 @@ void vp9_quantize_b_32x32_c(int16_t *coeff_ptr, intptr_t n_coeffs,
|
||||
|
||||
if (x >= zbin) {
|
||||
x += ROUND_POWER_OF_TWO(round_ptr[rc != 0], 1);
|
||||
x = clamp(x, INT16_MIN, INT16_MAX);
|
||||
y = (((int)(((int)(x * quant_ptr[rc != 0]) >> 16) + x)) *
|
||||
quant_shift_ptr[rc != 0]) >> 15; // quantize (x)
|
||||
|
||||
|
Reference in New Issue
Block a user