In the variance calculations the difference is summed and later squared.
When the sum exceeds sqrt(2^31) the value is treated as a negative when
it is shifted which gives incorrect results.
To fix this we force the multiplication to be unsigned.
The alternative fix is to shift sum down by 4 before multiplying.
However that will reduce precision.
For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and
change).
This change is based on:
1698234 Missed some variance casts
fea3556 Fix variance overflow
Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
s/([vV][pP])8/$19/
additionally dct.h was removed; declare the _c functions that are used
in the tests. the TODO for conversion to parameterized tests still
remains.
Change-Id: I73db9425a57075bbb78a92693ba6b320578981cd
there are still a couple type of warning left, which are related to
double constants assigned to float type. As those would be addressed
by the conversion of transforms into integer version. This commit
has left those un-dealt with.
Change-Id: I48fd9b489c0c27ad6b543f4177423419f929f2bb
Documentation is typically auto-detected by checking for php and
doxygen. Add an option to explicitly disable it.
Remove toggle keywords from libraries, examples, documentation and
unit tests. They were not consistent with the default status.
Change-Id: I21049675ccfd8e58ac612cd058641b197db5c0eb
calculate the txfm_cache difference first as both values may be
INT64_MAX with the intent that they cancel each other out.
Change-Id: I214d072458e1b24f60289974e6302af1aff7b66c
rd_pick_intra4x4mby_modes / rd_pick_intra8x8mby_modes would both use the
input value of 'rate_y' in the return calculation. In many places this
value is uninitialized. Remove the unneeded sum.
Change-Id: Icbd3df685303000301e69291c0ebc06f74bd548d
The block sizes for decoding tokens are up to 16x16, which means
eobs is within [0, 256]. Using (signed) char is not enough. Changed
eobs data type to unsigned short to fix the problem.
Change-Id: I88a7d3098e1f1604c336d6adb88ffec971fb03a6
Most of these were picked up by jenkins in the commit that changed
the vp8 namespace to vp9 in common/.
Change-Id: I5cbd56ffc753b92ef805133cda6acc1713a13878
For non-static functions, change the prefix to vp9_. For static functions,
remove the prefix. Also fix some comments, remove unused code or unused
function prototypes.
Change-Id: I1f8be05362f66060fe421c3d4c9a906fdf835de5
Most of these were picked up in the previous commit (prefix change from
vp8_ to vp9_), but I'm pushing this separately so that it's easier to
review.
Change-Id: Ifce2cdd6f008f4b1fbc2d89b5196d75e35fe115d
Most of these were picked up in the previous commit (prefix change from
vp8_ to vp9_), but I'm pushing this separately so that it's easier to
review.
Change-Id: I91e959895778b8632d7d33375523df8a7568a490
Converted the forward and inverse transforms to integer forms.
Modify #define TEST_INT 1/0
in the code to call integer/float version of transforms.
The tests showed that average OVERALL PSNR loss was less than 0.1%.
Change-Id: I1dfa4eeab6412597e3b970ce299cf0e116a917e6
