generic-library/vpx
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
3656835771
vpx/test/variance_test.cc

212 lines
7.1 KiB
C++
Raw Normal View History

Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
/*
* Copyright (c) 2012 The WebM project authors. All Rights Reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
#include <stdlib.h>
#include <new>
#include "third_party/googletest/src/include/gtest/gtest.h"
tests: clear system state after non-API calls add ClearSystemState() to reset MMX registers avoiding corrupting subsequent tests. Change-Id: I668deb09aa7aa467709776e5819f936910698bc0
2013-06-18 07:58:40 +02:00
#include "test/clear_system_state.h"
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
#include "vpx/vpx_integer.h"
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
#include "vpx_config.h"
extern "C" {
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
#if CONFIG_VP8_ENCODER
# include "vp8/common/variance.h"
# include "vp8_rtcd.h"
#endif
#if CONFIG_VP9_ENCODER
# include "vp9/encoder/vp9_variance.h"
# include "vp9_rtcd.h"
#endif
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
}
namespace {
using ::std::tr1::get;
using ::std::tr1::make_tuple;
using ::std::tr1::tuple;
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
template<typename VarianceFunctionType>
class VarianceTest :
public ::testing::TestWithParam<tuple<int, int, VarianceFunctionType> > {
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
public:
virtual void SetUp() {
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
const tuple<int, int, VarianceFunctionType>& params = this->GetParam();
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
width_ = get<0>(params);
height_ = get<1>(params);
variance_ = get<2>(params);
block_size_ = width_ * height_;
src_ = new uint8_t[block_size_];
ref_ = new uint8_t[block_size_];
ASSERT_TRUE(src_ != NULL);
ASSERT_TRUE(ref_ != NULL);
}
virtual void TearDown() {
delete[] src_;
delete[] ref_;
tests: clear system state after non-API calls add ClearSystemState() to reset MMX registers avoiding corrupting subsequent tests. Change-Id: I668deb09aa7aa467709776e5819f936910698bc0
2013-06-18 07:58:40 +02:00
libvpx_test::ClearSystemState();
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
}
protected:
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
void ZeroTest();
void OneQuarterTest();
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
uint8_t* src_;
uint8_t* ref_;
int width_;
int height_;
int block_size_;
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
VarianceFunctionType variance_;
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
};
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
template<typename VarianceFunctionType>
void VarianceTest<VarianceFunctionType>::ZeroTest() {
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
for (int i = 0; i <= 255; ++i) {
memset(src_, i, block_size_);
for (int j = 0; j <= 255; ++j) {
memset(ref_, j, block_size_);
unsigned int sse;
const unsigned int var = variance_(src_, width_, ref_, width_, &sse);
EXPECT_EQ(0u, var) << "src values: " << i << "ref values: " << j;
}
}
}
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
template<typename VarianceFunctionType>
void VarianceTest<VarianceFunctionType>::OneQuarterTest() {
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
memset(src_, 255, block_size_);
const int half = block_size_ / 2;
memset(ref_, 255, half);
memset(ref_ + half, 0, half);
unsigned int sse;
const unsigned int var = variance_(src_, width_, ref_, width_, &sse);
const unsigned int expected = block_size_ * 255 * 255 / 4;
EXPECT_EQ(expected, var);
}
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
// -----------------------------------------------------------------------------
// VP8 test cases.
namespace vp8 {
#if CONFIG_VP8_ENCODER
typedef VarianceTest<vp8_variance_fn_t> VP8VarianceTest;
TEST_P(VP8VarianceTest, Zero) { ZeroTest(); }
TEST_P(VP8VarianceTest, OneQuarter) { OneQuarterTest(); }
const vp8_variance_fn_t variance4x4_c = vp8_variance4x4_c;
const vp8_variance_fn_t variance8x8_c = vp8_variance8x8_c;
const vp8_variance_fn_t variance8x16_c = vp8_variance8x16_c;
const vp8_variance_fn_t variance16x8_c = vp8_variance16x8_c;
const vp8_variance_fn_t variance16x16_c = vp8_variance16x16_c;
INSTANTIATE_TEST_CASE_P(
C, VP8VarianceTest,
::testing::Values(make_tuple(4, 4, variance4x4_c),
make_tuple(8, 8, variance8x8_c),
make_tuple(8, 16, variance8x16_c),
make_tuple(16, 8, variance16x8_c),
make_tuple(16, 16, variance16x16_c)));
#if HAVE_MMX
const vp8_variance_fn_t variance4x4_mmx = vp8_variance4x4_mmx;
const vp8_variance_fn_t variance8x8_mmx = vp8_variance8x8_mmx;
const vp8_variance_fn_t variance8x16_mmx = vp8_variance8x16_mmx;
const vp8_variance_fn_t variance16x8_mmx = vp8_variance16x8_mmx;
const vp8_variance_fn_t variance16x16_mmx = vp8_variance16x16_mmx;
INSTANTIATE_TEST_CASE_P(
MMX, VP8VarianceTest,
::testing::Values(make_tuple(4, 4, variance4x4_mmx),
make_tuple(8, 8, variance8x8_mmx),
make_tuple(8, 16, variance8x16_mmx),
make_tuple(16, 8, variance16x8_mmx),
make_tuple(16, 16, variance16x16_mmx)));
#endif
#if HAVE_SSE2
const vp8_variance_fn_t variance4x4_wmt = vp8_variance4x4_wmt;
const vp8_variance_fn_t variance8x8_wmt = vp8_variance8x8_wmt;
const vp8_variance_fn_t variance8x16_wmt = vp8_variance8x16_wmt;
const vp8_variance_fn_t variance16x8_wmt = vp8_variance16x8_wmt;
const vp8_variance_fn_t variance16x16_wmt = vp8_variance16x16_wmt;
INSTANTIATE_TEST_CASE_P(
SSE2, VP8VarianceTest,
::testing::Values(make_tuple(4, 4, variance4x4_wmt),
make_tuple(8, 8, variance8x8_wmt),
make_tuple(8, 16, variance8x16_wmt),
make_tuple(16, 8, variance16x8_wmt),
make_tuple(16, 16, variance16x16_wmt)));
#endif
#endif // CONFIG_VP8_ENCODER
} // namespace vp8
// -----------------------------------------------------------------------------
// VP9 test cases.
namespace vp9 {
#if CONFIG_VP9_ENCODER
typedef VarianceTest<vp9_variance_fn_t> VP9VarianceTest;
TEST_P(VP9VarianceTest, Zero) { ZeroTest(); }
TEST_P(VP9VarianceTest, OneQuarter) { OneQuarterTest(); }
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
const vp9_variance_fn_t variance4x4_c = vp9_variance4x4_c;
const vp9_variance_fn_t variance8x8_c = vp9_variance8x8_c;
const vp9_variance_fn_t variance8x16_c = vp9_variance8x16_c;
const vp9_variance_fn_t variance16x8_c = vp9_variance16x8_c;
const vp9_variance_fn_t variance16x16_c = vp9_variance16x16_c;
INSTANTIATE_TEST_CASE_P(
C, VP9VarianceTest,
::testing::Values(make_tuple(4, 4, variance4x4_c),
make_tuple(8, 8, variance8x8_c),
make_tuple(8, 16, variance8x16_c),
make_tuple(16, 8, variance16x8_c),
make_tuple(16, 16, variance16x16_c)));
#if HAVE_MMX
const vp9_variance_fn_t variance4x4_mmx = vp9_variance4x4_mmx;
const vp9_variance_fn_t variance8x8_mmx = vp9_variance8x8_mmx;
const vp9_variance_fn_t variance8x16_mmx = vp9_variance8x16_mmx;
const vp9_variance_fn_t variance16x8_mmx = vp9_variance16x8_mmx;
const vp9_variance_fn_t variance16x16_mmx = vp9_variance16x16_mmx;
INSTANTIATE_TEST_CASE_P(
MMX, VP9VarianceTest,
::testing::Values(make_tuple(4, 4, variance4x4_mmx),
make_tuple(8, 8, variance8x8_mmx),
make_tuple(8, 16, variance8x16_mmx),
make_tuple(16, 8, variance16x8_mmx),
make_tuple(16, 16, variance16x16_mmx)));
#endif
#if HAVE_SSE2
Optimize variance functions Added SSE2 version of variance functions for super blocks. Change-Id: Ibeaae8771ca21c99d41dd74067574a51e97b412d
2013-05-22 18:07:30 +02:00
const vp9_variance_fn_t variance4x4_wmt = vp9_variance4x4_sse2;
const vp9_variance_fn_t variance8x8_wmt = vp9_variance8x8_sse2;
const vp9_variance_fn_t variance8x16_wmt = vp9_variance8x16_sse2;
const vp9_variance_fn_t variance16x8_wmt = vp9_variance16x8_sse2;
const vp9_variance_fn_t variance16x16_wmt = vp9_variance16x16_sse2;
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
INSTANTIATE_TEST_CASE_P(
SSE2, VP9VarianceTest,
::testing::Values(make_tuple(4, 4, variance4x4_wmt),
make_tuple(8, 8, variance8x8_wmt),
make_tuple(8, 16, variance8x16_wmt),
make_tuple(16, 8, variance16x8_wmt),
make_tuple(16, 16, variance16x16_wmt)));
#endif
add vp8 variance test Change-Id: I4e94ee2c4e2360d6a11a454c323f2899c1bb6f72
2013-02-23 01:23:36 +01:00
#endif // CONFIG_VP9_ENCODER
} // namespace vp9
Fix variance (signed integer) overflow In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f
2012-11-07 01:58:11 +01:00
} // namespace
Reference in New Issue Copy Permalink