generic-library/vpx
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
4821084b3f
vpx/test/error_resilience_test.cc

234 lines
7.3 KiB
C++
Raw Normal View History

Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
/*
Cleanup copyright headers. Change-Id: I7ff2b18e085f3fffaa7f25d40e4e50427b808f43
2013-12-12 17:36:34 +01:00
* Copyright (c) 2013 The WebM project authors. All Rights Reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
#include "third_party/googletest/src/include/gtest/gtest.h"
Support multiple codecs in test infrastructure This commit starts to convert the tests to a system where the codec to be used is provided by a factory object. Currently no tests are instantiated for VP9 since they all fail for various reasons, but it was verified that they're called and the correct codec is instantiated. Change-Id: Ia7506df2ca3a7651218ba3ca560634f08c9fbdeb
2013-01-18 20:51:12 +01:00
#include "test/codec_factory.h"
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
#include "test/encode_test_driver.h"
#include "test/i420_video_source.h"
Support multiple codecs in test infrastructure This commit starts to convert the tests to a system where the codec to be used is provided by a factory object. Currently no tests are instantiated for VP9 since they all fail for various reasons, but it was verified that they're called and the correct codec is instantiated. Change-Id: Ia7506df2ca3a7651218ba3ca560634f08c9fbdeb
2013-01-18 20:51:12 +01:00
#include "test/util.h"
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
namespace {
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
const int kMaxErrorFrames = 8;
const int kMaxDroppableFrames = 8;
Support multiple codecs in test infrastructure This commit starts to convert the tests to a system where the codec to be used is provided by a factory object. Currently no tests are instantiated for VP9 since they all fail for various reasons, but it was verified that they're called and the correct codec is instantiated. Change-Id: Ia7506df2ca3a7651218ba3ca560634f08c9fbdeb
2013-01-18 20:51:12 +01:00
class ErrorResilienceTest : public ::libvpx_test::EncoderTest,
public ::libvpx_test::CodecTestWithParam<libvpx_test::TestMode> {
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
protected:
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
ErrorResilienceTest() : EncoderTest(GET_PARAM(0)),
psnr_(0.0),
nframes_(0),
mismatch_psnr_(0.0),
mismatch_nframes_(0),
encoding_mode_(GET_PARAM(1)) {
Reset();
}
Support multiple codecs in test infrastructure This commit starts to convert the tests to a system where the codec to be used is provided by a factory object. Currently no tests are instantiated for VP9 since they all fail for various reasons, but it was verified that they're called and the correct codec is instantiated. Change-Id: Ia7506df2ca3a7651218ba3ca560634f08c9fbdeb
2013-01-18 20:51:12 +01:00
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
virtual ~ErrorResilienceTest() {}
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
void Reset() {
error_nframes_ = 0;
droppable_nframes_ = 0;
}
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
virtual void SetUp() {
InitializeConfig();
SetMode(encoding_mode_);
}
virtual void BeginPassHook(unsigned int /*pass*/) {
psnr_ = 0.0;
nframes_ = 0;
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
mismatch_psnr_ = 0.0;
mismatch_nframes_ = 0;
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
}
virtual void PSNRPktHook(const vpx_codec_cx_pkt_t *pkt) {
psnr_ += pkt->data.psnr.psnr[0];
nframes_++;
}
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
virtual void PreEncodeFrameHook(libvpx_test::VideoSource *video) {
frame_flags_ &= ~(VP8_EFLAG_NO_UPD_LAST |
VP8_EFLAG_NO_UPD_GF |
VP8_EFLAG_NO_UPD_ARF);
if (droppable_nframes_ > 0 &&
(cfg_.g_pass == VPX_RC_LAST_PASS || cfg_.g_pass == VPX_RC_ONE_PASS)) {
for (unsigned int i = 0; i < droppable_nframes_; ++i) {
Make "good" quality 2-pass vpxenc encoding default Currently, the best quality mode in VP9 is not very well developed, and unnecessarily makes the encode too slow. Hence the command line default is changed to "good" quality. Also, the number of passes default is changed to 2 passes as well, since 1-pass encoding is not very efficient in VP9. Besides, a number of VP9 defaults are set to the currently recommended settings. With these changes, vpxenc run with --codec=vp9 --kf-max-dist=9999 --cpu-used=0 should work about the same as our borg results. Note when the --cpu-used=0 option is dropped there will be a slight difference in the output, because of a difference in the cpu-used value for the first pass. Specifically, the default when unspecified is to use cpu_used=1 for the first pass and cpu_used=0 for the second pass. But when specified, both passes will use the cpu-used value specified. Note that this also changes the default for VP8 as being "good" but other options stay unchanged. Change-Id: Ib23c1a05ae2f36ee076c0e34403efbda518c5066
2013-08-19 23:16:26 +02:00
if (droppable_frames_[i] == video->frame()) {
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
std::cout << " Encoding droppable frame: "
<< droppable_frames_[i] << "\n";
frame_flags_ |= (VP8_EFLAG_NO_UPD_LAST |
VP8_EFLAG_NO_UPD_GF |
VP8_EFLAG_NO_UPD_ARF);
return;
}
}
}
}
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
double GetAveragePsnr() const {
if (nframes_)
return psnr_ / nframes_;
return 0.0;
}
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
double GetAverageMismatchPsnr() const {
if (mismatch_nframes_)
return mismatch_psnr_ / mismatch_nframes_;
return 0.0;
}
virtual bool DoDecode() const {
if (error_nframes_ > 0 &&
(cfg_.g_pass == VPX_RC_LAST_PASS || cfg_.g_pass == VPX_RC_ONE_PASS)) {
for (unsigned int i = 0; i < error_nframes_; ++i) {
if (error_frames_[i] == nframes_ - 1) {
std::cout << " Skipping decoding frame: "
<< error_frames_[i] << "\n";
return 0;
}
}
}
return 1;
}
virtual void MismatchHook(const vpx_image_t *img1,
const vpx_image_t *img2) {
double mismatch_psnr = compute_psnr(img1, img2);
mismatch_psnr_ += mismatch_psnr;
++mismatch_nframes_;
// std::cout << "Mismatch frame psnr: " << mismatch_psnr << "\n";
}
void SetErrorFrames(int num, unsigned int *list) {
if (num > kMaxErrorFrames)
num = kMaxErrorFrames;
else if (num < 0)
num = 0;
error_nframes_ = num;
for (unsigned int i = 0; i < error_nframes_; ++i)
error_frames_[i] = list[i];
}
void SetDroppableFrames(int num, unsigned int *list) {
if (num > kMaxDroppableFrames)
num = kMaxDroppableFrames;
else if (num < 0)
num = 0;
droppable_nframes_ = num;
for (unsigned int i = 0; i < droppable_nframes_; ++i)
droppable_frames_[i] = list[i];
}
unsigned int GetMismatchFrames() {
return mismatch_nframes_;
}
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
private:
double psnr_;
unsigned int nframes_;
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
unsigned int error_nframes_;
unsigned int droppable_nframes_;
double mismatch_psnr_;
unsigned int mismatch_nframes_;
unsigned int error_frames_[kMaxErrorFrames];
unsigned int droppable_frames_[kMaxDroppableFrames];
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
libvpx_test::TestMode encoding_mode_;
};
TEST_P(ErrorResilienceTest, OnVersusOff) {
const vpx_rational timebase = { 33333333, 1000000000 };
cfg_.g_timebase = timebase;
cfg_.rc_target_bitrate = 2000;
Make "good" quality 2-pass vpxenc encoding default Currently, the best quality mode in VP9 is not very well developed, and unnecessarily makes the encode too slow. Hence the command line default is changed to "good" quality. Also, the number of passes default is changed to 2 passes as well, since 1-pass encoding is not very efficient in VP9. Besides, a number of VP9 defaults are set to the currently recommended settings. With these changes, vpxenc run with --codec=vp9 --kf-max-dist=9999 --cpu-used=0 should work about the same as our borg results. Note when the --cpu-used=0 option is dropped there will be a slight difference in the output, because of a difference in the cpu-used value for the first pass. Specifically, the default when unspecified is to use cpu_used=1 for the first pass and cpu_used=0 for the second pass. But when specified, both passes will use the cpu-used value specified. Note that this also changes the default for VP8 as being "good" but other options stay unchanged. Change-Id: Ib23c1a05ae2f36ee076c0e34403efbda518c5066
2013-08-19 23:16:26 +02:00
cfg_.g_lag_in_frames = 10;
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
init_flags_ = VPX_CODEC_USE_PSNR;
libvpx_test::I420VideoSource video("hantro_collage_w352h288.yuv", 352, 288,
timebase.den, timebase.num, 0, 30);
// Error resilient mode OFF.
cfg_.g_error_resilient = 0;
ASSERT_NO_FATAL_FAILURE(RunLoop(&video));
const double psnr_resilience_off = GetAveragePsnr();
EXPECT_GT(psnr_resilience_off, 25.0);
// Error resilient mode ON.
cfg_.g_error_resilient = 1;
ASSERT_NO_FATAL_FAILURE(RunLoop(&video));
const double psnr_resilience_on = GetAveragePsnr();
EXPECT_GT(psnr_resilience_on, 25.0);
// Test that turning on error resilient mode hurts by 10% at most.
if (psnr_resilience_off > 0.0) {
const double psnr_ratio = psnr_resilience_on / psnr_resilience_off;
EXPECT_GE(psnr_ratio, 0.9);
EXPECT_LE(psnr_ratio, 1.1);
}
}
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
TEST_P(ErrorResilienceTest, DropFramesWithoutRecovery) {
const vpx_rational timebase = { 33333333, 1000000000 };
cfg_.g_timebase = timebase;
Revert "Adjust error resilience test data rate." This reverts commit b24735c622cefe3a03633edd69dff65d2a6ee0dd since the adjusted threshold doesn't allow the existing tests to pass. Will disable the failing test in a separate commit. Change-Id: I26d41cf6175f300bbad493cecdc96e6b0dd6f2fe
2013-05-07 21:58:32 +02:00
cfg_.rc_target_bitrate = 500;
Make "good" quality 2-pass vpxenc encoding default Currently, the best quality mode in VP9 is not very well developed, and unnecessarily makes the encode too slow. Hence the command line default is changed to "good" quality. Also, the number of passes default is changed to 2 passes as well, since 1-pass encoding is not very efficient in VP9. Besides, a number of VP9 defaults are set to the currently recommended settings. With these changes, vpxenc run with --codec=vp9 --kf-max-dist=9999 --cpu-used=0 should work about the same as our borg results. Note when the --cpu-used=0 option is dropped there will be a slight difference in the output, because of a difference in the cpu-used value for the first pass. Specifically, the default when unspecified is to use cpu_used=1 for the first pass and cpu_used=0 for the second pass. But when specified, both passes will use the cpu-used value specified. Note that this also changes the default for VP8 as being "good" but other options stay unchanged. Change-Id: Ib23c1a05ae2f36ee076c0e34403efbda518c5066
2013-08-19 23:16:26 +02:00
// FIXME(debargha): Fix this to work for any lag.
// Currently this test only works for lag = 0
cfg_.g_lag_in_frames = 0;
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
init_flags_ = VPX_CODEC_USE_PSNR;
libvpx_test::I420VideoSource video("hantro_collage_w352h288.yuv", 352, 288,
timebase.den, timebase.num, 0, 30);
// Error resilient mode ON.
cfg_.g_error_resilient = 1;
// Set an arbitrary set of error frames same as droppable frames
unsigned int num_droppable_frames = 2;
unsigned int droppable_frame_list[] = {5, 16};
SetDroppableFrames(num_droppable_frames, droppable_frame_list);
SetErrorFrames(num_droppable_frames, droppable_frame_list);
ASSERT_NO_FATAL_FAILURE(RunLoop(&video));
// Test that no mismatches have been found
std::cout << " Mismatch frames: "
<< GetMismatchFrames() << "\n";
EXPECT_EQ(GetMismatchFrames(), (unsigned int) 0);
// reset previously set error/droppable frames
Reset();
Partially disable error resilience test Disables the part of the error-resilient test that tests the quality after dropping undroppable frames. It's not clear how to set the threshold for this correctly at the moment. Change-Id: I3ee4a0d475498f44711fdef05749f305e8d08591
2013-05-07 22:02:45 +02:00
#if 0
// TODO(jkoleszar): This test is disabled for the time being as too
// sensitive. It's not clear how to set a reasonable threshold for
// this behavior.
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
// Now set an arbitrary set of error frames that are non-droppable
unsigned int num_error_frames = 3;
unsigned int error_frame_list[] = {3, 10, 20};
SetErrorFrames(num_error_frames, error_frame_list);
ASSERT_NO_FATAL_FAILURE(RunLoop(&video));
Partially disable error resilience test Disables the part of the error-resilient test that tests the quality after dropping undroppable frames. It's not clear how to set the threshold for this correctly at the moment. Change-Id: I3ee4a0d475498f44711fdef05749f305e8d08591
2013-05-07 22:02:45 +02:00
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
// Test that dropping an arbitrary set of inter frames does not hurt too much
// Note the Average Mismatch PSNR is the average of the PSNR between
// decoded frame and encoder's version of the same frame for all frames
// with mismatch.
const double psnr_resilience_mismatch = GetAverageMismatchPsnr();
std::cout << " Mismatch PSNR: "
<< psnr_resilience_mismatch << "\n";
EXPECT_GT(psnr_resilience_mismatch, 20.0);
Partially disable error resilience test Disables the part of the error-resilient test that tests the quality after dropping undroppable frames. It's not clear how to set the threshold for this correctly at the moment. Change-Id: I3ee4a0d475498f44711fdef05749f305e8d08591
2013-05-07 22:02:45 +02:00
#endif
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
}
Support multiple codecs in test infrastructure This commit starts to convert the tests to a system where the codec to be used is provided by a factory object. Currently no tests are instantiated for VP9 since they all fail for various reasons, but it was verified that they're called and the correct codec is instantiated. Change-Id: Ia7506df2ca3a7651218ba3ca560634f08c9fbdeb
2013-01-18 20:51:12 +01:00
VP8_INSTANTIATE_TEST_CASE(ErrorResilienceTest, ONE_PASS_TEST_MODES);
Adds an error-resilient mode with test Adds an error-resilient mode where frames can be continued to be decoded even when there are errors (due to network losses) on a prior frame. Specifically, backward updates are turned off and probabilities of various symbols are reset to defaults at the beginning of each frame. Further, the last frame's mvs are not used for the mv reference list, and the sorting of the initial list based on search on previous frames is turned off as well. Also adds a test where an arbitrary set of frames are skipped from decoding to simulate errors. The test verifies (1) that if the error frames are droppable - i.e. frame buffer updates have been turned off - there are no mismatch errors for the remaining frames after the error frames; and (2) if the error-frames are non droppable, there are not only no decoding errors but the mismatch PSNR between the decoder's version of the post-error frames and the encoder's version is at least 20 dB. Change-Id: Ie6e2bcd436b1e8643270356d3a930e8989ff52a5
2013-01-15 15:43:35 +01:00
VP9_INSTANTIATE_TEST_CASE(ErrorResilienceTest, ONE_PASS_TEST_MODES);
Unit Test for Error Resilience Mode This unit test compares the difference in quality with error resilience enabled and disabled. The test runs for all of the one-pass encoding modes. The test ensures that the effect of turning on error resilience makes less than a 10% difference in PSNR. Further cases should be added to do a more comprehensive test. Change-Id: I1fc747fc78c9459bc6c74494f4b38308dbed0c32
2012-10-02 21:16:27 +02:00
} // namespace
Reference in New Issue Copy Permalink