Commit Graph

40 Commits

Author SHA1 Message Date
Linfeng Zhang
dbbbd44304 fix signed integer overflow of idct
Exposed by fuzz test in high bitdepth.
The bug is introduced in commit 64653fa.

BUG=webm:1466

Change-Id: Idd77d5c6a60efb9241471611ce1aba0646cb6ff5
2017-09-27 11:17:54 -07:00
clang-format
7587a97551 apply clang-format
Change-Id: If4c3e8a396d0fcb304f407b44e28cac3219f038c
2017-09-01 01:24:03 -07:00
James Zern
45daecb4f7 vp8_decode_frame: fix oob read on truncated key frame
the check for error correction being disabled was overriding the data
length checks. this avoids returning incorrect information (width /
height) for the decoded frame which could result in inconsistent sizes
returned in to an application causing it to read beyond the bounds of
the frame allocation.

BUG=webm:1443
BUG=b/62458770

Change-Id: I063459674e01b57c0990cb29372e0eb9a1fbf342
2017-06-08 23:16:04 +00:00
James Zern
cb22359d02 vp9,read_inter_block_mode_info: quiet msan warning
best_sub8x8[1] won't be used meaningfully when is_compound is false, but
may trigger an msan warning as the value is copied around and later
clamped.

BUG=667044

Change-Id: Icc24c3b72cdb550bebea44d4aaa4ff8bf3fbab56
2016-11-22 15:32:00 -08:00
James Zern
1961a92a94 vp9,tile_worker_hook: correctly set jmp target
vp9_init_macroblockd() resets the error_info to cm's global copy; this
needs to be set to the thread-level target to avoid jumping to the
incorrect stack, resulting in hang or crash.
broken since:
1f4a6c8 vp9/tile_worker_hook: add multiple tile decoding
includes v1.5.0, v1.6.0

BUG=629481

Change-Id: Icbf1696b25ba8c479e845fbf227b3c3ca73542f5
2016-11-01 18:45:50 -07:00
James Zern
fca2196a2e invalid_file_test: quiet unused const warning
with --disable-vp9

Change-Id: I81bd603b02ee5d1b45a50aa9e7534f9da498b0e0
2016-10-01 11:49:02 -07:00
James Zern
450d89034b vp9_detokenize,decode_coefs: fix signed int overflow
when decoding an invalid bitstream with --enable-vp9-highbitdepth

BUG=webm:1297

Change-Id: I401d87033b4293f2ca595bc51678aad9951ecf15
2016-09-28 22:42:03 -07:00
James Zern
e6f0c26268 invalid_file_test: quiet -Wunused-const-variable warnings
present when --disable-vp8(-decoder) or --disable-vp9(-decoder) was used

Change-Id: I31ebb7a55c6f1af3c744982f56b78e80116cc845
2016-09-01 19:54:34 -07:00
James Zern
0f42d1fa85 vp8: fix decoder crash with invalid leading keyframes
decoding the same invalid keyframe twice would result in a crash as the
second time through the decoder would be assumed to have been
initialized as there was no resolution change. in this case the
resolution was itself invalid (0x6), but vp8_peek_si() was only failing
in the case of 0x0.
invalid-vp80-00-comprehensive-018.ivf.2kf_0x6.ivf tests this case by
duplicating the first keyframe and additionally adds a valid one to
ensure decoding can resume without error.

BUG=b/30593765

Change-Id: If0859035908b7870d67a7f3f646b5a080252eb6d
2016-08-23 16:27:52 -07:00
James Zern
8b4c31584e vp9_alloc_context_buffers: clear cm->mi* on failure
this fixes a crash in vp9_dec_setup_mi() via
vp9_init_context_buffers() should decoding continue and the decoder
resyncs on a smaller frame

BUG=b/30593752

Change-Id: I9ce8d94abe89bcd058697e8bd8599690e61bd380
2016-08-19 00:18:11 -07:00
James Zern
475e9d26e0 tests: use scoped_ptr for local video source vars
prevents leak warnings on ASSERT*() failures

Change-Id: I1d3edbdbb18dbbe3b17691971348a8121cf09afa
2016-08-08 14:43:14 -07:00
clang-format
33e40cb5db test: apply clang-format
Change-Id: I0d9ab85855eb723f653a7bb09b3d0d31dd6cfd2f
2016-07-27 01:58:52 +00:00
James Zern
fb209003a8 invalid_file_test: loosen error check w/tile-threading
The serial decode check is too strict for tile-threaded decoding as
there is no guarantee on the decode order nor which specific error
will take precedence. Currently a tile-level error is not forwarded so
the frame will simply be marked corrupt.

Change-Id: I51cf1e39e44bedeac93746154b36a4ccb2f059b1
2015-10-06 16:40:20 -07:00
Debargha Mukherjee
1c8567ff09 Remove some trailing whitespaces
Change-Id: Icf06d35ca347713253d1eba341a894b51efa81a9
2015-09-08 01:31:04 -07:00
Yaowu Xu
7c0c62df1d Correctly report "Unsupported bitstream profile"
For vp9 decoder build without profile 2 and profile 3 support, this
commit changes to report error "Unsupported bitstream profile" for
input streams in profile 2 or 3, rather than other misleading error
information.

In addition, one of the invalid files in unit tests is actually coded
profile 2, this commit makes it tested only when the decoder is built
with vp9-highbitdepth.

This fixes issue #1028.

Change-Id: I8b6c1210787c8f89c703a546687dcf973ac20fc0
2015-07-17 10:51:02 -07:00
James Zern
0261fb4c4f vp9: fix segfault w/corrupt data post frame-parallel merge
cm->frame_bufs[].idx values were made consistent in:
61c5e94 Use -1 consistently as invalid buffer idx
update the initialization in swap_frame_buffers() to match.
additionally:
  - remove some shadowed variables in the former and marked them volatile

Change-Id: Ie3f9636c405bd822112bb56bd22d28024ae98909
2015-02-05 12:11:40 -08:00
Yunqing Wang
1228433430 Modify the frame context memory deallocation
This patch was to fix the vpxdec fuzzing3 test failure. When an
error occurs, setjmp() is invoked, which calls the decoder
removing routine. In multiple thread situation, other threads
could try to access the frame context memory that is already
deallocated, thus causing a segfault.

An invalid unit test was added for this issue.

Change-Id: Ida7442154f3d89759483f0f4fe0324041fffb952
2014-11-06 11:34:19 -08:00
James Zern
7ee073e61d vp9: wait for key/intra-only frame after corruption
don't bother decoding any further after receiving an earlier decode
error until a key/intra-only frame is encountered.

Change-Id: I381917b70d7a9e6f8d6de42e3d181bb113a4cec4
2014-09-09 19:36:11 -07:00
Alex Converse
864fd85011 Add an invalid file test for referencing a different color space.
Change-Id: I46472a524f5188b293332946be943bd15cfc8777
2014-09-08 11:12:15 -07:00
James Zern
bb4950dfdf vp9: correct context buffer resize check
allocations within vp9_alloc_context_buffers() rely on mi_rows/mi_cols
individually, use those to determine whether to realloc rather than
stride and stride * rows. this fixes a crash with some fuzzed files for
invalid accesses into last_frame_seg_map and above_context.

Change-Id: I7b9f40dcf170d443890f3bd2acd285507943c7d4
2014-09-04 19:14:21 -07:00
James Zern
440f5097c7 vp9: fail decode if block/frame refs are corrupt
proceeding using a corrupt (incompletely decoded) frame reference may
lead to incorrect assumptions about allocation sizes leading to a crash.

Change-Id: I76e74f2e1be127c2e2c7e1174bb3307497dfd23d
2014-09-04 19:14:00 -07:00
James Zern
fec40f9269 vp9: fix m/t loop filter invalid free
store the number of allocated rows in VP9LfSync, the calculated values
can not be relied on when dealing with corrupt material.

Change-Id: I13b8bcec9738c299a71df726772ab7ac05511e5b
2014-08-29 11:04:45 -07:00
James Zern
cde790c36d vp9: fix crash in inline loopfilter w/corrupt file
attempting to decode a frame after the previous frame failed has the
potential of interrupting an earlier loop filter task

Change-Id: I6f2b1ddcdf5b89c3e2ee8caf5289dada2a087d66
2014-08-27 16:55:31 -07:00
James Zern
4f27202df7 vp9: fix crash in mt loopfilter w/corrupt file
if the first frame was corrupt and loop filter not called, the next call
would assume the necessary allocations had been done and segfault when
accessing a NULL pointer

Change-Id: Ib6ef505e5c594e6f0fe65ab0700172bcf06b92a6
2014-08-27 14:21:14 -07:00
James Zern
79bb2cddd3 invalid_file_test: correct HandlePeekResult signature
restore const lost in:
f0f9ab6 invalid_file_test.cc: remove unused param warnings

Change-Id: I0c5318f84c81e3dd7bb44468463faf9edd6bc18a
2014-08-22 23:06:13 -07:00
James Zern
aaea40d847 Merge "tests: use vpx_codec_dec_cfg_t() to initialize vars" 2014-08-22 17:05:03 -07:00
James Zern
f0f9ab68ae invalid_file_test.cc: remove unused param warnings
Change-Id: Ifde2849f45acb59fbcf29c8735ef0a10d4d9041e
2014-08-22 12:47:49 -07:00
James Zern
b4b191aba0 tests: use vpx_codec_dec_cfg_t() to initialize vars
0-initializes [1] and removes a warning for missing initializer fields

[1] http://en.cppreference.com/w/cpp/language/value_initialization

Change-Id: I364248010c8fa663c71d8f06a9999c730e92db4c
2014-08-22 12:16:20 -07:00
Deb Mukherjee
727f384085 Merge "Separates profile 2 into 2 profiles 2 and 3" 2014-07-18 03:23:51 -07:00
Deb Mukherjee
c447a50aea Separates profile 2 into 2 profiles 2 and 3
Separates HBD profile int two profiles (2 and 3) consistent with the
highbitdepth branch. This patch is ported from the original highbitdepth
branch patch: https://gerrit.chromium.org/gerrit/#/c/70460/

Two of the invalid file tests needed to be updated.

Change-Id: I6a4acd2f7a60b1fb4cbcc8e0dad4eab4248431e3
2014-07-17 20:51:59 -07:00
James Zern
c1259aa343 invalid_file_test: add an operator<< for DecodeParam
Improves the --gtest_list_tests output and avoids a valgrind warning in
gtest's testing::internal2::PrintBytesInObjectTo() due to padding in the
structure.

old:
VP9/InvalidFileTest.
  ReturnCode/0  # GetParam() = (0x9d5308, 16-byte object <01-00 00-00 00-00 00-00 37-02 73-00 00-00 00-00>)

new:
VP9/InvalidFileTest.
  ReturnCode/0  # GetParam() = (0x9d5308, threads: 1 file: invalid-vp90-01-v2.webm)

Change-Id: Ifb9c66fba2e72272bd591a3f6273aeb6bda6af4a
2014-07-14 18:45:50 -07:00
James Zern
44eb577cf7 invalid_file_test: convert test param from tuple to struct
fixes visual studio 9 + apple clang builds where the template type is
interpreted as char[] rather than const char*:

::f1_' : cannot specify explicit initializer for arrays
error: array initializer must be an initializer list or string literal

Change-Id: I27286ce341b2f7a09b6202caffd6b72f64fd2234
2014-07-12 13:34:36 -07:00
hkuang
c147cf3d3b Add unit test to test tile decoding error handling.
Also fix bugs related with corrupted frame handling.
Return VPX_CODEC_CORRUPT_FRAME when getting corrupted
block.

Change-Id: I7207ccc7c68c4df2b40b561315d16e49ccf7ff41
2014-07-11 13:50:05 -07:00
James Zern
ec942e6e82 invalid_file_test: output error detail on mismatch
Change-Id: I6abf139961fbb6a46db740dc023e26ac1e8e9305
2014-07-01 20:18:00 -07:00
Jim Bankoski
ee3802150c Add a test that tests invalid partitions for profile 1
Change-Id: I3d95fbe9a8098256582b5386881d5b7bbdb317ed
2014-07-01 15:37:26 -07:00
Jim Bankoski
9f37d149c1 Better validation of invalid files
This patch checks that a decoder never tries to reference frame that's
outside the range of 2x to 1/16th the size of this frame.  Any attempt
to do so causes a failure.

Change-Id: I5c98fa7bb95ac4f29146f29dd92b62fe96164e4c
2014-06-27 10:03:15 -07:00
James Zern
b2b07755e0 vp9: check tile column count
the max is 6. there are assumptions throughout the decode regarding
this; fixes a crash with a fuzzed bitstream

$ zzuf -s 5861 -r 0.01:0.05 -b 6- \
  < vp90-2-00-quantizer-00.webm.ivf \
  | dd of=invalid-vp90-2-00-quantizer-00.webm.ivf.s5861_r01-05_b6-.ivf \
    bs=1 count=81883

Change-Id: I6af41bb34252e88bc156a4c27c80d505d45f5642
2014-06-24 19:26:11 -07:00
Jim Bankoski
c3db2d8bc8 error check vp9 superframe parsing
This patch insures that the last byte of a chunk that contains a
valid superframe marker byte,  actually has a proper superframe index.
If not it returns an error.

As part of doing that the file : vp90-2-15-fuzz-flicker.webm now fails
to decode properly and moves to the invalid file test from the test
vector suite.

Change-Id: I5f1da7eb37282ec0c6394df5c73251a2df9c1744
2014-06-23 07:04:57 -07:00
Jim Bankoski
88ba08818e Fix bug in error handling that causes segfault
See: https://code.google.com/p/chromium/issues/detail?id=362697

The code properly catches an invalid stream but seg faults instead of
returning an error due to a buffer not having been initialized. This
code fixes that.

Change-Id: I695595e742cb08807e1dfb2f00bc097b3eae3a9b
2014-06-20 14:44:50 -07:00
Jim Bankoski
dc2f2ce594 Validate error checking code in decoder.
This patch adds a mechanism for insuring error checking on invalid files
by creating a unit test that runs the decoder and tests that the error
code matches what's expected on each frame in the decoder.

Disabled for now as this unit test will segfault with existing code.

Change-Id: I896f9686d9ebcbf027426933adfbea7b8c5d956e
2014-06-20 13:52:06 -07:00