962 Commits

Author SHA1 Message Date
Janne Grunau
6f4404b24b h264: set parameters from SPS whenever it changes
Fixes a crash in the fuzzed sample sample_varPAR.avi_s26638 with
alternating bit depths.
2014-05-31 20:07:52 -04:00
Anton Khirnov
079758e49a h264: reset next_output_pic earlier in start_frame()
In case start_frame() fails, this potentially invalid frame can still be
output to the caller.

Bug-Id: 672
Bug-Id: debian/741240
Bug-Id: ubuntu/1288206
2014-05-31 20:07:51 -04:00
Anton Khirnov
27ac9585c9 h264: reset data partitioning at the beginning of each decode call
Prevents using GetBitContexts with data from previous calls.

Fixes access to freed memory.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:19 -04:00
Anton Khirnov
35ba079fbf h264: reset ref count if decoding the slice header fails
Otherwise the ER code might try to use some already freed references.

Fixes possible access to freed memory.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:19 -04:00
Anton Khirnov
a7cce9ebf3 h264: reset first_field if frame_start() fails for missing refs
In this case we may not have a current frame, while first_field being
set implies we do.

Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:19 -04:00
Anton Khirnov
51ae8e26af h264: limit allowed pred modes in ff_h264_check_intra_pred_mode() to 3
Higher modes are not allowed for 16x16/chroma, which is what this
function is used for. Otherwise this function would return 0 (vertical
prediction) for invalid higher modes, which could result in invalid
reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:19 -04:00
Anton Khirnov
c4033cd4eb h264: reject mismatching luma/chroma bit depths during sps parsing
There is no point in delaying the check and it avoids bugs with a
half-initialized context.

Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:19 -04:00
Anton Khirnov
7f33a24e82 h264: check that execute_decode_slices() is not called too many times
Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:18 -04:00
Anton Khirnov
3ee26080d6 h264: reset data_partitioning if decoding the slice header for NAL_DPA fails
If it was set before then we can end up trying to decode a slice without
a valid slice header, which can lead to invalid memory access.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:18 -04:00
Anton Khirnov
2cbc8dfedd h264: check buffer size before accessing it
Fixes invalid reads.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-05-31 20:05:18 -04:00
Luca Barbato
4279e0e8d0 h264: Fix a typo from the previous commit
f777504f640260337974848c7d5d7a3f064bbb45 changed a - in +

CC: libav-stable@libav.org
(cherry picked from commit d922c5a5fbaf0b6c73bd8c81ae059bc6e406961c)
(cherry picked from commit 3ce77e04c2ca4b9e7fa6b94b51e8d7c5f188da86)
(cherry picked from commit 8cba6f58c8acaa0ca6749110a2746bbe60ff2dab)
2014-02-28 23:07:40 -05:00
Vittorio Giovara
a6003760bd h264: Lower bound check for slice offsets
And use the value from the specification.

Sample-Id: 00000451-google
Found-by: Mateusz j00ru Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit f777504f640260337974848c7d5d7a3f064bbb45)
(cherry picked from commit 5bd083d0216d9ee649039c84999fb61386536ac1)

Conflicts:
	libavcodec/h264.c

(cherry picked from commit 41380e017afcca3119acb560c08a60a97d416c3c)

Conflicts:
	libavcodec/h264.c
2014-02-28 23:07:40 -05:00
Anton Khirnov
979f77b0dc h264: check that an IDR NAL only contains I slices
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
(cherry picked from commit 8b2e5e42bb9d6a59ede5af2e6df4aaf7750d1195)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 62ed6da016b789eee00e0fff517df4a254e12e5d)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/h264.c
2014-02-02 12:48:04 -05:00
Luca Barbato
9e48d77158 h264: check for luma and chroma bit depth being equal
The decoder assumes a single bit depth for all the planes while
the specification allows different bit depths for luma and chroma.

Avoid the possible problems described in CVE-2013-2277
2013-03-12 20:05:39 +01:00
Janne Grunau
adef01c370 h264: check ref_count validity for num_ref_idx_active_override_flag
Fixes segfault in the fuzzed sample bipbop234.ts_s226407.
CC: libav-stable@libav.org
(cherry-picked from commit 6e5cdf26281945ddea3aaf5eca4d127791f23ca8)
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
2013-01-12 17:59:41 +01:00
Janne Grunau
06312bbb10 h264: check context state before decoding slice data partitions
Fixes mov_h264_aac__Demo_FlagOfOurFathers.mov.SIGSEGV.4e9.656.

Found-by: Mateusz "j00ru" Jurczyk
CC: libav-stable@libav.org
(cherry-picked from commit c1fcf563b13051f280db169ba41c6a1b21b25e08)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:41 +01:00
Janne Grunau
1d98811b95 h264: slice-mt: get last_pic_dropable from master context
Fixes fate-h264-conformance-cvnlfi2_sony_h and smllwebdl.mkv from
https://github.com/OpenELEC/OpenELEC.tv/issues/1557 .

CC: libav-stable@libav.org
(cherry picked from commit a8cb1746c5b6307b2e820f965a7da8d907893b38)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
f1b3cc02ec h264: error out on unset current_picture_ptr for h->current_slice > 0
Fixes a segfault with fuzzed sample sample_varPAR_s11622_r001-02.avi.

CC: libav-stable@libav.org
(cherry picked from commit 0b300daad2f5cb59a7c06dde5ac701685e6edf16)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
6cd92c3880 h264: enable low delay only if no delayed frames were seen
Dropping frames is undesirable but that is the only way by which the
decoder could return to low delay mode. Instead emit a warning and
continue with delayed frames.
Fixes a crash in fuzzed sample nasa-8s2.ts_s20033 caused by a larger
than expected has_b_frames value. Low delay keeps getting re-enabled
from a presumely broken SPS.

CC: libav-stable@libav.org
(cherry picked from commit 706acb558a38eba633056773280155d66c2f4b24)

Conflicts:

	libavcodec/h264.c
2013-01-12 17:59:40 +01:00
Janne Grunau
52adbc0e17 h264: Fix parameters to ff_er_add_slice() call
s->mb_x is reset to zero a couple of lines above. It does not make
sense to call ff_er_add_slice() with 0 as endx when the end of the
macroblock row was reached. Fixes unnecessary and counterproductive
error resilience in https://bugzilla.libav.org/show_bug.cgi?id=394.

(cherry picked from commit e6160bda98641b7d4f86de15761ad2a962f21a36)

Conflicts:

	libavcodec/h264.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-05 00:25:31 +01:00
Jindřich Makovička
9822e3aa52 h264: avoid stuck buffer pointer in decode_nal_units
When decode_nal_units() previously encountered a NAL_END_SEQUENCE,
and there are some junk bytes left in the input buffer, but no start codes,
buf_index gets stuck 3 bytes before the end of the buffer.

This can trigger an infinite loop in the caller code, eg. in
try_decode_trame(), as avcodec_decode_video() then keeps returning zeroes,
with 3 bytes of the input packet still available.

With this change, the remaining bytes are skipped so the whole packet gets
consumed.

CC:libav-stable@libav.org

Signed-off-by: Jindřich Makovička <makovick@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 1a8c6917f68f7378465e18f7615762bfd22704c2)

Conflicts:

	libavcodec/h264.c
2012-10-19 19:23:14 +02:00
Ronald S. Bultje
79fb7bc667 h264: fix deadlocks on incomplete reference frame decoding.
If decoding a second complementary field, and the first was
decoded in our thread, mark decoding of that field as complete.
If decoding fails, mark the decoded field/frame as complete.
Do not allow switching between field modes or field/frame mode
between slices within the same field/frame. Ensure that two
subsequent fields cover top/bottom (rather than top/frame,
bottom/frame or such nonsense situations).

Fixes various deadlocks when decoding samples with errors in
reference frames.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1e26a48fa23ef8e1cbc424667d387184d8155f15)

Fixes Bug 118

Conflicts:
	libavcodec/h264.c

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:00:14 +02:00
Mans Rullgard
e1608014c5 h264: allow cropping to AVCodecContext.width/height
Override the frame size from the SPS with AVCodecContext values
if the latter specify a size smaller by less than one macroblock.
This is required for correct cropping of MOV files from Canon cameras.

Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit 30f515091c323da59c0f1b533703dedca2f4b95d)

Conflicts:

	libavcodec/h264.c
2012-06-10 09:47:45 +02:00
Ronald S. Bultje
746f1594d7 h264: additional protection against unsupported size/bitdepth changes.
Fixes crashes in codepaths not covered by original checks.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit 732f9fcfe54fc9a0a7bbce53fe86b38744c2d301)

Conflicts:

	libavcodec/h264.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-05-04 22:09:27 +02:00
Ronald S. Bultje
4d343a6f47 h264: stricter reference limit enforcement.
Progressive images can have only 16 references, error out if there are
more, since the data is almost certainly corrupt, and the invalid value
will lead to random crashes or invalid writes later on.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit e0febda22d0e0fab094a9c886b0e0f0f662df1ef)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:33:15 +01:00
Michael Niedermayer
a81a6d9c80 h264: improve parsing of broken AVC SPS
Parsing the entire NAL as SPS fixes decoding of some AVC bitstreams
with broken escaping. Since the size of the NAL unit is known and
checked against the buffer end we can parse it entirely without buffer
overreads.

Fixes playback of
http://streams.videolan.org/streams/mp4/Mr_MrsSmith-h264_aac.mp4

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
(cherry picked from commit 3aa661ec561d7a20812b84b353b0d7855ac346c8)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:27:22 +01:00
Alex Converse
48f0eeb2e5 Replace computations of remaining bits with calls to get_bits_left().
(cherry picked from commit 3574a85ce57366ba7429edef93d5cad8640fb68c)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 21:27:16 +01:00
Janne Grunau
d94256d36c Revert "h264: clear trailing bits in partially parsed NAL units"
This reverts commit 729ebb2f185244b0ff06d48edbbbbb02ceb4ed4e.

There was an off-by-one error in the bit mask calculation clearing
actually the last valid bit and causing
http://bugzilla.libav.org/show_bug.cgi?id=227

The broken sample (Mr_MrsSmith-h264_aac.mp4) the commit was fixing
does not work after correcting the off-by-one error.

CC: libav-stable@libav.org
(cherry picked from commit 8a6037c3900875ccab8d553d2cc659bdef2c9d0e)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-14 20:56:55 +01:00
Ronald S. Bultje
750f5baf30 h264: error out on invalid bitdepth.
Fixes invalid reads while initializing the dequant tables, which uses
the bit depth to determine the QP table size.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 0ce4fe482c27abfa7eac503a52fdc50b70ccd871)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-03-04 12:26:06 +01:00
Ronald S. Bultje
4c7879775e h264: disallow constrained intra prediction modes for luma.
Conversion of the luma intra prediction mode to one of the constrained
("alzheimer") ones can happen by crafting special bitstreams, causing
a crash because we'll call a NULL function pointer for 16x16 block intra
prediction, since constrained intra prediction functions are only
implemented for chroma (8x8 blocks).

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 45b7bd7c53b41bc5ff6fc2158831f2b1b1256113)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-02-26 09:09:27 +01:00
Diego Biurrun
55b9ef18e4 cabac: split cabac.h into declarations and function definitions
This fixes standalone compilation of some decoders with --disable-optimizations.
cabac.h defines some inline functions that use symbols from cabac.c.  Without
optimizations these inline functions are not eliminated and linking fails with
references to non-existing symbols.

Splitting the inline functions off into their own header and only #including
it in the places where the inline functions are used allows #including cabac.h
from anywhere without ill effects.
2012-01-12 23:08:23 +01:00
Diego Biurrun
a7e3cb9d32 h264-test: Initialize AVCodecContext.av_class.
This fixes a segfault on startup.

Also remove a commented-out and completely unused variable.
2012-01-07 22:13:10 +01:00
Diego Biurrun
301fb92131 h264: Only use symbols from the SVQ3 decoder under proper conditionals.
Fixes --disable-everything --enable-decoder=h264 --disable-optimizations.
2012-01-06 13:37:53 +01:00
Ronald S. Bultje
400ba1d735 h264: return index in buffer on end-of-sequence.
Fixes hangs if the last packet contains an end-of-sequence NAL unit,
bug 158.
2012-01-03 19:50:22 -08:00
Diego Biurrun
54e68fb3b8 Merge some declarations and initializations.
This fixes compilation failures related to START_TIMER/STOP_TIMER macros and
-Werror=declaration-after-statement.  START_TIMER declares variables and thus
may not be placed after statements outside of a new block.
2012-01-04 01:28:28 +01:00
Janne Grunau
881a5e047d mpegenc: use avctx->slices as number of slices
Adds a new member to MpegEncContext to hold the number of used slice
contexts. Fixes segfaults with '-threads 17 -thread_type slice' and
fate-vsynth{1,2}-mpeg{2,4}thread{,_ilace} with --disable-pthreads.
2012-01-02 19:20:23 +01:00
Alexander Strange
bc6a3bd4a5 h264: Fix a possible overread in decode_nal_units()
Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-12-23 08:32:18 +01:00
Janne Grunau
729ebb2f18 h264: clear trailing bits in partially parsed NAL units
Trailing bits are likely to be non-zero if the NAL unit is truncated.
Clearing the bits make overreads of the bitstream less likely in this
case. Fixes playback of
http://streams.videolan.org/streams/mp4/Mr_MrsSmith-h264_aac.mp4 which
has a forbidden byte sequence of 0x00 0x00 0x00 in it SPS.
2011-12-19 23:14:21 +01:00
Janne Grunau
358ea75e9e Revert "h264: skip start code search if the size of the nal unit is known"
This reverts commit 87eebb3454ff0cd6af6ebf9e1d31bdfd1c3b601b.
2011-12-19 03:24:32 +01:00
Janne Grunau
87eebb3454 h264: skip start code search if the size of the nal unit is known
Start code emulation prevention is only required in Annex B bytestream
packed NAL units. For other coding formats the size is already known.
Looking for a start code prefix can result in false positives like in
http://streams.videolan.org/streams/mp4/Mr_MrsSmith-h264_aac.mp4
which has a false positive in the SPS.
2011-12-18 23:52:53 +01:00
Ronald S. Bultje
0b4c323213 h264: don't drop B-frames after next keyframe on POC reset.
The keyframe after a POC reset may not be the first to be returned to
the user. Therefore, don't reset the expected next POC once we return
a keyframe to the user, but once we know that the next frame in the
return-queue is a keyframe.
2011-12-16 11:24:18 -08:00
Luca Barbato
63ccd46687 lavc: introduce ER_MB_END and ER_MB_ERROR
Simplify a little error resilience calls

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2011-12-13 16:20:58 +01:00
Luca Barbato
5bf2ac2b37 error_resilience: use the ER_ namespace
Add the namespace to {AC_,DC_,MV_}{END,ERROR} macros

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2011-12-13 16:20:58 +01:00
Diego Biurrun
58c42af722 doxygen: misc consistency, spelling and wording fixes 2011-12-12 23:06:23 +01:00
Dustin Brody
5b22d6e132 lavc: convert error_recognition to err_recognition.
Signed-off-by: Anton Khirnov <anton@khirnov.net>
2011-12-12 19:42:50 +01:00
Mans Rullgard
5695ae46f8 h264: remove useless cast
Signed-off-by: Mans Rullgard <mans@mansr.com>
2011-12-12 00:02:48 +00:00
Mans Rullgard
373211d828 Remove extraneous semicolons
These semicolons cause invalid empty top-level declarations.

Signed-off-by: Mans Rullgard <mans@mansr.com>
2011-12-11 17:23:24 +00:00
Diego Biurrun
da9cea77e3 Fix a bunch of common typos. 2011-12-11 00:32:25 +01:00
Ronald S. Bultje
adedd840e2 h264: fix frame reordering code.
Fixes fate-h264-conformance-{mr2_tandberg_e,mr3_tandberg_b} without
requiring -strict 1.
2011-12-03 08:24:27 -08:00
Ronald S. Bultje
2574f08d4c h264: cap max has_b_frames at MAX_DELAYED_PIC_COUNT - 1.
This prevents frame allocation overflows, and fixed
fate-h264-conformance-mr3_tandberg_b with 2 threads.
2011-12-01 12:46:28 -08:00