Commit Graph

1538 Commits

Author SHA1 Message Date
Michael Niedermayer
72e6913140 avcodec/h264: clear chroma planes when flags gray is used
Fixes Ticket3397
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-23 18:22:13 +01:00
Michael Niedermayer
8c55ff3933 avcodec/h264: use subsample factors of the used pixel format
Fixes out of array read
Fixes: 1cb91c36c4e55463f14aacb9bdf55b38-asan_heap-oob_106cbce_5617_cov_11212800_h264_mmx_chroma_intra_lf.mp4
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-21 23:53:49 +01:00
Michael Niedermayer
76dd01ecd4 avcodec/h264: fix sign error
regression since f777504f64

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-21 00:33:57 +01:00
Michael Niedermayer
de7b50e9cd Merge remote-tracking branch 'qatar/master'
* qatar/master:
  h264: Lower bound check for slice offsets

Conflicts:
	libavcodec/h264.c

See: 91253839e1
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-21 00:20:55 +01:00
Vittorio Giovara
f777504f64 h264: Lower bound check for slice offsets
And use the value from the specification.

Sample-Id: 00000451-google
Found-by: Mateusz j00ru Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2014-02-20 18:58:38 +01:00
Michael Niedermayer
d0e236292d Merge remote-tracking branch 'qatar/master'
* qatar/master:
  h264: informative error reporting in decode_slice_header()

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-19 02:25:02 +01:00
Luca Barbato
fea6db064b h264: informative error reporting in decode_slice_header()
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2014-02-18 23:47:55 +01:00
Luca Barbato
96f9fbe109 h264: fix slice_type value reported in decode_slice_header()
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2014-02-18 23:47:32 +01:00
Michael Niedermayer
91253839e1 avcodec/h264: more completely check the loop filter parameters
Fixes out of array read
Fixes: caa65cc01655505705129b677189f036-signal_sigsegv_fdcc43_2681_cov_3043376737_PPH422I5_Panasonic_A.264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-17 01:29:34 +01:00
Michael Niedermayer
a7eb93b367 Merge commit '15210354cf27cf4e24d91f84d66cf471511ce718'
* commit '15210354cf27cf4e24d91f84d66cf471511ce718':
  h264: drop outdated comments

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-14 14:07:19 +01:00
Michael Niedermayer
99b12357f4 Merge commit '3a0576702825423abecb32627c530dbc4c0f73bc'
* commit '3a0576702825423abecb32627c530dbc4c0f73bc':
  h264: store current_sps_id inside the current sps

Conflicts:
	libavcodec/h264.c
	libavcodec/h264_ps.c

The current_sps_id is not removed as it used in security related code.

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-14 13:58:02 +01:00
Michael Niedermayer
60b46a00c6 Merge commit '73e8fab31dc19c4371499e612856accbc00b2820'
* commit '73e8fab31dc19c4371499e612856accbc00b2820':
  h264: print values in case of error

Conflicts:
	libavcodec/h264.c
	libavcodec/h264_ps.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-14 13:35:45 +01:00
Vittorio Giovara
15210354cf h264: drop outdated comments 2014-02-14 05:08:37 +01:00
Vittorio Giovara
3a05767028 h264: store current_sps_id inside the current sps
In preparation for MVC support.
2014-02-14 05:05:46 +01:00
Vittorio Giovara
73e8fab31d h264: print values in case of error
Also make error style consistent and drop redundant information.
2014-02-14 05:05:35 +01:00
Michael Niedermayer
84873794ad Merge commit 'f795a8a8bf5e312dad2c2829c543b9d309376ca1'
* commit 'f795a8a8bf5e312dad2c2829c543b9d309376ca1':
  h264: make context_count unsigned

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-14 01:24:26 +01:00
Janne Grunau
f795a8a8bf h264: make context_count unsigned
Removes the bogus but scary looking warning 'libavcodec/h264.c:4529:49:
warning: array subscript is below array bounds [-Warray-bounds]'.
2014-02-12 15:48:03 +01:00
Michael Niedermayer
c93e691369 Merge commit '1f097d168d9cad473dd44010a337c1413a9cd198'
* commit '1f097d168d9cad473dd44010a337c1413a9cd198':
  h264: reset data partitioning at the beginning of each decode call

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-04 16:44:32 +01:00
Anton Khirnov
1f097d168d h264: reset data partitioning at the beginning of each decode call
Prevents using GetBitContexts with data from previous calls.

Fixes access to freed memory.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-02-04 11:26:17 +01:00
Michael Niedermayer
8a3b85f3a7 avcodec/h264: update current_sps & sps->new only after the whole slice header decoder and init code finished
This avoids them being cleared before the full initialization finished

Fixes out of array read
Fixes: asan_heap-oob_f0c5e6_7071_cov_1605985132_mov_h264_aac__Demo_FlagOfOurFathers.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-04 04:49:25 +01:00
Michael Niedermayer
e708424b70 avcodec/h264: Disallow pps_id changing between slices
Such changes are forbidden in H.264 and lead to race conditions

Fixes out of array read
Fixes: signal_sigsegv_f9796a_1613_cov_3114610371_FM1_BT_B.h264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-04 00:07:27 +01:00
Michael Niedermayer
1a96b27ebf avcodec/h264: clear dequant8_coeff pointers if 8x8 mode is not enabled
This prevents stale pointers from being left

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-03 22:15:32 +01:00
Michael Niedermayer
965fa6b0d9 Merge commit 'fb0c9d41d685abb58575c5482ca33b8cd457c5ec'
* commit 'fb0c9d41d685abb58575c5482ca33b8cd457c5ec':
  avutil: remove timer.h include from internal.h

Conflicts:
	libavcodec/ffv1dec.c
	libavutil/internal.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-26 01:54:55 +01:00
Janne Grunau
fb0c9d41d6 avutil: remove timer.h include from internal.h
Added libavutil/timer.h include to all files with {START,STOP}_TIMER.
2014-01-25 21:50:20 +01:00
Janne Grunau
ea49f60523 h264: skip chroma edges at the picture boundary while deblocking 4:4:4
This handles macroblock edges for the chroma components in the same way
as for the luma compoment for 4:4:4 streams. The Spec explicitly states
that the deblocking filter is not applied to edges at the boundary of
the picture.

Signed-off-by: Janne Grunau <janne-libav@jannau.net>
2014-01-22 13:44:28 +01:00
Michael Niedermayer
a52fbe5119 Merge remote-tracking branch 'qatar/master'
* qatar/master:
  h264: check that an IDR NAL only contains I slices

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-18 23:13:33 +01:00
Michael Niedermayer
87e46dd5fb Merge commit '0652e024c680420d298cdf3719d0a0c030173fe3'
* commit '0652e024c680420d298cdf3719d0a0c030173fe3':
  h264: reset ref count if decoding the slice header fails

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-18 22:51:59 +01:00
Michael Niedermayer
329610303a Merge commit '00dbff4c3e048b4abd01bf805725aabff0fa5ee1'
* commit '00dbff4c3e048b4abd01bf805725aabff0fa5ee1':
  h264: do not call field_end if we do not have a current picture

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-18 21:51:40 +01:00
Michael Niedermayer
f2cb3b36ac Merge commit '7f0e81db3c4ee6f8ce15058bafa72ce928a89f3f'
* commit '7f0e81db3c4ee6f8ce15058bafa72ce928a89f3f':
  h264: limit allowed pred modes in ff_h264_check_intra_pred_mode() to 3

Conflicts:
	libavcodec/h264.c

See: d6a33f5d20
See: 2005fddcbb
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-18 21:44:18 +01:00
Michael Niedermayer
357a733f91 Merge commit 'd1b3fabe6945e511bb20fc9ca52b47eb952526ee'
* commit 'd1b3fabe6945e511bb20fc9ca52b47eb952526ee':
  h264: reset first_field if frame_start() fails for missing refs

See: d7599bd8e2
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-18 21:29:37 +01:00
Anton Khirnov
8b2e5e42bb h264: check that an IDR NAL only contains I slices
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-18 20:53:31 +01:00
Anton Khirnov
0652e024c6 h264: reset ref count if decoding the slice header fails
Otherwise the ER code might try to use some already freed references.

Fixes possible access to freed memory.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-18 20:51:04 +01:00
Anton Khirnov
00dbff4c3e h264: do not call field_end if we do not have a current picture
Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-18 20:42:21 +01:00
Anton Khirnov
7f0e81db3c h264: limit allowed pred modes in ff_h264_check_intra_pred_mode() to 3
Higher modes are not allowed for 16x16/chroma, which is what this
function is used for. Otherwise this function would return 0 (vertical
prediction) for invalid higher modes, which could result in invalid
reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-18 20:41:59 +01:00
Anton Khirnov
d1b3fabe69 h264: reset first_field if frame_start() fails for missing refs
In this case we may not have a current frame, while first_field being
set implies we do.

Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-18 20:41:24 +01:00
Michael Niedermayer
64591f8f86 Merge commit '9eef9eb3014b2ed9c3ff4aac510a9f04edb555cf'
* commit '9eef9eb3014b2ed9c3ff4aac510a9f04edb555cf':
  h264: check that execute_decode_slices() is not called too many times

Conflicts:
	libavcodec/h264.c

The check is replaced by an assert() as the mb index should not ever go out
of bounds.

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-06 16:39:38 +01:00
Michael Niedermayer
a60abb1ee0 Merge commit 'bfd26b7ce6efea594f2b99441d900419df3af638'
* commit 'bfd26b7ce6efea594f2b99441d900419df3af638':
  h264: reject mismatching luma/chroma bit depths during sps parsing

Conflicts:
	libavcodec/h264_ps.c

See: bdeb61ccc6
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-06 16:28:55 +01:00
Michael Niedermayer
98dcbb47fa avcodec/h264: reset list_count too in case of error in ff_set_ref_count()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-06 16:17:45 +01:00
Michael Niedermayer
7ee8a1c562 Merge commit '9a026c72982faf20e1c8dfbe48f0b312cdea69c8'
* commit '9a026c72982faf20e1c8dfbe48f0b312cdea69c8':
  h264: rebuild the default ref list if the reference count changes

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-06 16:17:38 +01:00
Anton Khirnov
9eef9eb301 h264: check that execute_decode_slices() is not called too many times
Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-06 08:25:25 +01:00
Anton Khirnov
bfd26b7ce6 h264: reject mismatching luma/chroma bit depths during sps parsing
There is no point in delaying the check and it avoids bugs with a
half-initialized context.

Fixes invalid reads.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-06 08:23:45 +01:00
Anton Khirnov
9a026c7298 h264: rebuild the default ref list if the reference count changes
Fixes possible access to freed memory.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-06 08:23:17 +01:00
Michael Niedermayer
8e6af036b9 Merge commit '58312b2472d3a44d7458865c459d59ef2e02bf1a'
* commit '58312b2472d3a44d7458865c459d59ef2e02bf1a':
  h264: reset data_partitioning if decoding the slice header for NAL_DPA fails

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-01-04 02:06:21 +01:00
Anton Khirnov
58312b2472 h264: reset data_partitioning if decoding the slice header for NAL_DPA fails
If it was set before then we can end up trying to decode a slice without
a valid slice header, which can lead to invalid memory access.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC:libav-stable@libav.org
2014-01-03 16:42:02 +01:00
Dale Curtis
4feca2214a h264: Clear ERContext.cur_pic when unref'ing current picture.
Signed-off-by: Dale Curtis <dalecurtis@chromium.org>
2014-01-02 23:49:06 +01:00
Michael Niedermayer
74a9c92840 Merge remote-tracking branch 'qatar/master'
* qatar/master:
  h264: namespace the decode function

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-12-29 11:37:42 +01:00
Michael Niedermayer
6ea05ef278 avcodec/h264: remove unused variable
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-12-28 18:19:46 +01:00
Luca Barbato
4d2bb28931 h264: namespace the decode function
Make much easier debugging.
2013-12-27 07:48:13 +01:00
Michael Niedermayer
e3578fd525 Merge commit '598ce4ab4f1893e0661fc038101487e511937877'
* commit '598ce4ab4f1893e0661fc038101487e511937877':
  h264: call av_frame_unref() instead of avcodec_get_frame_defaults().

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-12-12 00:25:31 +01:00
Anton Khirnov
598ce4ab4f h264: call av_frame_unref() instead of avcodec_get_frame_defaults().
This is a temporary workaround to allow deprecating
avcodec_get_frame_defaults(). The proper solution will be using a
properly allocated AVFrame in Picture.
2013-12-11 20:39:55 +01:00