57 Commits

Author SHA1 Message Date
Michael Niedermayer
8158fb129e avcodec/h264_slice: Fix integer overflow in implicit weight computation
Fixes mozilla bug 1230423

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 7cc01c25727a96eaaa0c177234b626e47c8ea491)

Conflicts:

	libavcodec/h264_slice.c

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2016-01-31 00:25:20 +01:00
Michael Niedermayer
20de3b007b avcodec/cabac: Check initial cabac decoder state
Fixes integer overflows
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2340_591e9810c7b09efe501ad84638c9e9f8.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Found-by: xiedingbao (Ticket4727)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8000d484b83aafa752d84fbdbfb352ffe0dc64f8)

Conflicts:

	libavcodec/cabac.h

Conflicts:

	libavcodec/h264_cabac.c
	libavcodec/h264_slice.c
2015-12-06 12:40:50 +01:00
Michael Niedermayer
e9ec9be146 avcodec/h264_slice: Limit max_contexts when slice_context_count is initialized
Fixes out of array access
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2049_f2192b6829ab6e0eefcb035329c03c60.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4ea4d2f438c9a7eba37980c9a87be4b34943e4d5)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-06 12:40:50 +01:00
Michael Niedermayer
47a5cde6ac avcodec/h264_slice: Use w/h from the AVFrame instead of mb_w/h
Fixes out of array access
Fixes: asan_heap-oob_4d5bb0_682_cov_3124593265_Fraunhofer__a_driving_force_in_innovation__small.mp4

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 330863c9f19a23c500ba7901a23f1cc377b353bb)

Conflicts:

	libavcodec/h264_slice.c
2015-08-20 14:38:26 +02:00
Andreas Cadhalpun
5302adb323 h264: er: Copy from the previous reference only if compatible
Also use the frame pixel format instead of the one from the codec
context, which is more robust.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit fdc64a104410f5fcc7f35b62287b0ae502b7061a)

Conflicts:

	libavcodec/h264_slice.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-08-20 14:29:12 +02:00
Michael Niedermayer
6ff5c4cd49 avcodec/h264_slice: Use AVFrame diemensions for grayscale handling
The AVFrame values are closer to the AVFrame bitmap changed instead of
the AVCodecContext values, so this should be more robust

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit aef0e0f009802f1a5e21eb6465498632071e4475)

Conflicts:

	libavcodec/h264_slice.c
2015-08-20 14:29:11 +02:00
Michael Niedermayer
b50fa26885 Merge commit '3b69f245dbe6e2016659a45c4bfe284f6c5ac57e' into release/2.4
* commit '3b69f245dbe6e2016659a45c4bfe284f6c5ac57e':
  h264: Make sure reinit failures mark the context as not initialized

Conflicts:
	libavcodec/h264_slice.c

See: e8714f6f93d1a32f4e4655209960afcf4c185214
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-31 00:13:22 +02:00
Luca Barbato
3b69f245db h264: Make sure reinit failures mark the context as not initialized
Bug-Id: CVE-2015-3417
CC: libav-stable@libav.org
2015-05-30 16:32:02 +02:00
Rainer Hochecker
07a7f38421 h264: avoid unnecessary calls to get_format
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 2197b4018920e5cd7ac465de007b675565687b23)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-25 15:06:54 +02:00
Michael Niedermayer
e617d6f988 avcodec/h264_slice: Dont reset mb_aff_frame per slice
Fixes null pointer dereference
Fixes Ticket4440

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 386601286fed2dff5e1955bc21a0256f6f35ab19)

Conflicts:

	libavcodec/h264_slice.c
(cherry picked from commit ce6d38e9ed0842870f3cd5414937bb6d1f2417d9)
2015-04-25 15:06:54 +02:00
Michael Niedermayer
27993da8e7 avcodec/h264: Fix race between slices where one overwrites data from the next
Fixes non deterministic crash in ticket4408/fuzz2.264
Likely fixes other samples as well

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 43b434210e597d484aef57c4139c3126d22b7e2b)

Conflicts:

	libavcodec/h264.h
	libavcodec/h264_slice.c
(cherry picked from commit dbbc42858e87cdd04e6c3b7694f8b394d4bfcdc6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-25 15:06:54 +02:00
Michael Niedermayer
ba59d92128 avcodec/h264: Only reinit quant tables if a new PPS is allowed
Fixes null pointer dereference
Fixes: signal_sigsegv_3042097_3007_cov_1741463594_non_monotone_timestamps1.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c23a0e77dd492d6c794f89dbff3a438c95745e70)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-29 03:34:21 +02:00
Michael Niedermayer
a8eb02da80 Merge commit '1dbfaa34e615606cb3f1a3ecabb117e354459edc' into release/2.4
* commit '1dbfaa34e615606cb3f1a3ecabb117e354459edc':
  h264: only ref cur_pic in update_thread_context if it is initialized

Conflicts:
	libavcodec/h264_slice.c

See: 0fc01ae33c7712168aab0f98c5715b40da0b5f03
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-28 19:03:51 +01:00
Anton Khirnov
1dbfaa34e6 h264: only ref cur_pic in update_thread_context if it is initialized
It may be empty if the previous thread's decode call did not contain a
valid frame.

(cherry picked from commit 0dea4c77ccf5956561bb8991311b3d834bb5fa40)
Signed-off-by: Anton Khirnov <anton@khirnov.net>
2015-02-21 09:34:17 +01:00
Michael Niedermayer
6005f375aa avcodec/h264_slice: assert that reinit does not occur after the first slice
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 2fd9ce92af43e6dcbc8ed7c26c00b052de48ccad)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-13 20:48:08 +01:00
Michael Niedermayer
4d5beea7a1 avcodec/h264_slice: ignore SAR changes in slices after the first
Fixes race condition and null pointer dereference
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 38d5241b7f36c1571a88517a0650caade16dd5f4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-13 20:48:08 +01:00
Michael Niedermayer
2073ab266e avcodec/h264_slice: Check picture structure before setting the related fields
This might fix a hypothetical race condition

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f111831ed61103f9fa8fdda41473a23da016bdaa)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-13 20:48:08 +01:00
Michael Niedermayer
7b213e88b5 avcodec/h264_slice: Do not change frame_num after the first slice
Fixes potential race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f906982c9411f3062e3ce68013309b37c213c4dd)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-13 20:48:08 +01:00
Michael Niedermayer
9dc8f44829 avcodec/h264: Be more strict on rejecting pps/sps changes
Fixes race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6fafc62b0bd0e206deb77a7aabbf3a370ad80789)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-13 20:48:08 +01:00
Michael Niedermayer
043f326060 avcodec/h264_slice: Clear table pointers to avoid stale pointers
Might fix Ticket3889

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 547fce95858ef83f8c25ae347e3ae3b8ba437fd9)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-02 02:10:51 +01:00
Michael Niedermayer
91ff803a74 Merge commit '09c848855a7bf11a9ab4a69971cced260b17817d' into release/2.4
* commit '09c848855a7bf11a9ab4a69971cced260b17817d':
  h264: Always invoke the get_format() callback

Conflicts:
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-04 23:13:51 +02:00
Rémi Denis-Courmont
09c848855a h264: Always invoke the get_format() callback
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2014-09-26 22:05:34 +02:00
Michael Niedermayer
570397c731 Merge commit 'e87f5e4e5f2e2e36b0b7826d708cda7049877af0'
* commit 'e87f5e4e5f2e2e36b0b7826d708cda7049877af0':
  h264: fully check cropping amount from sps

Conflicts:
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-26 22:01:45 +02:00
Vittorio Giovara
e87f5e4e5f h264: fully check cropping amount from sps
Even if sps.crop is true, the cropping amount may be zero.
Fixes a sample with a valid but broken container cropping.
2014-08-26 13:31:29 -04:00
Carl Eugen Hoyos
cc0acdbd68 lavc/h264_slice: Add a missing newline to an error message. 2014-08-24 15:33:10 +02:00
Michael Niedermayer
1fa35e4352 avcodec/h264_slice: More complete cleanup in h264_slice_header_init()
Fixes null pointer dereference
Fixes Ticket3873

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-23 03:47:14 +02:00
Michael Niedermayer
c1df467d73 Merge commit '835f798c7d20bca89eb4f3593846251ad0d84e4b'
* commit '835f798c7d20bca89eb4f3593846251ad0d84e4b':
  mpegvideo: cosmetics: Lowercase ugly uppercase MPV_ function name prefixes

Conflicts:
	libavcodec/h261dec.c
	libavcodec/intrax8.c
	libavcodec/mjpegenc.c
	libavcodec/mpeg12dec.c
	libavcodec/mpeg12enc.c
	libavcodec/mpeg4videoenc.c
	libavcodec/mpegvideo.c
	libavcodec/mpegvideo.h
	libavcodec/mpegvideo_enc.c
	libavcodec/rv10.c
	libavcodec/x86/mpegvideoenc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-15 20:11:56 +02:00
Diego Biurrun
835f798c7d mpegvideo: cosmetics: Lowercase ugly uppercase MPV_ function name prefixes 2014-08-15 01:26:33 -07:00
Michael Niedermayer
3a2d1465c8 Merge commit '2d60444331fca1910510038dd3817bea885c2367'
* commit '2d60444331fca1910510038dd3817bea885c2367':
  dsputil: Split motion estimation compare bits off into their own context

Conflicts:
	configure
	libavcodec/Makefile
	libavcodec/arm/Makefile
	libavcodec/dvenc.c
	libavcodec/error_resilience.c
	libavcodec/h264.h
	libavcodec/h264_slice.c
	libavcodec/me_cmp.c
	libavcodec/me_cmp.h
	libavcodec/motion_est.c
	libavcodec/motion_est_template.c
	libavcodec/mpeg4videoenc.c
	libavcodec/mpegvideo.c
	libavcodec/mpegvideo_enc.c
	libavcodec/x86/Makefile
	libavcodec/x86/me_cmp_init.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-07-17 23:27:40 +02:00
Diego Biurrun
2d60444331 dsputil: Split motion estimation compare bits off into their own context 2014-07-17 09:07:10 -07:00
Michael Niedermayer
5fce29ef29 Merge commit '2db953f84671997e936f91140ffb5143c1537844'
* commit '2db953f84671997e936f91140ffb5143c1537844':
  h264: K&R formatting cosmetics

Conflicts:
	libavcodec/h264.c
	libavcodec/h264_direct.c
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-07-17 14:00:01 +02:00
Luca Barbato
2db953f846 h264: K&R formatting cosmetics
Signed-off-by: Diego Biurrun <diego@biurrun.de>
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2014-07-17 03:05:24 -07:00
Michael Niedermayer
0dceefc5fa Merge commit '9e500efdbe0deeff1602500ebc229a0a6b6bb1a2'
* commit '9e500efdbe0deeff1602500ebc229a0a6b6bb1a2':
  Add av_image_check_sar() and use it to validate SAR

Conflicts:
	libavcodec/dpx.c
	libavcodec/dvdec.c
	libavcodec/ffv1dec.c
	libavcodec/utils.c
	libavutil/version.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-06-20 22:20:28 +02:00
Justin Ruggles
9e500efdbe Add av_image_check_sar() and use it to validate SAR 2014-06-20 10:39:33 -04:00
Stefano Sabatini
d381109ce8 lavc/h264_slice: clarify informational message in case of disabled slice multi-threaded decoding
Also suggest what can be done to workaround the limitation.
2014-06-20 12:12:46 +02:00
Michael Niedermayer
5bf5e6b1c0 avcodec/h264: Use named identifier for single thread fall-back return from ff_h264_decode_slice_header()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-06-19 19:49:26 +02:00
Michael Niedermayer
7eae8cd870 avcodec/h264: fix frame skip code
Fixes Ticket3475

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-06-16 13:28:52 +02:00
Michael Niedermayer
9025072e6c avcodec/h264_slice: support skipping loop filtering for non key frames
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-06-16 13:18:55 +02:00
Michael Niedermayer
9140d37af2 avcodec/h264: use the correct level for droping non intra frames
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-06-16 13:18:43 +02:00
Michael Niedermayer
0f45e643cc avcodec/h264_slice: fix macro ()
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-05-25 03:45:10 +02:00
Michael Niedermayer
5449239a03 Merge commit '67afcefb35932b420998f6f3fda46c7c85848a3f'
* commit '67afcefb35932b420998f6f3fda46c7c85848a3f':
  lavc: Add new VDA hwaccel

Conflicts:
	configure
	libavcodec/vda.h
	libavcodec/vda_h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-05-12 01:57:02 +02:00
Michael Niedermayer
fa0f7c024e Merge commit 'a871ef0cc936612bdb0e643196be72efcf0c8986'
* commit 'a871ef0cc936612bdb0e643196be72efcf0c8986':
  hwaccel: Rename priv_data_size to frame_priv_data_size

Conflicts:
	libavcodec/avcodec.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-05-11 23:49:13 +02:00
Michael Niedermayer
f2f99f07a5 Merge commit '5c1d7246cd65dc4db1b6dc36e29ce39fc1068f3f'
* commit '5c1d7246cd65dc4db1b6dc36e29ce39fc1068f3f':
  lavc: set AVCodecContext.hwaccel in ff_get_format()

Conflicts:
	libavcodec/mpeg12dec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-05-11 23:28:40 +02:00
Anton Khirnov
67afcefb35 lavc: Add new VDA hwaccel
It leverages the new hwaccel 1.2 features:

- get_buffer2 is never called
- the internal context is automatically initialized/deinitialized

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2014-05-11 15:00:03 +02:00
Anton Khirnov
a871ef0cc9 hwaccel: Rename priv_data_size to frame_priv_data_size
This describes more accurately what this field is for.
2014-05-11 14:59:07 +02:00
Anton Khirnov
5c1d7246cd lavc: set AVCodecContext.hwaccel in ff_get_format()
This way each decoder does not have to do the same thing manually.
2014-05-11 14:59:07 +02:00
Anton Khirnov
632ad2248e lavc: Add an internal wrapper around get_format()
It will be useful in the following commits.
2014-05-11 14:59:07 +02:00
Michael Niedermayer
7ef01a7878 avcodec/h264: mark recovery_cnt==0 frames as keyframes
Fixes Ticket3063

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-05-03 16:01:45 +02:00
Michael Niedermayer
37ce3188de Merge commit '9d76dd952ffd9fffc7f9fb5cc5af6d093ba54521'
* commit '9d76dd952ffd9fffc7f9fb5cc5af6d093ba54521':
  h264: drop commented out cruft

Conflicts:
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-04-30 00:38:13 +02:00
Anton Khirnov
9d76dd952f h264: drop commented out cruft 2014-04-29 14:49:42 +02:00