1579 Commits

Author SHA1 Message Date
Michael Niedermayer
cea2106fb2 avcodec/h264_slice: More complete cleanup in h264_slice_header_init()
Fixes null pointer dereference
Fixes Ticket3873

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1fa35e4352cc39894987e14de464e3d72b55739f)

Conflicts:

	libavcodec/h264_slice.c
2015-06-17 21:50:09 +02:00
Michael Niedermayer
443b9a7d08 Merge commit '964fef3f3ced60e67831549df223bc177e1537c9' into release/2.2
* commit '964fef3f3ced60e67831549df223bc177e1537c9':
  h264: Make sure reinit failures mark the context as not initialized

See: e8714f6f93d1a32f4e4655209960afcf4c185214
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-06-01 13:56:18 +02:00
Luca Barbato
964fef3f3c h264: Make sure reinit failures mark the context as not initialized
Bug-Id: CVE-2015-3417
CC: libav-stable@libav.org
(cherry picked from commit 3b69f245dbe6e2016659a45c4bfe284f6c5ac57e)
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/h264_slice.c
2015-05-31 11:35:35 -04:00
Michael Niedermayer
82d3dd44aa avcodec/h264: Be more tolerant to changing pps id between slices
Fixes Ticket4446

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 98d0c4236c7542c87f012228d3bc88aea67bddc2)

Conflicts:

	libavcodec/h264.c
(cherry picked from commit 0cd0fa9d0baabd2dc0442ed8b53ba65282733b61)

Conflicts:

	libavcodec/h264.c
2015-04-16 16:03:24 +02:00
Michael Niedermayer
2b69da7b5a avcodec/h264: reset the counts in the correct context
Fixes null pointer dereference

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8f8d632220100bfde26587b27da73901b05cb774)

Conflicts:

	libavcodec/h264.c
2015-04-16 16:03:24 +02:00
Michael Niedermayer
0df90898f5 avcodec/h264_slice: Dont reset mb_aff_frame per slice
Fixes null pointer dereference
Fixes Ticket4440

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 386601286fed2dff5e1955bc21a0256f6f35ab19)

Conflicts:

	libavcodec/h264_slice.c
(cherry picked from commit ce6d38e9ed0842870f3cd5414937bb6d1f2417d9)

Conflicts:

	libavcodec/h264_slice.c
2015-04-16 16:03:24 +02:00
Michael Niedermayer
8f026e2b38 avcodec/h264: finish previous slices before switching to single thread mode
Fixes null pointer dereference
Fixes Ticket4438

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c4b2017ba66e1623da9f527704c61c86a6e74844)

Conflicts:

	libavcodec/h264.c
(cherry picked from commit 09cc7aee3f4d1bd1d7107d38520f782c62c14036)

Conflicts:

	libavcodec/h264.c
2015-04-07 12:43:06 +02:00
Michael Niedermayer
3852b172e7 avcodec/h264: Fix race between slices where one overwrites data from the next
Fixes non deterministic crash in ticket4408/fuzz2.264
Likely fixes other samples as well

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 43b434210e597d484aef57c4139c3126d22b7e2b)

Conflicts:

	libavcodec/h264.h
	libavcodec/h264_slice.c
(cherry picked from commit dbbc42858e87cdd04e6c3b7694f8b394d4bfcdc6)

Conflicts:

	libavcodec/h264_slice.c
2015-04-07 12:43:06 +02:00
Michael Niedermayer
5b4e58ed4c avcodec/h264: Fail for invalid mixed IDR / non IDR frames in slice threading mode
Fixes Ticket4408

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fc58d5c43b4c7396fc69081eb0dfe5b6a21cb10d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-07 12:43:06 +02:00
Michael Niedermayer
ec7c1cd733 avcodec/h264: Only reinit quant tables if a new PPS is allowed
Fixes null pointer dereference
Fixes: signal_sigsegv_3042097_3007_cov_1741463594_non_monotone_timestamps1.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c23a0e77dd492d6c794f89dbff3a438c95745e70)

Conflicts:

	libavcodec/h264_slice.c
2015-03-14 14:27:23 +01:00
Michael Niedermayer
019b4b0650 Merge commit '3670942fae7beb2bfde52557ee95eab5f536e624' into release/2.2
* commit '3670942fae7beb2bfde52557ee95eab5f536e624':
  h264: initialize H264Context.avctx in init_thread_copy

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-09 12:24:50 +01:00
Michael Niedermayer
e3654b25cf Merge commit 'fa4604d80580dde45bfce32ebe04a5c13c233895' into release/2.2
* commit 'fa4604d80580dde45bfce32ebe04a5c13c233895':
  h264: Do not share rbsp_buffer across threads

Conflicts:
	libavcodec/h264.c

See: ecbf838c7d81ebd3b89fe75d83ff29150dbda27a
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-09 12:24:04 +01:00
Michael Niedermayer
b837d5bf83 Merge commit '03fbb6ff3d28f639ea5a35aba3c6dca09c17225d' into release/2.2
* commit '03fbb6ff3d28f639ea5a35aba3c6dca09c17225d':
  h264: only ref cur_pic in update_thread_context if it is initialized

Conflicts:
	libavcodec/h264.c

See: 0fc01ae33c7712168aab0f98c5715b40da0b5f03
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-09 12:11:40 +01:00
Anton Khirnov
3670942fae h264: initialize H264Context.avctx in init_thread_copy
This prevents using a wrong (first thread's) AVCodecContext if decoding
a frame in the first pass over all threads fails.

(cherry picked from commit a06b0b1295c51d100101e0ca0434e199ad6de6b5)
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 2686dab45eec54f99866413153aa0b36381e48be)
Signed-off-by: Anton Khirnov <anton@khirnov.net>
2015-02-21 09:41:30 +01:00
Michael Niedermayer
fa4604d805 h264: Do not share rbsp_buffer across threads
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

CC: libav-stable@libav.org
(cherry picked from commit 61928b68dc28e080b8c8191afe5541123c682bbd)
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 06d433366c02ab81a1aaad33d32934b4180d354b)
Signed-off-by: Anton Khirnov <anton@khirnov.net>
2015-02-21 09:41:24 +01:00
Anton Khirnov
03fbb6ff3d h264: only ref cur_pic in update_thread_context if it is initialized
It may be empty if the previous thread's decode call did not contain a
valid frame.

(cherry picked from commit 0dea4c77ccf5956561bb8991311b3d834bb5fa40)
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 1dbfaa34e615606cb3f1a3ecabb117e354459edc)
Signed-off-by: Anton Khirnov <anton@khirnov.net>

Conflicts:
	libavcodec/h264_slice.c
2015-02-21 09:40:43 +01:00
Michael Niedermayer
f0526bc21e avcodec/h264_slice: ignore SAR changes in slices after the first
Fixes race condition and null pointer dereference
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 38d5241b7f36c1571a88517a0650caade16dd5f4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Conflicts:

	libavcodec/h264_slice.c
2015-02-17 19:43:19 +01:00
Michael Niedermayer
0afe061f28 avcodec/h264_slice: Check picture structure before setting the related fields
This might fix a hypothetical race condition

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f111831ed61103f9fa8fdda41473a23da016bdaa)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Conflicts:

	libavcodec/h264_slice.c
2015-02-17 19:43:19 +01:00
Michael Niedermayer
e6093f5b85 avcodec/h264_slice: Do not change frame_num after the first slice
Fixes potential race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f906982c9411f3062e3ce68013309b37c213c4dd)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Conflicts:

	libavcodec/h264_slice.c
2015-02-17 19:43:19 +01:00
Michael Niedermayer
0c9d465e98 avcodec/h264: Be more strict on rejecting pps/sps changes
Fixes race condition
Fixes: signal_sigsegv_1472ac3_468_cov_2915641226_CABACI3_Sony_B.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6fafc62b0bd0e206deb77a7aabbf3a370ad80789)

Conflicts:

	libavcodec/h264_slice.c
2015-02-17 19:43:18 +01:00
Michael Niedermayer
a3dca10470 avcodec/h264: Be more strict on rejecting pps_id changes
Fixes race condition
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 31cc9c04ca386dce289864021982da62190982ab)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-02-17 19:43:18 +01:00
Michael Niedermayer
f1d59a207f avcodec/h264: Check *log2_weight_denom
Fixes undefined behavior
Fixes: signal_sigsegv_14768d2_2248_cov_3629497219_h264_h264___pi_20070614T182942.h264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 61296d41e2de3b41304339e4631dd44c2e15f805)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-01-20 03:27:17 +01:00
Michael Niedermayer
25dc978bb1 avcodec/h264: Clear delayed_pic on deallocation
Fixes use of freed memory

Fixes: case5_av_frame_copy_props.mp4
Found-by: Michal Zalewski <lcamtuf@coredump.cx>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e8714f6f93d1a32f4e4655209960afcf4c185214)

Conflicts:

	libavcodec/h264.c
2015-01-20 03:27:17 +01:00
Michael Niedermayer
8b8d794800 avcodec/h264_slice: Clear table pointers to avoid stale pointers
Might fix Ticket3889

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 547fce95858ef83f8c25ae347e3ae3b8ba437fd9)

Conflicts:

	libavcodec/h264_slice.c
2014-12-08 19:08:30 +01:00
Michael Niedermayer
2d1d053c5d Merge commit 'c7caed88a03567e8777a606f4bd42f093c6b302c' into release/2.2
* commit 'c7caed88a03567e8777a606f4bd42f093c6b302c':
  h264: Always invoke the get_format() callback

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-11-01 15:08:51 +01:00
Michael Niedermayer
26da47a09b avcodec/h264: Check mode before considering mixed mode intra prediction
Fixes out of array read
Fixes: asan_heap-oob_e476fc_2_asan_heap-oob_1333ec6_61_CAMACI3_Sony_C.jsv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9734a7a1de3043f012ad0f1ef11027d9488067e6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-10-05 14:22:09 +02:00
Rémi Denis-Courmont
c7caed88a0 h264: Always invoke the get_format() callback
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2014-09-27 15:49:06 +02:00
Michael Niedermayer
9e1ce9a8ee avcodec/h264: Allow partial escaping
Fixes Ticket3923

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 033a5334badd8af48f13c6fd1e6827f8e3f2c2f3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-09-20 18:40:33 +02:00
Michael Niedermayer
0dc5868f14 Merge commit '67134ad31f1f3bc1515eae129e4368401f7c3342' into release/2.2
* commit '67134ad31f1f3bc1515eae129e4368401f7c3342':
  h264: fix interpretation of interleaved stereo modes

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-08-11 18:51:29 +02:00
Felix Abecassis
67134ad31f h264: fix interpretation of interleaved stereo modes
Column and row frame packing arrangements were inverted.

Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2014-08-07 11:40:08 +01:00
Michael Niedermayer
989adf5ee5 avcodec/h264: in the absence of recovery points, be more tolerant on accepting plain I frames
Fixes: Ticket3652
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 19c9d1e8e71da04c3ac940206619c0a2d01e5193)
2014-06-09 23:50:12 +02:00
Michael Niedermayer
02bae9f013 avcodec/h264: clear cur_pic structure instead of duplicating it in ff_h264_update_thread_context()
Fixes crash

Found-by: iive
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8710ee11d75eebc17e7d63bc6ffb91766933bd68)

Conflicts:

	libavcodec/h264_slice.c
2014-04-10 04:53:00 +02:00
Michael Niedermayer
d8fe695779 avcodec/h264: be more tolerant on what pixel format changes trigger reinits
Fixes Ticket3260

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8e92ff25469f75f5c1fcbb9ba5721cea341ca34a)
2014-03-17 15:45:36 +01:00
Luca Barbato
de187e3e9e h264: Fix a typo from the previous commit
f777504f640260337974848c7d5d7a3f064bbb45 changed a - in +

CC: libav-stable@libav.org
(cherry picked from commit d922c5a5fbaf0b6c73bd8c81ae059bc6e406961c)
2014-03-02 11:42:36 -05:00
Vittorio Giovara
63169474b3 h264: Lower bound check for slice offsets
And use the value from the specification.

Sample-Id: 00000451-google
Found-by: Mateusz j00ru Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit f777504f640260337974848c7d5d7a3f064bbb45)
2014-03-02 11:42:36 -05:00
Michael Niedermayer
5c634cbeb7 Merge remote-tracking branch 'qatar/master'
* qatar/master:
  Give IDCT matrix transpose macro a more descriptive name

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-28 02:08:11 +01:00
Diego Biurrun
f2408ec9d7 Give IDCT matrix transpose macro a more descriptive name
This also avoids a macro name clash and related warning on ARM.
2014-02-27 13:38:00 -08:00
Michael Niedermayer
4f4cc43fd8 avcodec/h264: allow mixing idr and non idr slices with frame threading again
This combination exists in the wild
Fixes Ticket3131

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-27 15:49:25 +01:00
Michael Niedermayer
64bb64f704 avcodec/h264: fix droped frame handling also for threads > 1
Seems i mistakely tested just with threads=1
Fixes part of Ticket3386

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-27 08:07:46 +01:00
Michael Niedermayer
b5005def8a avcodec/h264: avoid using lost frames as references
Fixes Ticket3386

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-27 03:26:03 +01:00
Michael Niedermayer
72e6913140 avcodec/h264: clear chroma planes when flags gray is used
Fixes Ticket3397
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-23 18:22:13 +01:00
Luca Barbato
d922c5a5fb h264: Fix a typo from the previous commit
f777504f640260337974848c7d5d7a3f064bbb45 changed a - in +

CC: libav-stable@libav.org
2014-02-22 12:26:32 +01:00
Michael Niedermayer
8c55ff3933 avcodec/h264: use subsample factors of the used pixel format
Fixes out of array read
Fixes: 1cb91c36c4e55463f14aacb9bdf55b38-asan_heap-oob_106cbce_5617_cov_11212800_h264_mmx_chroma_intra_lf.mp4
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-21 23:53:49 +01:00
Michael Niedermayer
76dd01ecd4 avcodec/h264: fix sign error
regression since f777504f640260337974848c7d5d7a3f064bbb45

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-21 00:33:57 +01:00
Michael Niedermayer
de7b50e9cd Merge remote-tracking branch 'qatar/master'
* qatar/master:
  h264: Lower bound check for slice offsets

Conflicts:
	libavcodec/h264.c

See: 91253839e14cce9793ee93f184cef609ca8195d5
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-21 00:20:55 +01:00
Vittorio Giovara
f777504f64 h264: Lower bound check for slice offsets
And use the value from the specification.

Sample-Id: 00000451-google
Found-by: Mateusz j00ru Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2014-02-20 18:58:38 +01:00
Michael Niedermayer
d0e236292d Merge remote-tracking branch 'qatar/master'
* qatar/master:
  h264: informative error reporting in decode_slice_header()

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-19 02:25:02 +01:00
Luca Barbato
fea6db064b h264: informative error reporting in decode_slice_header()
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2014-02-18 23:47:55 +01:00
Luca Barbato
96f9fbe109 h264: fix slice_type value reported in decode_slice_header()
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
2014-02-18 23:47:32 +01:00
Michael Niedermayer
91253839e1 avcodec/h264: more completely check the loop filter parameters
Fixes out of array read
Fixes: caa65cc01655505705129b677189f036-signal_sigsegv_fdcc43_2681_cov_3043376737_PPH422I5_Panasonic_A.264
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-02-17 01:29:34 +01:00