198 Commits

Author SHA1 Message Date
Michael Niedermayer
ed3d433676 avcodec/cabac: Check initial cabac decoder state
Fixes integer overflows
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2340_591e9810c7b09efe501ad84638c9e9f8.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Found-by: xiedingbao (Ticket4727)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 8000d484b83aafa752d84fbdbfb352ffe0dc64f8)

Conflicts:

	libavcodec/cabac.h
2015-12-14 16:51:00 +01:00
Michael Niedermayer
54e94522b8 avcodec/h264_slice: Limit max_contexts when slice_context_count is initialized
Fixes out of array access
Fixes: 1430e9c43fae47a24c179c7c54f94918/signal_sigsegv_421427_2049_f2192b6829ab6e0eefcb035329c03c60.264

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 4ea4d2f438c9a7eba37980c9a87be4b34943e4d5)

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-14 16:51:00 +01:00
Michael Niedermayer
ab6f4bc064 avcodec/h264_slice: Disable slice threads if there are multiple access units in a packet
Fixes null pointer dereference
Fixes part of Ticket4977

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit 9368d2da3d33cac845f2fdf663df500b53625c5e)

Conflicts:

	libavcodec/h264_slice.c
2015-11-12 02:55:46 +01:00
Michael Niedermayer
65aac419e5 avcodec/h264_slice: Use w/h from the AVFrame instead of mb_w/h
Fixes out of array access
Fixes: asan_heap-oob_4d5bb0_682_cov_3124593265_Fraunhofer__a_driving_force_in_innovation__small.mp4

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 330863c9f19a23c500ba7901a23f1cc377b353bb)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-07-18 20:23:45 +02:00
Andreas Cadhalpun
3ba55ea4ae h264: er: Copy from the previous reference only if compatible
Also use the frame pixel format instead of the one from the codec
context, which is more robust.

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Reviewed-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
(cherry picked from commit fdc64a104410f5fcc7f35b62287b0ae502b7061a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-06-18 22:09:01 +02:00
Michael Niedermayer
5a0862af55 avcodec/h264_slice: Use AVFrame diemensions for grayscale handling
The AVFrame values are closer to the AVFrame bitmap changed instead of
the AVCodecContext values, so this should be more robust

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit aef0e0f009802f1a5e21eb6465498632071e4475)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-06-10 12:05:42 +02:00
Michael Niedermayer
e9d646f44c Merge commit '0181ae9af2de1526464d23209b82e6674d362f5d'
* commit '0181ae9af2de1526464d23209b82e6674d362f5d':
  h264: Make sure reinit failures mark the context as not initialized

Conflicts:
	libavcodec/h264_slice.c

See: e8714f6f93d1a32f4e4655209960afcf4c185214
Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-30 21:33:00 +02:00
Luca Barbato
0181ae9af2 h264: Make sure reinit failures mark the context as not initialized
Bug-Id: CVE-2015-3417
CC: libav-stable@libav.org
2015-05-30 16:28:32 +02:00
Michael Niedermayer
1b236541a6 avcodec/h264: Fix HWACCEL_MAX for D3D11
Found-by: philipl
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-26 05:23:29 +02:00
Michael Niedermayer
947b74ee7d Merge commit 'd8039ef8d221ea273aa4f1e62e5df21bf618c772'
* commit 'd8039ef8d221ea273aa4f1e62e5df21bf618c772':
  D3D11va: add a Direct3D11 video decoder similar to DXVA2

Conflicts:
	Changelog
	configure
	libavcodec/Makefile
	libavcodec/allcodecs.c
	libavcodec/dxva2_vc1.c
	libavcodec/version.h
	libavutil/pixdesc.c
	libavutil/pixfmt.h
	libavutil/version.h

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-25 22:24:34 +02:00
Steve Lhomme
d8039ef8d2 D3D11va: add a Direct3D11 video decoder similar to DXVA2
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2015-05-25 11:47:08 +02:00
Michael Niedermayer
47cbcf20d6 avcodec/h264_slice: Fix ranges in assert
Fixes CID1297592, CID1297593

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-05-15 00:30:32 +02:00
Michael Niedermayer
36f862e04c Merge commit 'a0f2946068c62e18cb05ac25c0df3d86077251a6'
* commit 'a0f2946068c62e18cb05ac25c0df3d86077251a6':
  h264: use properly allocated AVFrames

Conflicts:
	libavcodec/h264.c
	libavcodec/h264.h
	libavcodec/h264_refs.c
	libavcodec/h264_slice.c
	libavcodec/svq3.c
	libavcodec/vda_h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 16:35:24 +02:00
Michael Niedermayer
07a79cf869 Merge commit '7a4f74eed51f914e9bbfebaffd4a92ac6791f819'
* commit '7a4f74eed51f914e9bbfebaffd4a92ac6791f819':
  h264: embed the DPB in the context

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 15:33:46 +02:00
Michael Niedermayer
4a95cd5d9b avcodec/h264_slice: Also copy x264_build in ff_h264_update_thread_context()
Fixes fate-h264-lossless

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 15:21:28 +02:00
Michael Niedermayer
bd7dce594e Merge commit '88c612e3a4d5b584e2d9f6e2541de78d67bdfb9f'
* commit '88c612e3a4d5b584e2d9f6e2541de78d67bdfb9f':
  h264: merge the init and reinit paths in update_thread_context()

Conflicts:
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 15:21:20 +02:00
Michael Niedermayer
cc87493f06 Merge commit '8a66fd40260b7aae6226d68c4dbad43b05a8e524'
* commit '8a66fd40260b7aae6226d68c4dbad43b05a8e524':
  h264: drop the reinit parameter from init_slice_header()

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 13:55:58 +02:00
Michael Niedermayer
51f7ba9cee Merge commit 'c9ccbc7333eddd025ebbde5cc4f27d68a950c623'
* commit 'c9ccbc7333eddd025ebbde5cc4f27d68a950c623':
  h264: update the current bit depth/chroma type in init_slice_header()

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 13:48:45 +02:00
Michael Niedermayer
16b90d0bcb Merge commit '29dcde2b87559b72957454a7e9bcd5227d8057ba'
* commit '29dcde2b87559b72957454a7e9bcd5227d8057ba':
  h264: copy nal_length_size in update_thread_context

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 13:41:24 +02:00
Michael Niedermayer
d3a5dba81b Merge commit '11f024ef0ab923ed8680fc35a087d576e549c849'
* commit '11f024ef0ab923ed8680fc35a087d576e549c849':
  h264: move freeing the escaped RBSP buffer to free_context()

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 13:26:33 +02:00
Michael Niedermayer
ab7ad5e6a1 Merge commit 'ee62b364be0c30cba83b5ff10a3ca8c3e866ade6'
* commit 'ee62b364be0c30cba83b5ff10a3ca8c3e866ade6':
  h264: eliminate ff_h264_set_parameter_from_sps()

Conflicts:
	libavcodec/h264.c
	libavcodec/h264.h
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 13:23:45 +02:00
Michael Niedermayer
580c86925d Merge commit '54986d6db9116ef9704b1ce8414ffdb7f1ca127f'
* commit '54986d6db9116ef9704b1ce8414ffdb7f1ca127f':
  h264: move context reinit lower down in update_thread_context()

Conflicts:
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 12:11:09 +02:00
Michael Niedermayer
cd63252f1f Merge commit 'a6cd154463bea7eb56d28192db4c8c6d83f67fd7'
* commit 'a6cd154463bea7eb56d28192db4c8c6d83f67fd7':
  h264: move the DPB init/uninit to init_context()/free_context()

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 11:48:39 +02:00
Michael Niedermayer
e8a460672f Merge commit '9d33bab583a82cf12286c65258a29c6888e1ff98'
* commit '9d33bab583a82cf12286c65258a29c6888e1ff98':
  h264: drop H264Context.ouputed_poc

Conflicts:
	libavcodec/h264.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 11:38:42 +02:00
Michael Niedermayer
0d5e615c10 Merge commit '27b0e6ebfd47b0c11156c18b90fa8c571f0f60c3'
* commit '27b0e6ebfd47b0c11156c18b90fa8c571f0f60c3':
  h264: drop needs_realloc

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-29 11:28:28 +02:00
Anton Khirnov
a0f2946068 h264: use properly allocated AVFrames 2015-04-29 05:52:58 +02:00
Anton Khirnov
88c612e3a4 h264: merge the init and reinit paths in update_thread_context()
There is no real need to handle the init case specially, everything
necessary is initialized in the reinit code as well.
2015-04-29 05:52:58 +02:00
Anton Khirnov
8a66fd4026 h264: drop the reinit parameter from init_slice_header()
It is only used to decide whether to call free_tables(), but that
function is safe to call on an uninitialized context as well.
2015-04-29 05:52:58 +02:00
Anton Khirnov
c9ccbc7333 h264: update the current bit depth/chroma type in init_slice_header()
With frame threading, it is currently only updated in the context where
the change occurs, but not in any other contexts.
2015-04-29 05:52:58 +02:00
Anton Khirnov
29dcde2b87 h264: copy nal_length_size in update_thread_context
It does not make sense to copy is_avc without copying this as well. This
patch should not change anything for now, but will have an effect in
later commits.
2015-04-29 05:52:58 +02:00
Anton Khirnov
11f024ef0a h264: move freeing the escaped RBSP buffer to free_context()
It does not logically belong in free_tables(), since it's not allocated
in alloc_tables() and its size has nothing to do with the frame size.
2015-04-29 05:52:58 +02:00
Anton Khirnov
ee62b364be h264: eliminate ff_h264_set_parameter_from_sps()
That function currently does two things -- reinitializing the DSP
contexts and setting low_delay based on the SPS values.

The former more appropriately belongs in h264_slice_header_init(), while
the latter only really makes sense in decode_slice_header().

The third call to ff_h264_set_parameter_from_sps(), done immediately
after parsing a new SPS, appears to serve no useful purpose, so it is
just dropped.

Also, drop now unneeded H264Context.cur_chroma_format_idc.
2015-04-29 05:52:57 +02:00
Anton Khirnov
54986d6db9 h264: move context reinit lower down in update_thread_context()
It uses some fields from the SPS, which is not yet set where the reinit
is called currently.
2015-04-29 05:52:57 +02:00
Anton Khirnov
a6cd154463 h264: move the DPB init/uninit to init_context()/free_context()
Currently, the DPB is initialized in alloc_tables() and uninitialized in
free_tables(), but those functions manage frame size-dependent
variables, so DPB management does not logically belong in there.

Since we want the init/uninit to happen exactly once per the context
lifetime, init_context()/free_context() are the proper place for this
code.
2015-04-29 05:52:57 +02:00
Anton Khirnov
9d33bab583 h264: drop H264Context.ouputed_poc
It is only set, but never used for anything.
2015-04-29 05:52:57 +02:00
Anton Khirnov
27b0e6ebfd h264: drop needs_realloc
It is not needed anymore since switching to refcounted frames.
2015-04-29 05:52:57 +02:00
Michael Niedermayer
febbb3cbd3 avcodec/h264_slice: Change a few asserts to av_assert*
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-26 12:59:30 +02:00
Michael Niedermayer
8f7b022c8c Merge commit '6a85dfc830f51f1f5c2d36d4182d265c1ea3ba25'
* commit '6a85dfc830f51f1f5c2d36d4182d265c1ea3ba25':
  lavc: Replace av_dlog and tprintf with internal macros

Conflicts:
	libavcodec/aacdec.c
	libavcodec/audio_frame_queue.c
	libavcodec/bitstream.c
	libavcodec/dcadec.c
	libavcodec/dnxhddec.c
	libavcodec/dvbsubdec.c
	libavcodec/dvdec.c
	libavcodec/dvdsubdec.c
	libavcodec/get_bits.h
	libavcodec/gifdec.c
	libavcodec/h264.h
	libavcodec/h264_cabac.c
	libavcodec/h264_cavlc.c
	libavcodec/h264_loopfilter.c
	libavcodec/h264_refs.c
	libavcodec/imc.c
	libavcodec/interplayvideo.c
	libavcodec/jpeglsdec.c
	libavcodec/libopencore-amr.c
	libavcodec/mjpegdec.c
	libavcodec/mpeg12dec.c
	libavcodec/mpegvideo_enc.c
	libavcodec/mpegvideo_parser.c
	libavcodec/pngdec.c
	libavcodec/ratecontrol.c
	libavcodec/rv10.c
	libavcodec/svq1dec.c
	libavcodec/vqavideo.c
	libavcodec/wmadec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-20 04:10:10 +02:00
Vittorio Giovara
6a85dfc830 lavc: Replace av_dlog and tprintf with internal macros 2015-04-19 12:41:59 +01:00
wm4
f4f3065c69 h264: remove bogus YCgCo warning message
YCgCo decoding works just fine. It depends on the API user what is done
with the output. Some API users might support it, some not. Some users
might support it under certain circumstances only.

It is not the job of the decoder to print this message. If the API user
supports it, this message is extremely unhelpful.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-13 13:46:38 +02:00
Michael Niedermayer
7be2d2a70c avcodec/h264: Use only one ERContext
This fixes slice threads with error concealment

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-09 17:42:04 +02:00
Michael Niedermayer
845ab37b05 avcodec/h264_slice: Fix uninitialized variable
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-08 13:05:38 +02:00
Michael Niedermayer
386601286f avcodec/h264_slice: Dont reset mb_aff_frame per slice
Fixes null pointer dereference
Fixes Ticket4440

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-08 12:29:47 +02:00
Michael Niedermayer
43b434210e avcodec/h264: Fix race between slices where one overwrites data from the next
Fixes non deterministic crash in ticket4408/fuzz2.264
Likely fixes other samples as well

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-07 00:40:21 +02:00
Michael Niedermayer
98c5db89bb avcodec/h264_slice: make current_slice reset code more strict
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-04 17:00:12 +02:00
Michael Niedermayer
6da7625cc3 Merge commit 'a4d34e218f548d381e09c483e8dc6ad18a8d571c'
* commit 'a4d34e218f548d381e09c483e8dc6ad18a8d571c':
  h264: disable ER by default

Conflicts:
	libavcodec/h264.c
	libavcodec/h264_picture.c
	libavcodec/h264_slice.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2015-04-04 00:52:54 +02:00
Anton Khirnov
a4d34e218f h264: disable ER by default
The way it is currently designed is fundamentally unsafe and cannot be
reasonably fixed without completely rewriting it.
2015-04-03 13:14:43 +02:00
Rainer Hochecker
8b152c355f h264: avoid unnecessary calls to get_format
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-31 12:14:39 +02:00
Michael Niedermayer
cd5a48e549 avcodec/h264_slice: Put CONFIG_GRAY first in if()
This is more consistent

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-26 15:41:27 +01:00
Michael Niedermayer
354db19ff4 avcodec/h264: Remove bits_per_raw_sample hacks
Fixes assertion failure

Fixes: f12c79769501ee471a5c731b8304ffcb/signal_sigabrt_7ffff6ac8cc9_3766_cov_4267830138_undemuxable_h264.ts

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2015-03-25 04:14:38 +01:00