15141 Commits

Author SHA1 Message Date
Anton Khirnov
96cf80609a dfa: check for invalid access in decode_wdlt().
This can happen when the number of skipped lines is not consistent with
the number of coded lines.

Reported-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 3623589edc7b1257bb45aa9e52c9631e133f22b6)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-05-09 20:05:52 +02:00
Michael Niedermayer
9c713f30e4 parser: fix large overreads
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>

(cherry picked from commit 096abfa15052977eed93f0b5e01afd2d47c53c1f)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2013-05-09 11:29:05 +02:00
Mans Rullgard
799000af70 dsputil: fix invalid array indexing
Indexing outside an array is invalid and causes errors with
gcc 4.8.

Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit 0a07f2b346433a9a2677c69c6b29a1a827e39109)

Signed-off-by: Diego Biurrun <diego@biurrun.de>
2013-05-09 11:29:05 +02:00
Xi Wang
78889be3fb atrac3: avoid oversized shifting in decode_bytes()
When `off' is 0, `0x537F6103 << 32' in the following expression invokes
undefined behavior, the result of which is not necessarily 0.

    (0x537F6103 >> (off * 8)) | (0x537F6103 << (32 - (off * 8)))

Avoid oversized shifting.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

(cherry picked from commit eba1ff31304e407db3cefd7532108408f364367b)

Conflicts:
	libavcodec/atrac3.c
2013-05-09 11:29:05 +02:00
Luca Barbato
6d4d186e9e shorten: use the unsigned type where needed
get_uint returns an unsigned value, use an unsigned to store
blocksize to make sure the comparison logic is correct and report
correctly the error for the channel count not supported.

CC: libav-stable@libav.org

(cherry picked from commit 5cf7c72757779a740e897a97710aac044fe5258c)
(cherry picked from commit 88089eecfd7e604d40d078b4f4206c647cb2e2b4)
(cherry picked from commit f42d03746afe491dd02bb6372961e85e78299864)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/shorten.c
2013-05-09 11:29:01 +02:00
Reinhard Tartler
5ebb5a32bd shorten: report meaningful errors
(cherry picked from commit 4c364eb2b856fc33cf7b42f7c7b979e69fde5f3a)
(cherry picked from commit 0daf1428e82926dc5a8c72a0ff4c93aaa8a84ed9)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Conflicts:
	libavcodec/shorten.c
2013-05-09 11:28:28 +02:00
Reinhard Tartler
a694b2b158 shorten: set invalid channels count to 0
Prevent the loop shorten_decode_close from writing and freeing out of
the array boundary.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

CC: libav-stable@libav.org

(cherry picked from commit c10da30d8426a1f681d99a780b6e311f7fb4e5c5)
(cherry picked from commit 21d568be179c54a1596d1377b4da7fbe755bfe7f)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Conflicts:
	libavcodec/shorten.c
2013-05-09 11:28:28 +02:00
Justin Ruggles
d785f69401 shorten: validate that the channel count in the header is not <= 0
(cherry picked from commit 4c53f4aed3edfa58360c7a2a468782eae31d3176)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/shorten.c
2013-05-09 11:28:25 +02:00
Anton Khirnov
5025dbc577 wmaprodec: return an error, not 0, when the input is too small.
Returning 0 may result in an infinite loop in valid calling programs. A
decoder should never return 0 without producing any output.

CC:libav-stable@libav.org
(cherry picked from commit 4c0080b7e7d501e2720d2a61f5186a18377f9d63)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 60dd8b5733f9ec4919fbc732ace1be8184dde880)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-05-09 11:20:11 +02:00
Michael Niedermayer
d86a5ce03f vorbisdec: Error on bark_map_size equal to 0.
The value is used to calculate output LSP curve and a division by zero
and out of array accesses would occur.

CVE-2013-0894

CC: libav-stable@libav.org

Reported-by: Dale Curtis <dalecurtis@chromium.org>
Found-by: inferno@chromium.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 11dcecfcca0eca1a571792c4fa3c21fb2cfddddc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 494ddd377ada76ed555f7a3f49391455daa099c9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-05-09 11:20:11 +02:00
Reinhard Tartler
7d4c38d58d h264: check for luma and chroma bit depth being equal
The decoder assumes a single bit depth for all the planes while
the specification allows different bit depths for luma and chroma.

Avoid the possible problems described in CVE-2013-2277

Conflicts:
	libavcodec/h264.c
2013-05-09 11:20:11 +02:00
Reinhard Tartler
7e6625a9af xxan: fix invalid memory access in xan_decode_frame_type0()
The loop a few lines below the xan_unpack() call accesses up to
dec_size * 2 bytes into y_buffer, so dec_size must be limited to
buffer_size / 2.

CC:libav-stable@libav.org
(cherry picked from commit 8a49d2bcbe7573bb4b765728b2578fac0d19763f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 62a657de168cf501acb23d48cc1aa00793dc83f3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>

Conflicts:
	libavcodec/xxan.c
2013-05-09 11:20:11 +02:00
Anton Khirnov
f13f6f82c6 wmadec: require block_align to be set.
Avoids an infinite loop in the calling programs with decoder not
consuming any input and not returning output.

CC:libav-stable@libav.org
(cherry picked from commit ea1136baafb1fe271cb56c3f4d7bff0267e3c70f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit c1f479e8df24284237c80ad959619fc85e29a26d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-05-09 11:20:11 +02:00
Michael Niedermayer
b6783b8826 indeo5: update AVCodecContext width/height on size change
Fixes CVE-2012-2787

Note that in 0.7, there is only indeo 5, no indeo 4 decoder

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit b146d74730ab9ec5abede9066f770ad851e45fbc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 2bc1e4fcb96c470e2ccb2a0a78a415d5eab960c8)

Conflicts:

	libavcodec/ivi_common.c
2013-05-09 11:20:10 +02:00
Carl Eugen Hoyos
e2d529424f Fix type of shared flac table ff_flac_blocksize_table[].
Fixes ticket #2533.
(cherry picked from commit a07ac1f7888fd08e42da2bed0421e74f1cfac177)
2013-05-05 20:39:34 +02:00
Carl Eugen Hoyos
11dd8eb48b Fix type of shared flac table ff_flac_blocksize_table[].
Fixes ticket #2533.
(cherry picked from commit a07ac1f7888fd08e42da2bed0421e74f1cfac177)
2013-05-05 20:39:15 +02:00
Michael Niedermayer
a92cdd8963 Merge branch 'release/0.8' into release/0.7
* release/0.8:
  cook: check js_subband_start for validity
  avcodec_align_dimensions2: Ensure cinepak has large enough buffers.
  Update for 0.8.14
  qdm2: increase noise_table size
  wma: check byte_offset_bits
  tiff: check bppcount
  vqavideo: fix return type

Conflicts:
	Doxyfile
	RELEASE
	VERSION

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-21 03:28:00 +01:00
Michael Niedermayer
dfeef3a209 cook: check js_subband_start for validity
Fixes out of array read

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c69315a5deb0f8095e6b4746b69171d6f3059b2f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-21 02:00:40 +01:00
Michael Niedermayer
2b6f3be082 avcodec_align_dimensions2: Ensure cinepak has large enough buffers.
This is partly redundant with the following patches, but its safer

Found-by: u-bo1b@0w.se
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f5c00b347dc76285c639d9878a014c40395c5228)

Conflicts:

	libavcodec/utils.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-20 04:33:01 +01:00
Michael Niedermayer
17966ae5bb qdm2: increase noise_table size
This prevents out of array reads. An alternative solution would be
to check the index but this would require several checks in the
inner loops

Yet another alternative would be to change the index reset logic
but this likely would introduce a difference to the binary decoder

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8c4aebb58d00fd613f3f684bf0f869966149ae78)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-20 01:23:54 +01:00
Michael Niedermayer
5af2fd317d wma: check byte_offset_bits
Fixes assertion failure

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 984add64a41c3296a8a82051cc90bff2eb449609)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-20 01:23:54 +01:00
Michael Niedermayer
8aedb75156 tiff: check bppcount
Fixes division by 0

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit a34418c28e0accd1468ca15fff4d4f138a609f4e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-20 01:23:54 +01:00
Michael Niedermayer
1fd86f9a21 vqavideo: fix return type
Fixes Ticket2281

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-20 01:23:54 +01:00
Michael Niedermayer
7378101d41 Merge branch 'release/0.8' into release/0.7
* release/0.8: (92 commits)
  Update for 0.8.13
  pngdec/filter: dont access out of array elements at the end
  aacdec: check channel count
  vqavideo: check chunk sizes before reading chunks
  eamad: fix out of array accesses
  roqvideodec: check dimensions validity
  qdm2: check array index before use, fix out of array accesses
  alsdec: check block length
  huffyuvdec: Skip len==0 cases
  huffyuvdec: Check init_vlc() return codes.
  Update changelog for 0.7.7 release
  mpeg12: do not decode extradata more than once.
  indeo4/5: check empty tile size in decode_mb_info().
  dfa: improve boundary checks in decode_dds1()
  indeo5dec: Make sure we have had a valid gop header.
  rv34: error out on size changes with frame threading
  rtmp: fix buffer overflows in ff_amf_tag_contents()
  rtmp: fix multiple broken overflow checks
  Revert "h264: allow cropping to AVCodecContext.width/height"
  h264: check ref_count validity for num_ref_idx_active_override_flag
  ...

Conflicts:
	Doxyfile
	RELEASE
	VERSION
	libavcodec/rv34.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-18 00:56:01 +01:00
Michael Niedermayer
41eda87048 pngdec/filter: dont access out of array elements at the end
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1ac0fa50eff30d413206cffa5f47f7fe6d4849b1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:23:05 +01:00
Michael Niedermayer
e6ac11e417 aacdec: check channel count
Prevent out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 96f452ac647dae33c53c242ef3266b65a9beafb6)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:22:44 +01:00
Michael Niedermayer
2cac35086c vqavideo: check chunk sizes before reading chunks
Fixes out of array writes

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ab6c9332bfa1e20127a16392a0b85a4aa4840889)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:22:42 +01:00
Michael Niedermayer
af343f5cdd eamad: fix out of array accesses
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 63ac64864c6e0e84355aa3caa5b92208997a9a8d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:22:21 +01:00
Michael Niedermayer
391e0fc6c9 roqvideodec: check dimensions validity
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3ae610451170cd5a28b33950006ff0bd23036845)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:22:18 +01:00
Michael Niedermayer
caeca53a09 qdm2: check array index before use, fix out of array accesses
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit a7ee6281f7ef1c29284e3a4cadfe0f227ffde1ed)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:22:13 +01:00
Michael Niedermayer
760929117d alsdec: check block length
Fix writing over the end

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0ceca269b66ec12a23bf0907bd2c220513cdbf16)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-17 23:22:12 +01:00
Michael Niedermayer
acada70ffb Merge remote-tracking branch 'qatar/release/0.7' into release/0.8
* qatar/release/0.7:
  Update changelog for 0.7.7 release
  mpeg12: do not decode extradata more than once.
  indeo4/5: check empty tile size in decode_mb_info().
  dfa: improve boundary checks in decode_dds1()
  indeo5dec: Make sure we have had a valid gop header.
  rv34: error out on size changes with frame threading

Conflicts:
	Changelog

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-14 14:12:14 +01:00
Michael Niedermayer
4f91c45644 huffyuvdec: Skip len==0 cases
Fixes vlc decoding for hypothetical files that would contain such cases.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0dfc01c2bbf4b71bb56201bc4a393321e15d1b31)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5ff41ffeb4cb9ea6df49757dc859619dc3d3ab4f)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit 9bc70fe1ae50fd2faa0b9429d47cfbda01a92ebc)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-14 14:11:58 +01:00
Michael Niedermayer
e4831bb9a6 huffyuvdec: Check init_vlc() return codes.
Prevents out of array writes

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f67a0d115254461649470452058fa3c28c0df294)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 95ab8d33e1a680f30a5a9605175112008ab81afc)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit 277def59fce10d91e3113e5c0f63e22bc4abfa88)

Conflicts:

	libavcodec/huffyuv.c
(cherry picked from commit adf022f458d75e2c8041262e1906a249366ad518)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-14 14:11:58 +01:00
Anton Khirnov
301761792a mpeg12: do not decode extradata more than once.
Fixes CVE-2012-2803.

(cherry picked from commit 582368626188c070d4300913c6da5efa4c24cfb2)

Conflicts:

	libavcodec/mpeg12.c
	libavcodec/mpeg12.h
2013-02-02 09:54:16 +01:00
Anton Khirnov
440e98574b indeo4/5: check empty tile size in decode_mb_info().
This prevents writing into a too small array if some parameters changed
without the tile being reallocated.

Based on a patch by Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2012-2800

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit ae3da0ae5550053583a6f281ea7fd940497ea0d1)

Conflicts:

	libavcodec/ivi_common.c
2013-02-02 09:54:16 +01:00
Anton Khirnov
604d72aa0d dfa: improve boundary checks in decode_dds1()
Fixes CVE-2012-2798

CC:libav-stable@libav.org
(cherry picked from commit d05f72c75445969cd7bdb1d860635c9880c67fb6)

Conflicts:

	libavcodec/dfa.c
2013-02-02 09:54:16 +01:00
Michael Niedermayer
03ddc26066 indeo5dec: Make sure we have had a valid gop header.
This prevents decoding happening on a half initialized context.

Fixes CVE-2012-2779

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 891918431db628db17885ed947ee387b29826a64)

Conflicts:

	libavcodec/ivi_common.c
	libavcodec/ivi_common.h
2013-02-01 06:46:04 +01:00
Janne Grunau
801eff785a rv34: error out on size changes with frame threading
(cherry picked from commit cb7190cd2c691fd93e4d3664f3fce6c19ee001dd)

Fixes: CVE-2012-2772 (according to Ubuntu)
2013-02-01 06:46:04 +01:00
Michael Niedermayer
56cc629a64 Revert "h264: allow cropping to AVCodecContext.width/height"
This reverts commit a2ae183a382f063c5403922b5151d865ce7252a2.

This removes a duplicate hunk

Found-by: Joakim Plate <elupus@ecce.se>
2013-01-19 13:34:41 +01:00
Michael Niedermayer
685321e4bd Merge remote-tracking branch 'qatar/release/0.7' into release/0.8
* qatar/release/0.7:
  h264: check ref_count validity for num_ref_idx_active_override_flag
  h264: check context state before decoding slice data partitions
  oggdec: free the ogg streams on read_header failure
  oggdec: check memory allocation
  Fix uninitialized reads on malformed ogg files.
  rtsp: Recheck the reordering queue if getting a new packet
  alacdec: do not be too strict about the extradata size
  h264: fix sps parsing for SVC and CAVLC 4:4:4 Intra profiles
  h264: check sps.log2_max_frame_num for validity
  ppc: always use pic for shared libraries
  h264: enable low delay only if no delayed frames were seen
  lavf: avoid integer overflow in ff_compute_frame_duration()

Conflicts:
	libavformat/oggdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-01-17 03:16:46 +01:00
Michael Niedermayer
3f1a58db6f Merge commit 'b143844ea0f6246e0d5a938d743e2e8a98453bec' into release/0.8
* commit 'b143844ea0f6246e0d5a938d743e2e8a98453bec': (22 commits)
  aacdec: Fix an off-by-one overwrite when switching to LTP profile from MAIN.
  vp6: properly fail on unsupported feature
  h264: Fix parameters to ff_er_add_slice() call
  flacenc: ensure the order is within the min/max range in LPC order search
  yuv4mpeg: reject unsupported codecs
  vp8: reset loopfilter delta values at keyframes.
  vp56: release frames on error
  vp56: make parse_header return standard error codes
  ivi_common: check that scan pattern is set before using it.
  Update RELEASE file for 0.7.7
  tiffenc: Check av_malloc() results.
  mpegaudiodec: fix short_start calculation
  h264: avoid stuck buffer pointer in decode_nal_units
  yuv4mpeg: return proper error codes.
  smacker audio: sign-extend the initial 16-bit predicted value
  vf_pad: don't give up its own reference to the output buffer.
  avidec: return 0, not packet size from read_packet().
  wmapro: prevent division by zero when sample rate is unspecified
  alsdec: fix number of decoded samples in first sub-block in BGMC mode.
  alsdec: remove dead assignments
  ...

Conflicts:
	RELEASE
	libavformat/avidec.c
	libavformat/yuv4mpeg.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-01-17 03:03:39 +01:00
Michael Niedermayer
597d709eb4 Merge commit 'aa45b90804ab21175b8c116bd8e5eb4b4e85fbcb' into release/0.8
* commit 'aa45b90804ab21175b8c116bd8e5eb4b4e85fbcb': (22 commits)
  alsdec: Check k used for rice decoder.
  cavsdec: check for changing w/h.
  avidec: use actually read size instead of requested size
  wmaprodec: check num_vec_coeffs for validity
  lagarith: check count before writing zeros.
  indeo5: check tile size in decode_mb_info().
  indeo5: prevent null pointer dereference on broken files
  indeo: check for invalid motion vectors
  indeo: clear allocated band buffers
  indeo: check custom Huffman tables for errors
  dfa: add some checks to ensure that decoder won't write past frame end
  dfa: check that the caller set width/height properly.
  bytestream: add a new set of bytestream functions with overread checking
  avsdec: Set dimensions instead of relying on the demuxer.
  lavfi: avfilter_merge_formats: handle case where inputs are same
  rv34: use AVERROR return values in ff_rv34_decode_frame()
  h263: Add ff_ prefix to nonstatic symbols
  eval: fix swapping of lt() and lte()
  bmpdec: only initialize palette for pal8.
  vc1dec: add flush function for WMV9 and VC-1 decoders
  ...

Conflicts:
	libavcodec/avs.c
	libavcodec/mpegvideo_enc.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-01-17 02:56:12 +01:00
Janne Grunau
dd0c5e0fa9 h264: check ref_count validity for num_ref_idx_active_override_flag
Fixes segfault in the fuzzed sample bipbop234.ts_s226407.
CC: libav-stable@libav.org
(cherry-picked from commit 6e5cdf26281945ddea3aaf5eca4d127791f23ca8)
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
2013-01-12 19:36:38 +01:00
Janne Grunau
ad02537746 h264: check context state before decoding slice data partitions
Fixes mov_h264_aac__Demo_FlagOfOurFathers.mov.SIGSEGV.4e9.656.

Found-by: Mateusz "j00ru" Jurczyk
CC: libav-stable@libav.org
(cherry-picked from commit c1fcf563b13051f280db169ba41c6a1b21b25e08)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 19:36:38 +01:00
Justin Ruggles
a39c6bf1b8 alacdec: do not be too strict about the extradata size
Sometimes the extradata has duplicate atoms, but that shouldn't prevent
decoding. Just ensure that it is at least 36 bytes as a sanity check.

CC: libav-stable@libav.org
(cherry picked from commit 68a04b0ccee66f57516e129dd3ec457fd50b4bec)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 19:34:10 +01:00
Victor Lopez
884a9b0d29 h264: fix sps parsing for SVC and CAVLC 4:4:4 Intra profiles
Fixes bug 396.

CC: libav-stable@libav.org
(cherry picked from commit 1c8bf3bfed5ff5c504c8e3de96188a977f67cce0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 19:34:10 +01:00
Janne Grunau
4457e6137d h264: check sps.log2_max_frame_num for validity
Fixes infinite or long taking loop in frame num gap code in
the fuzzed sample bipbop234.ts_s223302.

CC: libav-stable@libav.org
(cherry picked from commit d7d6efe42b0d2057e67999b96b9a391f533d2333)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 19:34:10 +01:00
Janne Grunau
5fa739e685 h264: enable low delay only if no delayed frames were seen
Dropping frames is undesirable but that is the only way by which the
decoder could return to low delay mode. Instead emit a warning and
continue with delayed frames.
Fixes a crash in fuzzed sample nasa-8s2.ts_s20033 caused by a larger
than expected has_b_frames value. Low delay keeps getting re-enabled
from a presumely broken SPS.

CC: libav-stable@libav.org
(cherry picked from commit 706acb558a38eba633056773280155d66c2f4b24)

Conflicts:

	libavcodec/h264.c
2013-01-12 19:32:24 +01:00
Alex Converse
b143844ea0 aacdec: Fix an off-by-one overwrite when switching to LTP profile from MAIN.
Found-by: pawlkt
CC: libav-stable@libav.org
Fixes: CVE-2012-5144
(cherry picked from commit 6d5b0092678b2a95dfe209a207550bd2fe9ef646)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 19:27:42 +01:00