15511 Commits

Author SHA1 Message Date
Luca Barbato
f42d03746a shorten: use the unsigned type where needed
get_uint returns an unsigned value, use an unsigned to store
blocksize to make sure the comparison logic is correct and report
correctly the error for the channel count not supported.

CC: libav-stable@libav.org

(cherry picked from commit 5cf7c72757779a740e897a97710aac044fe5258c)
(cherry picked from commit 88089eecfd7e604d40d078b4f4206c647cb2e2b4)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Conflicts:
	libavcodec/shorten.c
2013-03-12 20:05:39 +01:00
Luca Barbato
5df064df62 shorten: report meaningful errors
(cherry picked from commit 4c364eb2b856fc33cf7b42f7c7b979e69fde5f3a)
(cherry picked from commit 0daf1428e82926dc5a8c72a0ff4c93aaa8a84ed9)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2013-03-12 20:05:39 +01:00
Luca Barbato
ff5a06f7f0 shorten: K&R formatting cosmetics
(cherry picked from commit a2ad554def214d2d03b7c16f68dc081a8622f9ca)
(cherry picked from commit 97cc2f286f9e3eed1a00034367ebca58cc05ee39)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

Conflicts:
	libavcodec/shorten.c
2013-03-12 20:05:39 +01:00
Michael Niedermayer
a4a97e5f3c shorten: set invalid channels count to 0
Prevent the loop shorten_decode_close from writing and freeing out of
the array boundary.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

CC: libav-stable@libav.org

(cherry picked from commit c10da30d8426a1f681d99a780b6e311f7fb4e5c5)
(cherry picked from commit 21d568be179c54a1596d1377b4da7fbe755bfe7f)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2013-03-12 20:05:39 +01:00
Luca Barbato
9e48d77158 h264: check for luma and chroma bit depth being equal
The decoder assumes a single bit depth for all the planes while
the specification allows different bit depths for luma and chroma.

Avoid the possible problems described in CVE-2013-2277
2013-03-12 20:05:39 +01:00
Diego Biurrun
b9ec4414b5 vc1: Move init code shared between decoder and parser to common code file.
This fixes standalone compilation of the VC-1 parser.
(cherry picked from commit 3c715383ea7012ac69507e6b9189c98675c77461)

Conflicts:

	libavcodec/vc1data.h

Signed-off-by: Diego Biurrun <diego@biurrun.de>
2013-03-10 13:45:43 +01:00
Justin Ruggles
5dbb3298b9 libmp3lame: use the correct remaining buffer size when flushing
CC:libav-stable@libav.org
(cherry picked from commit e984f47873258b600fd88423f40e3cdaad179190)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit b77d9cbbd5050eda75030c8926241af3dbe1a8df)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:54:28 +01:00
Anton Khirnov
e10af023b2 xxan: fix invalid memory access in xan_decode_frame_type0()
The loop a few lines below the xan_unpack() call accesses up to
dec_size * 2 bytes into y_buffer, so dec_size must be limited to
buffer_size / 2.

CC:libav-stable@libav.org
(cherry picked from commit 8a49d2bcbe7573bb4b765728b2578fac0d19763f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 62a657de168cf501acb23d48cc1aa00793dc83f3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:54:28 +01:00
Anton Khirnov
b527dd30e9 wmadec: require block_align to be set.
Avoids an infinite loop in the calling programs with decoder not
consuming any input and not returning output.

CC:libav-stable@libav.org
(cherry picked from commit ea1136baafb1fe271cb56c3f4d7bff0267e3c70f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit c1f479e8df24284237c80ad959619fc85e29a26d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:54:28 +01:00
Anton Khirnov
9b79a05289 wmaprodec: return an error, not 0, when the input is too small.
Returning 0 may result in an infinite loop in valid calling programs. A
decoder should never return 0 without producing any output.

CC:libav-stable@libav.org
(cherry picked from commit 4c0080b7e7d501e2720d2a61f5186a18377f9d63)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 60dd8b5733f9ec4919fbc732ace1be8184dde880)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:54:28 +01:00
Anton Khirnov
98406bd26e vmdaudio: fix invalid reads when packet size is not a multiple of chunk size
CC:libav-stable@libav.org
(cherry picked from commit f86d66bcfa48998b0727aa0d1089a30cbeae0933)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 77cf052e395b1fac8dd181d4f76b0101d1acd625)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:54:28 +01:00
Michael Niedermayer
e050af9a80 vorbisdec: Error on bark_map_size equal to 0.
The value is used to calculate output LSP curve and a division by zero
and out of array accesses would occur.

CVE-2013-0894

CC: libav-stable@libav.org

Reported-by: Dale Curtis <dalecurtis@chromium.org>
Found-by: inferno@chromium.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 11dcecfcca0eca1a571792c4fa3c21fb2cfddddc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 494ddd377ada76ed555f7a3f49391455daa099c9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:54:28 +01:00
Diego Biurrun
6a9f050c22 build: Fix CAF demuxer dependencies
(cherry picked from commit a519463366238a7ec05d2bb76c4a67f42cf60ece)

Conflicts:

	libavcodec/Makefile
2013-02-25 20:34:26 +01:00
Anton Khirnov
a4c9260e69 pthread: set the frame properties from the thread context, not user.
Right now, the frame properties are set from the user-facing
AVCodecContext before it is updated from the thread context, which is
wrong since they may be invalid or obsolete.
2013-01-24 11:45:27 +01:00
Luca Barbato
1076ea8115 mp3: exit on parsing error in mp_decode_frame
Properly forward mp_decode_layer3 errors, mp_decode_layer1 and
mp_decode_layer2 do not return errors.

Based on a patch by Michael Niedermayer.
(cherry picked from commit 0c03cc68386443f1e96ab6fb358220faf67cd5ff)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2013-01-20 20:36:57 +01:00
Kostya Shishkov
a94f789c33 indeo3: initialise pixel planes on allocation
This prevents decoder from reading garbage from it in case of errors later.
(cherry picked from commit 81064a8045028838fd32d18490034c207c8ecc06)

Fixes an invalid read on sample from CVE-2012-2804

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2013-01-20 20:36:57 +01:00
Janne Grunau
adef01c370 h264: check ref_count validity for num_ref_idx_active_override_flag
Fixes segfault in the fuzzed sample bipbop234.ts_s226407.
CC: libav-stable@libav.org
(cherry-picked from commit 6e5cdf26281945ddea3aaf5eca4d127791f23ca8)
Signed-off-by: Janne Grunau <janne-libav@jannau.net>
2013-01-12 17:59:41 +01:00
Janne Grunau
06312bbb10 h264: check context state before decoding slice data partitions
Fixes mov_h264_aac__Demo_FlagOfOurFathers.mov.SIGSEGV.4e9.656.

Found-by: Mateusz "j00ru" Jurczyk
CC: libav-stable@libav.org
(cherry-picked from commit c1fcf563b13051f280db169ba41c6a1b21b25e08)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:41 +01:00
Justin Ruggles
77e6676d3e alacdec: do not be too strict about the extradata size
Sometimes the extradata has duplicate atoms, but that shouldn't prevent
decoding. Just ensure that it is at least 36 bytes as a sanity check.

CC: libav-stable@libav.org
(cherry picked from commit 68a04b0ccee66f57516e129dd3ec457fd50b4bec)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Victor Lopez
a335ffd7f4 h264: fix sps parsing for SVC and CAVLC 4:4:4 Intra profiles
Fixes bug 396.

CC: libav-stable@libav.org
(cherry picked from commit 1c8bf3bfed5ff5c504c8e3de96188a977f67cce0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
f620c12067 h264: check sps.log2_max_frame_num for validity
Fixes infinite or long taking loop in frame num gap code in
the fuzzed sample bipbop234.ts_s223302.

CC: libav-stable@libav.org
(cherry picked from commit d7d6efe42b0d2057e67999b96b9a391f533d2333)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
1d98811b95 h264: slice-mt: get last_pic_dropable from master context
Fixes fate-h264-conformance-cvnlfi2_sony_h and smllwebdl.mkv from
https://github.com/OpenELEC/OpenELEC.tv/issues/1557 .

CC: libav-stable@libav.org
(cherry picked from commit a8cb1746c5b6307b2e820f965a7da8d907893b38)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
f1b3cc02ec h264: error out on unset current_picture_ptr for h->current_slice > 0
Fixes a segfault with fuzzed sample sample_varPAR_s11622_r001-02.avi.

CC: libav-stable@libav.org
(cherry picked from commit 0b300daad2f5cb59a7c06dde5ac701685e6edf16)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
b6592b402c flashsv: make sure data for zlib priming is available
Fixes a segfault in the fuzzed sample resolutionchange.flv_s314809.

CC: libav-stable@libav.org
(cherry picked from commit 3ae69b91668e3d9b65af4007eb5871397cf0b0ab)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:40 +01:00
Janne Grunau
6cd92c3880 h264: enable low delay only if no delayed frames were seen
Dropping frames is undesirable but that is the only way by which the
decoder could return to low delay mode. Instead emit a warning and
continue with delayed frames.
Fixes a crash in fuzzed sample nasa-8s2.ts_s20033 caused by a larger
than expected has_b_frames value. Low delay keeps getting re-enabled
from a presumely broken SPS.

CC: libav-stable@libav.org
(cherry picked from commit 706acb558a38eba633056773280155d66c2f4b24)

Conflicts:

	libavcodec/h264.c
2013-01-12 17:59:40 +01:00
Janne Grunau
522e97bd9e flashsv: check for keyframe before using differential coding
Fixes a segfault in te fuzzed sample resolutionchange.flv_s211713.

CC: libav-stable@libav.org
(cherry picked from commit 5ae72f54532960cb9eae82a1c9e8d505106c022b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:39 +01:00
Alex Converse
a4a63bf5b5 aacdec: Fix an off-by-one overwrite when switching to LTP profile from MAIN.
Found-by: pawlkt
CC: libav-stable@libav.org
Fixes: CVE-2012-5144
(cherry picked from commit 6d5b0092678b2a95dfe209a207550bd2fe9ef646)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-12 17:59:39 +01:00
Luca Barbato
3e700cc66b vp6: properly fail on unsupported feature
Interlacing is not supported at all and mismanaged down the normal
codepaths causing possible buffer management issues.

Fixes: CVE-2012-2783
(cherry picked from commit be75fed9755c1285ba084574aff2d7ee0f81110d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-05 12:02:22 +01:00
Luca Barbato
a5290800f5 mp3: properly forward mp_decode_frame errors
The function can return either a parsing error or a memory management
error.

Fixes: CVE-2012-2797

(cherry picked from commit 9ab0874ea8b6774c6f5470dba2b5b4615a610d0d)

Conflicts:

	libavcodec/mpegaudiodec.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-05 11:49:25 +01:00
Anton Khirnov
56c1e18a52 mpeg12: do not decode extradata more than once.
Fixes CVE-2012-2803.

CC: libav-stable@libav.org
(cherry picked from commit 582368626188c070d4300913c6da5efa4c24cfb2)

Conflicts:

	libavcodec/mpeg12.c
2013-01-05 00:35:58 +01:00
Kostya Shishkov
c55ca98769 indeo3: when freeing buffers, set pointers referencing them to NULL as well
Related to CVE-2012-2804
(cherry picked from commit bc00da27010ed9e5dbe47e5b6fae3dcddb999d78)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-05 00:27:34 +01:00
Kostya Shishkov
e5ea6539d4 indeo3: ensure that decoded cell data is in 7-bit range as presumed by decoder
Related to CVE-2012-2804
(cherry picked from commit fc417db3f162d5269c0d22f8e467da4afa67c20a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-05 00:26:25 +01:00
Janne Grunau
52adbc0e17 h264: Fix parameters to ff_er_add_slice() call
s->mb_x is reset to zero a couple of lines above. It does not make
sense to call ff_er_add_slice() with 0 as endx when the end of the
macroblock row was reached. Fixes unnecessary and counterproductive
error resilience in https://bugzilla.libav.org/show_bug.cgi?id=394.

(cherry picked from commit e6160bda98641b7d4f86de15761ad2a962f21a36)

Conflicts:

	libavcodec/h264.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-05 00:25:31 +01:00
Justin Ruggles
0ba0e31955 flacenc: ensure the order is within the min/max range in LPC order search
This fixes use of uninitialized values when the FLAC encoder uses the
2-level, 4-level, and 8-level search methods. Fixes failure of the
fate-flac-24-comp-8 test when run using valgrind.
(cherry picked from commit 3a2731cbd31d0c5681ddbc7c78edd5c53c4d0032)

Conflicts:

	libavcodec/flacenc.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:44:31 +01:00
Sami Pietila
9837f19693 vp8: reset loopfilter delta values at keyframes.
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

(cherry picked from commit 0bf511d579c7b21f1244eec688abf571ca1235bd)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:29:41 +01:00
Luca Barbato
211badf068 vp56: release frames on error
Fixes CVE-2012-2783

CC: libav-stable@libav.org

(cherry picked from commit f33b5ba63eee96c9d1c7f0e568169cb0c3694238)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:28:18 +01:00
Luca Barbato
145317d220 vp56: make parse_header return standard error codes
Returning 0 for failure is misleading.

CC: libav-stable@libav.org

(cherry picked from commit bb675d3ac6d722d5e117ae9042a996b55ca05b1d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:28:08 +01:00
Anton Khirnov
3fca5799c6 ivi_common: check that scan pattern is set before using it.
Fixes CVE-2012-2791.

CC: libav-stable@libav.org

(cherry picked from commit deabb52ab4c1fdb3dd319f3980b1489a182011f1)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:25:32 +01:00
Mans Rullgard
6365b43295 svq3: replace unsafe pointer casting with intreadwrite macros
Signed-off-by: Mans Rullgard <mans@mansr.com>
2012-10-23 16:38:52 +02:00
Anton Khirnov
a0f6c93f52 lavc: remove stats_out from the options table.
Since it is declared as a string AVOption, the generic freeing code
attempts to free it on codec close. Some codecs might have already freed
it elsewhere (or didn't even allocate it with av_malloc() in the first
place), so this might lead to an invalid free.

There is no point in having this field accessible as an AVOption, so
remove it from the options table.

Fixes Bug 380.

CC: libav-stable@libav.org
(cherry picked from commit b691135d0c6a2b1cca91adadaf457c2989c6a55d)

Conflicts:

	libavcodec/options_table.h
2012-10-22 18:51:50 +02:00
Alex Converse
8076d32f30 tiffenc: Check av_malloc() results.
(cherry picked from commit b92dfb56d4582633571db18c3d904f8602eaa2a6)

Conflicts:

	libavcodec/tiffenc.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-19 19:23:14 +02:00
Luca Barbato
0f3381ad5b mpegaudiodec: fix short_start calculation
The value should be always 3, as it follows from the specification.

Fix a stack buffer overflow in exponents_from_scale_factors as reported
by asan. Thanks to Dale Curtis for the sample vector.
(cherry picked from commit 97cfa55eea39cef30abe14682c56c1e4e7f6f10d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-19 19:23:14 +02:00
Jindřich Makovička
9822e3aa52 h264: avoid stuck buffer pointer in decode_nal_units
When decode_nal_units() previously encountered a NAL_END_SEQUENCE,
and there are some junk bytes left in the input buffer, but no start codes,
buf_index gets stuck 3 bytes before the end of the buffer.

This can trigger an infinite loop in the caller code, eg. in
try_decode_trame(), as avcodec_decode_video() then keeps returning zeroes,
with 3 bytes of the input packet still available.

With this change, the remaining bytes are skipped so the whole packet gets
consumed.

CC:libav-stable@libav.org

Signed-off-by: Jindřich Makovička <makovick@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 1a8c6917f68f7378465e18f7615762bfd22704c2)

Conflicts:

	libavcodec/h264.c
2012-10-19 19:23:14 +02:00
Franz Brauße
443f1463c0 smacker audio: sign-extend the initial 16-bit predicted value
Fixes Bug #265

Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 12cbbbb4abda2de0ea123282ccf7ebee61517f7d)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 11:13:05 +02:00
Justin Ruggles
24025cc0b9 libvorbis: use VBR by default, with default quality of 3
(cherry picked from commit 147ff24a0e8d819615a0f596df3ea47dddd79fdc)

Conflicts:
	libavcodec/libvorbis.c

Fixes a part of Bug 277

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:53:51 +02:00
Justin Ruggles
5920d00d74 libvorbis: fix use of minrate/maxrate AVOptions
- enable the options for audio encoding
- properly check for user-set maxrate
- use correct calling order in vorbis_encode_setup_managed()
(cherry picked from commit 182d4f1f3855460ee8634ea052f33332cf9d174e)

Conflicts:
	libavcodec/libvorbis.c

Fixes a part of Bug 277

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:27:50 +02:00
Ronald S. Bultje
79fb7bc667 h264: fix deadlocks on incomplete reference frame decoding.
If decoding a second complementary field, and the first was
decoded in our thread, mark decoding of that field as complete.
If decoding fails, mark the decoded field/frame as complete.
Do not allow switching between field modes or field/frame mode
between slices within the same field/frame. Ensure that two
subsequent fields cover top/bottom (rather than top/frame,
bottom/frame or such nonsense situations).

Fixes various deadlocks when decoding samples with errors in
reference frames.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1e26a48fa23ef8e1cbc424667d387184d8155f15)

Fixes Bug 118

Conflicts:
	libavcodec/h264.c

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:00:14 +02:00
Sean McGovern
a2d4d9f4fb wmapro: prevent division by zero when sample rate is unspecified
This fixes Bugzilla #327:

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit 3680b2435101a5de56821718a71c828320d535a0)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 07:41:21 +02:00
Michael Niedermayer
3c55bf1201 vc1dec: check that coded slice positions and interlacing match.
This fixes out of array writes.

Addresses: CVE-2012-2796

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit 1100acbab26883007898c53efeb289f562c6e514)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:35:25 +02:00
Thilo Borgmann
dc5283dffc alsdec: fix number of decoded samples in first sub-block in BGMC mode.
Fixes CVE-2012-2790

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 66197988b1ee914825afbc3084e6da63f862068a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:31:21 +02:00