15278 Commits

Author SHA1 Message Date
Justin Ruggles
0ba0e31955 flacenc: ensure the order is within the min/max range in LPC order search
This fixes use of uninitialized values when the FLAC encoder uses the
2-level, 4-level, and 8-level search methods. Fixes failure of the
fate-flac-24-comp-8 test when run using valgrind.
(cherry picked from commit 3a2731cbd31d0c5681ddbc7c78edd5c53c4d0032)

Conflicts:

	libavcodec/flacenc.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:44:31 +01:00
Sami Pietila
9837f19693 vp8: reset loopfilter delta values at keyframes.
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>

(cherry picked from commit 0bf511d579c7b21f1244eec688abf571ca1235bd)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:29:41 +01:00
Luca Barbato
211badf068 vp56: release frames on error
Fixes CVE-2012-2783

CC: libav-stable@libav.org

(cherry picked from commit f33b5ba63eee96c9d1c7f0e568169cb0c3694238)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:28:18 +01:00
Luca Barbato
145317d220 vp56: make parse_header return standard error codes
Returning 0 for failure is misleading.

CC: libav-stable@libav.org

(cherry picked from commit bb675d3ac6d722d5e117ae9042a996b55ca05b1d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:28:08 +01:00
Anton Khirnov
3fca5799c6 ivi_common: check that scan pattern is set before using it.
Fixes CVE-2012-2791.

CC: libav-stable@libav.org

(cherry picked from commit deabb52ab4c1fdb3dd319f3980b1489a182011f1)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-01-02 20:25:32 +01:00
Mans Rullgard
6365b43295 svq3: replace unsafe pointer casting with intreadwrite macros
Signed-off-by: Mans Rullgard <mans@mansr.com>
2012-10-23 16:38:52 +02:00
Anton Khirnov
a0f6c93f52 lavc: remove stats_out from the options table.
Since it is declared as a string AVOption, the generic freeing code
attempts to free it on codec close. Some codecs might have already freed
it elsewhere (or didn't even allocate it with av_malloc() in the first
place), so this might lead to an invalid free.

There is no point in having this field accessible as an AVOption, so
remove it from the options table.

Fixes Bug 380.

CC: libav-stable@libav.org
(cherry picked from commit b691135d0c6a2b1cca91adadaf457c2989c6a55d)

Conflicts:

	libavcodec/options_table.h
2012-10-22 18:51:50 +02:00
Alex Converse
8076d32f30 tiffenc: Check av_malloc() results.
(cherry picked from commit b92dfb56d4582633571db18c3d904f8602eaa2a6)

Conflicts:

	libavcodec/tiffenc.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-19 19:23:14 +02:00
Luca Barbato
0f3381ad5b mpegaudiodec: fix short_start calculation
The value should be always 3, as it follows from the specification.

Fix a stack buffer overflow in exponents_from_scale_factors as reported
by asan. Thanks to Dale Curtis for the sample vector.
(cherry picked from commit 97cfa55eea39cef30abe14682c56c1e4e7f6f10d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-19 19:23:14 +02:00
Jindřich Makovička
9822e3aa52 h264: avoid stuck buffer pointer in decode_nal_units
When decode_nal_units() previously encountered a NAL_END_SEQUENCE,
and there are some junk bytes left in the input buffer, but no start codes,
buf_index gets stuck 3 bytes before the end of the buffer.

This can trigger an infinite loop in the caller code, eg. in
try_decode_trame(), as avcodec_decode_video() then keeps returning zeroes,
with 3 bytes of the input packet still available.

With this change, the remaining bytes are skipped so the whole packet gets
consumed.

CC:libav-stable@libav.org

Signed-off-by: Jindřich Makovička <makovick@gmail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 1a8c6917f68f7378465e18f7615762bfd22704c2)

Conflicts:

	libavcodec/h264.c
2012-10-19 19:23:14 +02:00
Franz Brauße
443f1463c0 smacker audio: sign-extend the initial 16-bit predicted value
Fixes Bug #265

Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 12cbbbb4abda2de0ea123282ccf7ebee61517f7d)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 11:13:05 +02:00
Justin Ruggles
24025cc0b9 libvorbis: use VBR by default, with default quality of 3
(cherry picked from commit 147ff24a0e8d819615a0f596df3ea47dddd79fdc)

Conflicts:
	libavcodec/libvorbis.c

Fixes a part of Bug 277

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:53:51 +02:00
Justin Ruggles
5920d00d74 libvorbis: fix use of minrate/maxrate AVOptions
- enable the options for audio encoding
- properly check for user-set maxrate
- use correct calling order in vorbis_encode_setup_managed()
(cherry picked from commit 182d4f1f3855460ee8634ea052f33332cf9d174e)

Conflicts:
	libavcodec/libvorbis.c

Fixes a part of Bug 277

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:27:50 +02:00
Ronald S. Bultje
79fb7bc667 h264: fix deadlocks on incomplete reference frame decoding.
If decoding a second complementary field, and the first was
decoded in our thread, mark decoding of that field as complete.
If decoding fails, mark the decoded field/frame as complete.
Do not allow switching between field modes or field/frame mode
between slices within the same field/frame. Ensure that two
subsequent fields cover top/bottom (rather than top/frame,
bottom/frame or such nonsense situations).

Fixes various deadlocks when decoding samples with errors in
reference frames.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
CC: libav-stable@libav.org
(cherry picked from commit 1e26a48fa23ef8e1cbc424667d387184d8155f15)

Fixes Bug 118

Conflicts:
	libavcodec/h264.c

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 10:00:14 +02:00
Sean McGovern
a2d4d9f4fb wmapro: prevent division by zero when sample rate is unspecified
This fixes Bugzilla #327:

Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit 3680b2435101a5de56821718a71c828320d535a0)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-18 07:41:21 +02:00
Michael Niedermayer
3c55bf1201 vc1dec: check that coded slice positions and interlacing match.
This fixes out of array writes.

Addresses: CVE-2012-2796

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Kostya Shishkov <kostya.shishkov@gmail.com>
(cherry picked from commit 1100acbab26883007898c53efeb289f562c6e514)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:35:25 +02:00
Thilo Borgmann
dc5283dffc alsdec: fix number of decoded samples in first sub-block in BGMC mode.
Fixes CVE-2012-2790

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 66197988b1ee914825afbc3084e6da63f862068a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:31:21 +02:00
Mans Rullgard
c28e1c12ad alsdec: remove dead assignments
Signed-off-by: Mans Rullgard <mans@mansr.com>
(cherry picked from commit 4ca6d206d1b5beea42c4290d2ee801aaf5cd31f0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:31:01 +02:00
Thilo Borgmann
c5f9c272e9 alsdec: Fix out of ltp_gain_values read.
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 97f0efbfb86d24f081b2caa39f6249e05c95c2ef)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:27:02 +02:00
Michael Niedermayer
0f81057c12 alsdec: Check that quantized parcor coeffs are within range.
ALS spec:
	11.6.3.1.1 Quantization and encoding of parcor coefficients
	...
	In all cases the resulting quantized values ak are restricted to the range [-64,63].

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 5b051ec3bdc78f3d89e8d1425674cde8fd6c9ccc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:26:46 +02:00
Michael Niedermayer
592ba67815 alsdec: Check k used for rice decoder.
Values that fail this check will cause failure of decode_rice()

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Justin Ruggles <justin.ruggles@gmail.com>
(cherry picked from commit 23aae62c2cb4504a09ceb8cd0cabc1c8b260f521)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-17 21:26:15 +02:00
Michael Niedermayer
2051adbfa0 cavsdec: check for changing w/h.
Our decoder does not support changing w/h.

Fixes CVE-2012-2777 and CVE-2012-2784.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit c20a69630619d14ae92c5541d52c579d7c8f3e94)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Michael Niedermayer
2bc1e4fcb9 indeo4: update AVCodecContext width/height on size change
Fixes CVE-2012-2787

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit b146d74730ab9ec5abede9066f770ad851e45fbc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Michael Niedermayer
6744eee1e5 wmaprodec: check num_vec_coeffs for validity
Fixes CVE-2012-2789

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 99f392a584dd10b553facc8e819f2c7e982e176d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Michael Niedermayer
14bba214fa lagarith: check count before writing zeros.
Fixes CVE-2012-2793

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit b631e4ed64f7d1b9ca8f897fda31140e8d1fad81)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Anton Khirnov
1c8e2561b4 indeo3: fix out of cell write.
Fixes CVE-2012-2776.

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit e4d4044339b9c3b0f45f7203cd026eda3c0414c0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Michael Niedermayer
5c413648c1 indeo5: check tile size in decode_mb_info().
This prevents writing into a too small array if some parameters changed
without the tile being reallocated.

Fixes CVE-2012-2794

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 2d09cdbaf2f449ba23d54e97e94bd97ca22208c6)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Janne Grunau
3efe6becc7 indeo5: prevent null pointer dereference on broken files
Found by John Villamil <johnv@matasano.com>
(cherry picked from commit 366ac22ea5a8bab63c7f46cdad2ddb2ff22cdbed)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Michael Niedermayer
dc8371b2b1 indeo5dec: Make sure we have had a valid gop header.
This prevents decoding happening on a half initialized context.

Fixes CVE-2012-2779

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 891918431db628db17885ed947ee387b29826a64)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Anton Khirnov
0815d9174c indeo4/5: check empty tile size in decode_mb_info().
This prevents writing into a too small array if some parameters changed
without the tile being reallocated.

Based on a patch by Michael Niedermayer <michaelni@gmx.at>

Fixes CVE-2012-2800

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
(cherry picked from commit ae3da0ae5550053583a6f281ea7fd940497ea0d1)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Anton Khirnov
332555f660 ivi_common: make ff_ivi_process_empty_tile() static.
It's not used outside of ivi_common.c
(cherry picked from commit 5d2170c53bf4c2b0499f230c43764e4acf228f88)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:24 -04:00
Kostya Shishkov
c5ec190859 indeo: check for invalid motion vectors
(cherry picked from commit cf61aaaca16810b9b3a28395ed48fda8db0e87d9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:23 -04:00
Kostya Shishkov
b561618014 indeo: clear allocated band buffers
(cherry picked from commit 23ba1503f2b11057c65052b4a07961236d8d69c7)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:23 -04:00
Kostya Shishkov
e0daa15a96 indeo: track tile macroblock size
(cherry picked from commit a6e4ac40a62930d3c90f869990f96fedb9a5d654)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:23 -04:00
Kostya Shishkov
911c250aef factor out common decoding code for Indeo 4 and Indeo 5
(cherry picked from commit aa372cf4705343a9fff422ab9ead99cef7e0b415)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:23 -04:00
Kostya Shishkov
965302c9f3 indeo: check custom Huffman tables for errors
(cherry picked from commit fe7a37c36febd71576cbefc385d995a8d6e444e7)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:23 -04:00
Anton Khirnov
0c19855539 dfa: improve boundary checks in decode_dds1()
Fixes CVE-2012-2798

CC:libav-stable@libav.org
(cherry picked from commit d05f72c75445969cd7bdb1d860635c9880c67fb6)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:23 -04:00
Kostya Shishkov
d0267ecf76 dfa: use more meaningful return codes
(cherry picked from commit fb5c1aaea60a714dab3d4e6e71228855fd816222)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:20 -04:00
Kostya Shishkov
2281ac9ffd dfa: add some checks to ensure that decoder won't write past frame end
(cherry picked from commit 8099187e897ddc90cb3902332c76fb2542dac308)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 16:03:18 -04:00
Paul B Mahol
12941dbe2c dfa: convert to bytestream2 API
Protects from overreads.

Signed-off-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Ronald S. Bultje <rsbultje@gmail.com>
(cherry picked from commit 29b0d94b43ac960cb442049a5d737a3386ff0337)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 15:33:38 -04:00
Anton Khirnov
9e575e54a0 dfa: check that the caller set width/height properly.
Fixes CVE-2012-2786.
(cherry picked from commit ee715f49a06bf3898246d01b056284a9bb1bcbb9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 15:31:46 -04:00
Michael Niedermayer
9a76b7375e avsdec: Set dimensions instead of relying on the demuxer.
The decode function assumes that the video will have those dimensions.

Fixes CVE-2012-2801

CC:libav-stable@libav.org

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit 85f477935cd6b34e6ec2716b20e15ce748277a89)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 15:29:00 -04:00
Justin Ruggles
d7de11260b ac3dec: ensure get_buffer() gets a buffer for the correct number of channels
If there is an error during frame parsing, but AVCodecContext.channels was
changed and AC3DecodeContext.out_channels was set previously, the two may not
match.

Fixes CVE-2012-2802
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind

CC: libav-stable@libav.org
(cherry picked from commit 56b6a43056235fc110a018678da590595734203d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-14 15:27:39 -04:00
Martin Storsjö
31bc3fb563 snow: Check mallocs at init
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 4d8516fdb15d0177ad745228508254dee187dff9)

Conflicts:

	libavcodec/snow.c
2012-10-06 08:43:43 +02:00
Alex Converse
9aaaeba45c vorbis: Validate that the floor 1 X values contain no duplicates.
Duplicate values in this vector are explicitly banned by the Vorbis I spec
and cause divide-by-zero crashes later on.
(cherry picked from commit ecf79c4d3e8baaf2f303278ef81db6f8407656bc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-06 08:43:43 +02:00
Justin Ruggles
e46cf805b1 vorbisenc: check all allocations for failure
(cherry picked from commit be8d812c9635f31f69c30dff9ebf565a07a7dab7)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-06 08:43:43 +02:00
Kostya Shishkov
d4f3abca6a indeo3: validate new frame size before resetting decoder
(cherry picked from commit 6de226a2b8b703abc823f18c3fd7f39a0787aeb5)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2012-10-06 08:43:43 +02:00
Janne Grunau
b1ad5a21da rv34: error out on size changes with frame threading
Fixes CVE-2012-2772

(cherry picked from commit cb7190cd2c691fd93e4d3664f3fce6c19ee001dd)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-06 08:43:42 +02:00
Janne Grunau
90575bd7dd rv34: Handle only complete frames in frame-mt.
Correct handling of errors to prevent hags or crashes is very complex
otherwise.

The frame initializing is also moved from decode_slice() to
decode_frame() for clarity.
(cherry picked from commit 73ad4471a48bd02b2c2a55de116161b87e061023)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-10-06 08:43:22 +02:00
Janne Grunau
f695bd6016 rv34: use AVERROR return values in ff_rv34_decode_frame()
Also adds an error message.
(cherry picked from commit 29330721b0e8514f9f8b4d54be75a662a2b79e44)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2012-09-29 09:34:19 +02:00