22711 Commits

Author SHA1 Message Date
Xi Wang
0b0e87bb54 atrac3: avoid oversized shifting in decode_bytes()
When `off' is 0, `0x537F6103 << 32' in the following expression invokes
undefined behavior, the result of which is not necessarily 0.

    (0x537F6103 >> (off * 8)) | (0x537F6103 << (32 - (off * 8)))

Avoid oversized shifting.

CC: libav-stable@libav.org

Signed-off-by: Xi Wang <xi.wang@gmail.com>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

(cherry picked from commit eba1ff31304e407db3cefd7532108408f364367b)
2013-03-15 13:20:55 +01:00
Michael Niedermayer
c8557235fd jpegdec: be less picky on padding
Fixes Ticket2353

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3c24fbbf651d4ec28eccfd7ef05a2595a11dd55a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 17:37:19 +01:00
Michael Niedermayer
b9a1efa6f4 msrledec: fix output_end checks
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e398990eb87785e20e065cd3f14d1dbb69df4392)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 04:58:54 +01:00
Michael Niedermayer
3ee967c1d8 msrledec: merge switches
More speedup and fixes 'may be used uninitialized in this function' warnings

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit d2e0a276d593ded94401e687f60bee266f3e725e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 04:58:01 +01:00
Michael Niedermayer
e44f89371c msrledec: move loop into switch
speeds up code and allows more simplifications

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit dbaae33c2c71862b8eaea978ed6dccc5ec03db89)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 04:57:44 +01:00
Michael Niedermayer
e586e4d93b msrledec: move output pointer test up
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c2992b705381e082e33633e62e151887da67b285)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 04:57:36 +01:00
Michael Niedermayer
f156dc54f8 mpegaudio_parser: fix off by 1 error
See:
commit 29d8cd265a536063420afe78375b2176a9e1abc5
Author: Alexander Kojevnikov <alexander@kojevnikov.com>
Date:   Tue Feb 26 21:47:11 2013 -0800

    mp3dec: Fix VBR bit rate parsing

    When parsing the Xing/Info tag, don't set the bit rate if it's an Info tag.

    When parsing the stream, don't override the bit rate if it's already set,
    otherwise calculate the mean bit rate from parsed frames. This way, the bit
    rate will be set correctly both for CBR and VBR streams.

    Signed-off-by: Alexander Kojevnikov <alexander@kojevnikov.com>
    Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 04:42:08 +01:00
Michael Niedermayer
685f50b374 Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  eamad: allocate a dummy reference frame when the real one is missing
  libmp3lame: use the correct remaining buffer size when flushing
  png: use av_mallocz_array() for the zlib zalloc function
  wmaprodec: require block_align to be set.
  ffv1: fix calculating slice dimensions for version 2
  xxan: fix invalid memory access in xan_decode_frame_type0()
  wmadec: require block_align to be set.
  ivi_common: do not call MC for intra frames when dc_transform is unset

Conflicts:
	libavcodec/ffv1dec.c
	libavcodec/ivi_common.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 04:27:35 +01:00
Michael Niedermayer
6086a4d74d Merge commit '747fbe0c212b81952bb27ec7b99fa709081e2d63' into release/1.1
* commit '747fbe0c212b81952bb27ec7b99fa709081e2d63':
  roqvideodec: fix a potential infinite loop in roqvideo_decode_frame().
  mp3dec: Fix VBR bit rate parsing
  wmaprodec: return an error, not 0, when the input is too small.
  vmdaudio: fix invalid reads when packet size is not a multiple of chunk size
  h264: check for luma and chroma bit dept being equal
  Prepare for 9.4 Release

Conflicts:
	RELEASE
	libavcodec/vmdav.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-14 02:49:31 +01:00
Luca Barbato
88089eecfd shorten: use the unsigned type where needed
get_uint returns an unsigned value, use an unsigned to store
blocksize to make sure the comparison logic is correct and report
correctly the error for the channel count not supported.

CC: libav-stable@libav.org

(cherry picked from commit 5cf7c72757779a740e897a97710aac044fe5258c)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2013-03-12 13:37:10 +01:00
Luca Barbato
0daf1428e8 shorten: report meaningful errors
(cherry picked from commit 4c364eb2b856fc33cf7b42f7c7b979e69fde5f3a)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2013-03-12 13:37:10 +01:00
Luca Barbato
97cc2f286f shorten: K&R formatting cosmetics
(cherry picked from commit a2ad554def214d2d03b7c16f68dc081a8622f9ca)

Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
2013-03-12 13:37:10 +01:00
Michael Niedermayer
21d568be17 shorten: set invalid channels count to 0
Prevent the loop shorten_decode_close from writing and freeing out of
the array boundary.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>

CC: libav-stable@libav.org

(cherry picked from commit c10da30d8426a1f681d99a780b6e311f7fb4e5c5)
2013-03-12 13:36:50 +01:00
Anton Khirnov
0cb3cab343 eamad: allocate a dummy reference frame when the real one is missing
Fixes invalid reads when the first frame is not an I-frame.

CC:libav-stable@libav.org
(cherry picked from commit 7b89cd20d844cbe763ca34e63e99d110043cf241)

Signed-off-by: Anton Khirnov <anton@khirnov.net>
2013-03-09 19:05:42 +01:00
Justin Ruggles
b77d9cbbd5 libmp3lame: use the correct remaining buffer size when flushing
CC:libav-stable@libav.org
(cherry picked from commit e984f47873258b600fd88423f40e3cdaad179190)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:10:16 +01:00
Justin Ruggles
905f5c8a1e png: use av_mallocz_array() for the zlib zalloc function
Fixes valgrind uninitialized memory errors when decoding png.

CC:libav-stable@libav.org
(cherry picked from commit 486f0b0cfc800cd38ec06635630539431d296774)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:10:16 +01:00
Anton Khirnov
20373a66ec wmaprodec: require block_align to be set.
Avoids an infinite loop in the calling programs with decoder not
consuming any input and not returning output.

CC:libav-stable@libav.org
(cherry picked from commit cacad1c058f66558ec727faac3b277d2dee264d4)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:10:16 +01:00
Anton Khirnov
d48da91373 ffv1: fix calculating slice dimensions for version 2
It got broken in 0f13cd3187192ba0cc2b043430de6e279e7b97c3.

CC:libav-stable@libav.org
(cherry picked from commit d243896987b8b2062d1faba4d8d6f0c62d2dbee9)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:10:16 +01:00
Anton Khirnov
62a657de16 xxan: fix invalid memory access in xan_decode_frame_type0()
The loop a few lines below the xan_unpack() call accesses up to
dec_size * 2 bytes into y_buffer, so dec_size must be limited to
buffer_size / 2.

CC:libav-stable@libav.org
(cherry picked from commit 8a49d2bcbe7573bb4b765728b2578fac0d19763f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:10:16 +01:00
Anton Khirnov
747fbe0c21 roqvideodec: fix a potential infinite loop in roqvideo_decode_frame().
When there is just 1 byte remanining in the buffer, nothing will be read
and the loop will continue forever. Check that there are at least 8
bytes, which are always read at the beginning.

CC:libav-stable@libav.org
(cherry picked from commit 3e2f200237af977b9253b0aff121eee27bcedb44)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Anton Khirnov
c1f479e8df wmadec: require block_align to be set.
Avoids an infinite loop in the calling programs with decoder not
consuming any input and not returning output.

CC:libav-stable@libav.org
(cherry picked from commit ea1136baafb1fe271cb56c3f4d7bff0267e3c70f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Alexander Kojevnikov
d3b40af01f mp3dec: Fix VBR bit rate parsing
When parsing the Xing/Info tag, don't set the bit rate if it's an Info tag.

When parsing the stream, don't override the bit rate if it's already set,
otherwise calculate the mean bit rate from parsed frames. This way, the bit
rate will be set correctly both for CBR and VBR streams.

CC:libav-stable@libav.org

Signed-off-by: Anton Khirnov <anton@khirnov.net>
(cherry picked from commit eae0879d961b78717dd2a0899809ad22819ae9e3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Anton Khirnov
74880e78d8 ivi_common: do not call MC for intra frames when dc_transform is unset
CC:libav-stable@libav.org
(cherry picked from commit 3ba40ebb6cc58753dc3746c718203bb31760deba)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Anton Khirnov
60dd8b5733 wmaprodec: return an error, not 0, when the input is too small.
Returning 0 may result in an infinite loop in valid calling programs. A
decoder should never return 0 without producing any output.

CC:libav-stable@libav.org
(cherry picked from commit 4c0080b7e7d501e2720d2a61f5186a18377f9d63)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Anton Khirnov
77cf052e39 vmdaudio: fix invalid reads when packet size is not a multiple of chunk size
CC:libav-stable@libav.org
(cherry picked from commit f86d66bcfa48998b0727aa0d1089a30cbeae0933)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Luca Barbato
146eac0a0c h264: check for luma and chroma bit dept being equal
The decoder assumes a single bit depth for all the planes
while the specification allows different bit depths for luma
and chroma.

Avoid the possible problems described in CVE-2013-2277

CC: libav-stable@libav.org
(cherry picked from commit 4987faee78b9869f8f4646b8dd971d459df218a5)

Conflicts:

	libavcodec/h264.c
2013-03-09 18:01:09 +01:00
Michael Niedermayer
41313bdcc5 aacsbr: Check for envelope scalefactors overflowing
This prevents various values from becoming stuck at NAN and
output to become silent
If someone knows a cleaner solution, thats welcome!

Fixes Ticket2335

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8978c743fb1d1f5a0d6dbdd83ff05817f8a41230)
2013-03-08 20:03:42 +01:00
Michael Niedermayer
088ba9bc3e psymodel: dont apply lowpass filters with a cutoff close to the nyquist
The IIR filter numerically diverges in such cases, this could easily be
fixed but would make the filter slower on some platforms

Fixes Ticket2246

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fee5da6b0a79bed9dc849f216b6da1e03132b668)
2013-03-07 19:58:05 +01:00
Michael Niedermayer
7c8beec48c buildsys: only include log2_tab per library for shared builds
Fix linking failures with -all_load due to multiple log2_tabs

Signed-off-by: Carl Eugen Hoyos <cehoyos@ag.or.at>
(cherry picked from commit 03148fd1743fca98c2f4b5920b796f381e820045)
2013-03-05 01:17:55 +01:00
Michael Niedermayer
992957ac30 Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  update Changelog
  h264: set ref_count to 0 for intra slices.
  h264: on reference overflow, reset the reference count to 0, not 1.
  flvdec: Check the return value of a malloc

Conflicts:
	Changelog
	libavcodec/h264.c
	libavformat/flvdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-03 12:15:14 +01:00
Michael Niedermayer
b3c8fd1f0e Merge commit '1b0082eabcc98e079d33c61da4d30ded89de68a9' into release/1.1
* commit '1b0082eabcc98e079d33c61da4d30ded89de68a9':
  flvdec: Don't read the VP6 header byte when setting codec type based on metadata
  vorbisdec: Accept 0 amplitude_bits
  vorbisdec: Error on bark_map_size equal to 0.
  vorbisdec: Add missing checks
  ac3dec: validate channel output mode against channel count

Conflicts:
	libavcodec/ac3dec.c
	libavformat/flvdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-03 11:56:42 +01:00
Anton Khirnov
704952fee5 h264: set ref_count to 0 for intra slices.
CC:libav-stable@libav.org
(cherry picked from commit 437211ae73ef1ed8285b4fed7620502ea4999e11)

Fixes deadlocks waiting for non-existing references with some fuzzed files.

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-02 11:20:59 +01:00
Anton Khirnov
b6f5a1ca58 h264: on reference overflow, reset the reference count to 0, not 1.
Since decode_slice_header() returns before the reference lists are
constructed, there are zero valid references.

CC:libav-stable@libav.org
(cherry picked from commit 668e16a0dd1ff56d4beeff5c658d8a2a08dbfac8)

Conflicts:

	libavcodec/h264.c
2013-03-02 11:20:59 +01:00
Luca Barbato
c6c4dc6935 vorbisdec: Accept 0 amplitude_bits
The specification does not prevent an encoder to write the amplitude 0
as 0 amplitude_bits.

Our get_bits() implementation might not support a zero sized read
properly, thus the additional branch.
(cherry picked from commit 23bd9ef4b209c789d5473d75f89a2e411d343d80)

Conflicts:

	libavcodec/vorbisdec.c
2013-02-26 20:21:01 +01:00
Michael Niedermayer
494ddd377a vorbisdec: Error on bark_map_size equal to 0.
The value is used to calculate output LSP curve and a division by zero
and out of array accesses would occur.

CVE-2013-0894

CC: libav-stable@libav.org

Reported-by: Dale Curtis <dalecurtis@chromium.org>
Found-by: inferno@chromium.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 11dcecfcca0eca1a571792c4fa3c21fb2cfddddc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-26 20:21:01 +01:00
Luca Barbato
37e99e384e vorbisdec: Add missing checks
Rate and order must not be 0 even if the specification does not say that
explicitly.
(cherry picked from commit 5b47c19bfda92273ae49e83db26a565afcaed80a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-26 20:21:01 +01:00
Justin Ruggles
73d6f4651e ac3dec: validate channel output mode against channel count
Damaged frames can lead to a mismatch, which can cause a segfault
due to using an incorrect channel mapping.

CC:libav-stable@libav.org
(cherry picked from commit d7c450436fcb9d3ecf59884a574e7684183e753d)

Conflicts:

	libavcodec/ac3dec.c
2013-02-26 20:21:01 +01:00
Michael Niedermayer
6e8ed38fab aac: reconfigure output on pop
Fixes Ticket1918

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6f77122bf5712da1d860a0ad7174181fd0bcffd9)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-26 03:39:44 +01:00
Michael Niedermayer
f64e4a8c9a Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  doc: developer: Allow tabs in the vim configuration for Automake files
  doc: filters: Correct BNF FILTER description
  Prepare for 9.3 Release
  update Changelog
  cavs: initialize various context tables to 0
  4xm: check the return value of read_huffman_tables().
  qtrle: add more checks against pixel_ptr being negative.
  mlpdec: do not try to allocate a zero-sized output buffer.
  av_memcpy_backptr: avoid an infinite loop for back = 0
  flicvideo: avoid an infinite loop in byte run compression
  lagarith: avoid infinite loop in lag_rac_refill()
  mov: use the format context for logging.
  loco: check that there is data left after decoding a plane.
  update Changelog
  x86: h264: Don't use redzone in AVX h264_deblock on Win64

Conflicts:
	Changelog
	RELEASE
	libavcodec/4xm.c
	libavcodec/loco.c
	libavcodec/qtrle.c
	libavutil/mem.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-26 03:09:41 +01:00
James Almer
d92a7870d7 lavc/bink: Chech for malloc failure
Based on commit 8ab2173ed141aa2c3336be7f9880340dfb8dcf5e
2013-02-25 05:53:20 -03:00
Anton Khirnov
77493bfd97 cavs: initialize various context tables to 0
Avoids crashes with corrupted files.

CC:libav-stable@libav.org
(cherry picked from commit 4f3b058c84f570e261d743c7c22f865617fd28ac)

Conflicts:

	libavcodec/cavs.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:46:54 +01:00
Anton Khirnov
bb3f1cad17 4xm: check the return value of read_huffman_tables().
CC:libav-stable@libav.org
(cherry picked from commit 8097fc9a2dd49d8e467b16c8bafaa96242b7fe46)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:41:24 +01:00
Anton Khirnov
a6403a3b69 qtrle: add more checks against pixel_ptr being negative.
CC:libav-stable@libav.org
(cherry picked from commit e10659244782b26061e7d52c06437de32a43a7af)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:41:10 +01:00
Anton Khirnov
e2cf32ca5f mlpdec: do not try to allocate a zero-sized output buffer.
CC:libav-stable@libav.org
(cherry picked from commit 0dff40bfb9a0b24d56ecd64cd90c8f724cc5745f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:40:48 +01:00
Anton Khirnov
612b28194b flicvideo: avoid an infinite loop in byte run compression
When byte_run is 0, pixel_countdown is not touched and the loop will run
forever.

CC:libav-stable@libav.org
(cherry picked from commit ddfe1246d98f70cdce368a2176196ba26ed7bf2d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:40:13 +01:00
Anton Khirnov
8bce2c60b8 lagarith: avoid infinite loop in lag_rac_refill()
range == 0 happens with corrupted files

CC:libav-stable@libav.org
(cherry picked from commit de6dfa2bb82df916a67e5036b0ef96a944781ed3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:40:04 +01:00
Anton Khirnov
b786ddc0f2 loco: check that there is data left after decoding a plane.
CC:libav-stable@libav.org
(cherry picked from commit 067432c1c95882c7221e694f33d9f3bdbe46de7f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:39:19 +01:00
Matt Wolenetz
5bed920971 Fix Win64 AVX h264_deblock by not using redzone on Win64
Thanks-to: "Ronald S. Bultje" <rsbultje@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 82a4a4e7caa96cea9aa2185c4c3110a5e9fde7c2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-23 01:47:05 +01:00
Michael Niedermayer
02d1efdd5b h264: check that luma and chroma depth match
Fixes out of array access

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bdeb61ccc67911cfc5e20c7cfb1312d0501ca90a)

Conflicts:

	libavcodec/h264_ps.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-22 22:53:11 +01:00
Michael Niedermayer
469cb61193 avcodec_decode_audio4: check got_frame_ptr before handling initial skip
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8a6449167a6da8cb747cfe3502ae86ffaac2ed48)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-22 22:53:11 +01:00