22388 Commits

Author SHA1 Message Date
Anton Khirnov
60dd8b5733 wmaprodec: return an error, not 0, when the input is too small.
Returning 0 may result in an infinite loop in valid calling programs. A
decoder should never return 0 without producing any output.

CC:libav-stable@libav.org
(cherry picked from commit 4c0080b7e7d501e2720d2a61f5186a18377f9d63)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Anton Khirnov
77cf052e39 vmdaudio: fix invalid reads when packet size is not a multiple of chunk size
CC:libav-stable@libav.org
(cherry picked from commit f86d66bcfa48998b0727aa0d1089a30cbeae0933)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-09 18:01:09 +01:00
Luca Barbato
146eac0a0c h264: check for luma and chroma bit dept being equal
The decoder assumes a single bit depth for all the planes
while the specification allows different bit depths for luma
and chroma.

Avoid the possible problems described in CVE-2013-2277

CC: libav-stable@libav.org
(cherry picked from commit 4987faee78b9869f8f4646b8dd971d459df218a5)

Conflicts:

	libavcodec/h264.c
2013-03-09 18:01:09 +01:00
Michael Niedermayer
41313bdcc5 aacsbr: Check for envelope scalefactors overflowing
This prevents various values from becoming stuck at NAN and
output to become silent
If someone knows a cleaner solution, thats welcome!

Fixes Ticket2335

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8978c743fb1d1f5a0d6dbdd83ff05817f8a41230)
2013-03-08 20:03:42 +01:00
Michael Niedermayer
088ba9bc3e psymodel: dont apply lowpass filters with a cutoff close to the nyquist
The IIR filter numerically diverges in such cases, this could easily be
fixed but would make the filter slower on some platforms

Fixes Ticket2246

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fee5da6b0a79bed9dc849f216b6da1e03132b668)
2013-03-07 19:58:05 +01:00
Michael Niedermayer
7c8beec48c buildsys: only include log2_tab per library for shared builds
Fix linking failures with -all_load due to multiple log2_tabs

Signed-off-by: Carl Eugen Hoyos <cehoyos@ag.or.at>
(cherry picked from commit 03148fd1743fca98c2f4b5920b796f381e820045)
2013-03-05 01:17:55 +01:00
Michael Niedermayer
992957ac30 Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  update Changelog
  h264: set ref_count to 0 for intra slices.
  h264: on reference overflow, reset the reference count to 0, not 1.
  flvdec: Check the return value of a malloc

Conflicts:
	Changelog
	libavcodec/h264.c
	libavformat/flvdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-03 12:15:14 +01:00
Michael Niedermayer
b3c8fd1f0e Merge commit '1b0082eabcc98e079d33c61da4d30ded89de68a9' into release/1.1
* commit '1b0082eabcc98e079d33c61da4d30ded89de68a9':
  flvdec: Don't read the VP6 header byte when setting codec type based on metadata
  vorbisdec: Accept 0 amplitude_bits
  vorbisdec: Error on bark_map_size equal to 0.
  vorbisdec: Add missing checks
  ac3dec: validate channel output mode against channel count

Conflicts:
	libavcodec/ac3dec.c
	libavformat/flvdec.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-03 11:56:42 +01:00
Anton Khirnov
704952fee5 h264: set ref_count to 0 for intra slices.
CC:libav-stable@libav.org
(cherry picked from commit 437211ae73ef1ed8285b4fed7620502ea4999e11)

Fixes deadlocks waiting for non-existing references with some fuzzed files.

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-03-02 11:20:59 +01:00
Anton Khirnov
b6f5a1ca58 h264: on reference overflow, reset the reference count to 0, not 1.
Since decode_slice_header() returns before the reference lists are
constructed, there are zero valid references.

CC:libav-stable@libav.org
(cherry picked from commit 668e16a0dd1ff56d4beeff5c658d8a2a08dbfac8)

Conflicts:

	libavcodec/h264.c
2013-03-02 11:20:59 +01:00
Luca Barbato
c6c4dc6935 vorbisdec: Accept 0 amplitude_bits
The specification does not prevent an encoder to write the amplitude 0
as 0 amplitude_bits.

Our get_bits() implementation might not support a zero sized read
properly, thus the additional branch.
(cherry picked from commit 23bd9ef4b209c789d5473d75f89a2e411d343d80)

Conflicts:

	libavcodec/vorbisdec.c
2013-02-26 20:21:01 +01:00
Michael Niedermayer
494ddd377a vorbisdec: Error on bark_map_size equal to 0.
The value is used to calculate output LSP curve and a division by zero
and out of array accesses would occur.

CVE-2013-0894

CC: libav-stable@libav.org

Reported-by: Dale Curtis <dalecurtis@chromium.org>
Found-by: inferno@chromium.org
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 11dcecfcca0eca1a571792c4fa3c21fb2cfddddc)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-26 20:21:01 +01:00
Luca Barbato
37e99e384e vorbisdec: Add missing checks
Rate and order must not be 0 even if the specification does not say that
explicitly.
(cherry picked from commit 5b47c19bfda92273ae49e83db26a565afcaed80a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-26 20:21:01 +01:00
Justin Ruggles
73d6f4651e ac3dec: validate channel output mode against channel count
Damaged frames can lead to a mismatch, which can cause a segfault
due to using an incorrect channel mapping.

CC:libav-stable@libav.org
(cherry picked from commit d7c450436fcb9d3ecf59884a574e7684183e753d)

Conflicts:

	libavcodec/ac3dec.c
2013-02-26 20:21:01 +01:00
Michael Niedermayer
6e8ed38fab aac: reconfigure output on pop
Fixes Ticket1918

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6f77122bf5712da1d860a0ad7174181fd0bcffd9)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-26 03:39:44 +01:00
Michael Niedermayer
f64e4a8c9a Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  doc: developer: Allow tabs in the vim configuration for Automake files
  doc: filters: Correct BNF FILTER description
  Prepare for 9.3 Release
  update Changelog
  cavs: initialize various context tables to 0
  4xm: check the return value of read_huffman_tables().
  qtrle: add more checks against pixel_ptr being negative.
  mlpdec: do not try to allocate a zero-sized output buffer.
  av_memcpy_backptr: avoid an infinite loop for back = 0
  flicvideo: avoid an infinite loop in byte run compression
  lagarith: avoid infinite loop in lag_rac_refill()
  mov: use the format context for logging.
  loco: check that there is data left after decoding a plane.
  update Changelog
  x86: h264: Don't use redzone in AVX h264_deblock on Win64

Conflicts:
	Changelog
	RELEASE
	libavcodec/4xm.c
	libavcodec/loco.c
	libavcodec/qtrle.c
	libavutil/mem.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-26 03:09:41 +01:00
James Almer
d92a7870d7 lavc/bink: Chech for malloc failure
Based on commit 8ab2173ed141aa2c3336be7f9880340dfb8dcf5e
2013-02-25 05:53:20 -03:00
Anton Khirnov
77493bfd97 cavs: initialize various context tables to 0
Avoids crashes with corrupted files.

CC:libav-stable@libav.org
(cherry picked from commit 4f3b058c84f570e261d743c7c22f865617fd28ac)

Conflicts:

	libavcodec/cavs.c

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:46:54 +01:00
Anton Khirnov
bb3f1cad17 4xm: check the return value of read_huffman_tables().
CC:libav-stable@libav.org
(cherry picked from commit 8097fc9a2dd49d8e467b16c8bafaa96242b7fe46)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:41:24 +01:00
Anton Khirnov
a6403a3b69 qtrle: add more checks against pixel_ptr being negative.
CC:libav-stable@libav.org
(cherry picked from commit e10659244782b26061e7d52c06437de32a43a7af)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:41:10 +01:00
Anton Khirnov
e2cf32ca5f mlpdec: do not try to allocate a zero-sized output buffer.
CC:libav-stable@libav.org
(cherry picked from commit 0dff40bfb9a0b24d56ecd64cd90c8f724cc5745f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:40:48 +01:00
Anton Khirnov
612b28194b flicvideo: avoid an infinite loop in byte run compression
When byte_run is 0, pixel_countdown is not touched and the loop will run
forever.

CC:libav-stable@libav.org
(cherry picked from commit ddfe1246d98f70cdce368a2176196ba26ed7bf2d)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:40:13 +01:00
Anton Khirnov
8bce2c60b8 lagarith: avoid infinite loop in lag_rac_refill()
range == 0 happens with corrupted files

CC:libav-stable@libav.org
(cherry picked from commit de6dfa2bb82df916a67e5036b0ef96a944781ed3)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:40:04 +01:00
Anton Khirnov
b786ddc0f2 loco: check that there is data left after decoding a plane.
CC:libav-stable@libav.org
(cherry picked from commit 067432c1c95882c7221e694f33d9f3bdbe46de7f)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-23 14:39:19 +01:00
Matt Wolenetz
5bed920971 Fix Win64 AVX h264_deblock by not using redzone on Win64
Thanks-to: "Ronald S. Bultje" <rsbultje@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 82a4a4e7caa96cea9aa2185c4c3110a5e9fde7c2)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-23 01:47:05 +01:00
Michael Niedermayer
02d1efdd5b h264: check that luma and chroma depth match
Fixes out of array access

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bdeb61ccc67911cfc5e20c7cfb1312d0501ca90a)

Conflicts:

	libavcodec/h264_ps.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-22 22:53:11 +01:00
Michael Niedermayer
469cb61193 avcodec_decode_audio4: check got_frame_ptr before handling initial skip
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8a6449167a6da8cb747cfe3502ae86ffaac2ed48)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-22 22:53:11 +01:00
Michael Niedermayer
a642be972d h264: ensure that get_format() is called when changing format but not otherwise.
Fixes Ticket2288

Tested-by: Stefano Pigozzi <stefano.pigozzi@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 04220b473e9d7b22d737405348f9904f22bcfb96)

Conflicts:

	libavcodec/h264.c
2013-02-22 22:53:11 +01:00
Matt Wolenetz
bc9d341be8 x86: h264: Don't use redzone in AVX h264_deblock on Win64
This fixes crashes in chromium on win64 on machines with AVX
(crashes that apparently aren't triggered by fate).

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit 311443f6c7eb230276e320f2d30a5d729cf32b76)

Signed-off-by: Martin Storsjö <martin@martin.st>
2013-02-22 23:48:35 +02:00
Michael Niedermayer
80ddf7889e Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  doc: Fix some obsolete references to av* tools as ff* tools
  vqavideo: check chunk sizes before reading chunks
  roqvideodec: check dimensions validity
  qdm2: check array index before use, fix out of array accesses
  mpegvideo: Do REBASE_PICTURE with byte pointers

Conflicts:
	libavcodec/qdm2.c
	libavcodec/roqvideodec.c
	libavcodec/vqavideo.c

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-22 22:44:15 +01:00
Carl Eugen Hoyos
4be63111d1 Fix bits_per_coded_sample when encoding png with frame-level multithreading.
Fixes ticket #2290.
(cherry picked from commit c4dc6c4c86a052b7ba53fa7ae0c1b0643ad70d0b)
2013-02-21 09:04:05 +01:00
Michael Niedermayer
ab434bf0d0 vqavideo: check chunk sizes before reading chunks
Fixes out of array writes

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ab6c9332bfa1e20127a16392a0b85a4aa4840889)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 13093f9767b922661132a3c1f4b5ba2c7338b660)

CC: libav-stable@libav.org

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit f7d18deb73d1dd1b27b2c7062c9a10d168a6c62a)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-19 08:10:24 +01:00
Michael Niedermayer
52b18c1fde roqvideodec: check dimensions validity
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3ae610451170cd5a28b33950006ff0bd23036845)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit fee26d352a52eb9f7fcd8d9167fb4a5ba015b612)

CC: libav-stable@libav.org

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 488f87be873506abb01d67708a67c10a4dd29283)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-19 08:10:03 +01:00
Michael Niedermayer
0b2b8ab979 qdm2: check array index before use, fix out of array accesses
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>

(cherry picked from commit a7ee6281f7ef1c29284e3a4cadfe0f227ffde1ed)

CC: libav-stable@libav.org

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
(cherry picked from commit 39bec05ed42e505d17877b0c23f16322f9b5883b)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-19 08:09:48 +01:00
Martin Storsjö
65bf4c9c45 mpegvideo: Do REBASE_PICTURE with byte pointers
REBASE_PICTURE (more specifically, this half of it) takes a Picture
pointer that points into one larger struct, finds the offset of
that Picture within the struct and finds the corresponding field
within another instance of a similar struct.

The pointer difference "pic - (Picture*)old_ctx" is a value given
in sizeof(Picture) units, and when applied back on
(Picture*)new_ctx gets multiplied back with sizeof(Picture). Many
compilers seem to optimize out this division/multiplication, but
not all do.

GCC 4.2 on OS X doesn't seem to remove the division/multiplication,
therefore the new pointer didn't turn out to point to exactly
the right place in the new struct since it only had sizeof(Picture)
granularity (and the Picture is not aligned on a sizeof(Picture)
boundary within the encompassing struct). This bug has been present
before 47318953d as well - with H264, pointers to h->ref_list[0][0]
pointed to 88 bytes before h->ref_list[0][0] after the rebase. After
shrinking Picture, the difference ended up even larger, making
writes via such a Picture pointer overwrite other fields at random
in H264Context, ending up in crashes later.

This fixes H264 multithreaded decoding on OS X with GCC 4.2.

Fixes Bug: #439

Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit a65f965c04bfa27adedc0409c14cc05903f483d0)

Signed-off-by: Reinhard Tartler <siretart@tauware.de>
2013-02-19 08:09:08 +01:00
Michael Niedermayer
811a504c6b shorten: dont leave invalid channel counts in the context.
Fixes freeing invalid addresses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 4f1279154ee9baf2078241bf5619774970d18b25)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:28:52 +01:00
Michael Niedermayer
75211f2b8c tiff: Check buffer allocation and pointer increment more carefully in shorts2str() and double2str()
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e1219cdaf9fb4bc8cea410e1caf802373c1bfe51)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:28:46 +01:00
Michael Niedermayer
f6687bbb64 pngdec/filter: dont access out of array elements at the end
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1ac0fa50eff30d413206cffa5f47f7fe6d4849b1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:28:41 +01:00
Michael Niedermayer
1400f1a1e4 sanm: Use the correct height variable in the decoded_size checks
Fixes integer overflow and out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5260edee7e5bd975837696c8c8c1a80eb2fbd7c1)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:28:35 +01:00
Michael Niedermayer
1ea5bbc594 sanm: add forgotten check for decoded_size in old_codec37()
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 365270aec5c2b9284230abc702b11168818f14cf)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:28:30 +01:00
Michael Niedermayer
f5955d9f6f targa: Fix y check in advance_line
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 796012af6c780b5b13ebca39a491f215515a18fe)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:28:24 +01:00
Michael Niedermayer
0f5a0a4155 Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  svq3: unbreak decoding
  build: make audio_frame_queue a stand-alone component
  build: The libopencore-amrnb encoder depends on audio_frame_queue
  libopencore-amrwb: Make AMR-WB ifdeffery more precise
  libopencore-amr: Conditionally compile decoder and encoder bits
  libopencore-amrnb: cosmetics: Group all encoder-related code together

Conflicts:
	configure
	libavcodec/Makefile

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-19 01:19:31 +01:00
Michael Niedermayer
c6f59b95c5 h264: avoid calling get_format() multiple times
Some applications do not like that.
Fixes VDA
Reduces noise for VDPAU

Tested-by: Guillaume POIRIER <poirierg@gmail.com>
Tested-by: Carl Eugen Hoyos <cehoyos@ag.or.at>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit dece584a639c9fd61a72e21800815e8397b3b617)

Conflicts:

	libavcodec/h264.c
2013-02-18 18:14:11 +01:00
Matti Hamalainen
d61c6ebccf svq3: unbreak decoding
a7d2861d36756b913e85681b86ed3385274e8ced removed necessary braces.
2013-02-18 02:49:45 +01:00
Luca Barbato
b9a287f237 build: make audio_frame_queue a stand-alone component
Encoders requiring it have the dependency expressed in the configure.
2013-02-17 22:38:37 +01:00
Diego Biurrun
6c62098827 build: The libopencore-amrnb encoder depends on audio_frame_queue
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit d0fd1dd559b8362bdbca3405f739e0cc202d62e7)
2013-02-16 23:41:31 +01:00
Diego Biurrun
a23d6ea1e4 libopencore-amrwb: Make AMR-WB ifdeffery more precise
The library might provide an encoder in the future, so it's better to
check for the presence of the decoder rather than just the library.

CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit ed89cad6aa04bbd692b3eb21c0e0bb56aca77130)
2013-02-16 23:41:31 +01:00
Diego Biurrun
e492818d89 libopencore-amr: Conditionally compile decoder and encoder bits
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit f6ad3ca159edcd2e48634bf39b9cd4a85af29cb1)
2013-02-16 23:41:31 +01:00
Diego Biurrun
1ca25bc387 libopencore-amrnb: cosmetics: Group all encoder-related code together
CC: libav-stable@libav.org
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
(cherry picked from commit 81ae57a269782fbfc9e11548d1e6605f13d65c9b)
2013-02-16 23:41:31 +01:00
Michael Niedermayer
7d3e217623 Merge remote-tracking branch 'qatar/release/9' into release/1.1
* qatar/release/9:
  arm: Fall back to runtime cpu feature detection via /proc/cpuinfo
  doc/platform: Fix 10l typo
  xxan: properly handle odd heights.

Merged-by: Michael Niedermayer <michaelni@gmx.at>
2013-02-14 13:50:08 +01:00