20736 Commits

Author SHA1 Message Date
Michael Niedermayer
e59fe78958 avcodec/utvideoenc: fix slice_bits size
Fixes assertion failure

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0a8c90202bb906747168a698b6837496f82c717c)

Conflicts:

	libavcodec/utvideoenc.c
(cherry picked from commit 57522ca79cc38c279123596d3288ddbf56fa8903)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
a0d15a6271 avcodec/snow: split block clipping checks
Fixes out of array read
Fixes: d4476f68ca1c1c57afbc45806f581963-asan_heap-oob_2266b27_8607_cov_4044577381_snow_chroma_bug.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 61d59703c91869f4e5cdacd8d6be52f8b89d4ba4)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
d7071f6765 avcodec/ansi: fix integer overflow
Fixes out of array read
Fixes: 5f9698e86d92f19bb08d54ff0d57027f-signal_sigsegv_b30756_3795_cov_2693691257_ansi256.ans
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit d42ec8433c687fcbccefa51a7716d81920218e4f)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
54b61181ed avcodec/msrle: use av_image_get_linesize() to calculate the linesize
Fixes out of array access
Fixes: 14a74a0a2dc67ede543f0e35d834fbbe-asan_heap-oob_49572c_556_cov_215466444_44_001_engine_room.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c919e1ca2ecfc47d796382973ba0e48b8f6f92a2)

Conflicts:

	libavcodec/msrle.c
(cherry picked from commit bc1c8ec5e65098fd2ccd8456f667151dfc9cda42)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
b9bf17c55a avcodec/wmalosslessdec: fix mclms_coeffs* array size
Fixes corruption of context
Fixes: 8835659dde6a4f7dcdf341de6a45c6c8-signal_sigsegv_1dce67b_4564_cov_2504444599_classical_22_16_1_14000_v3c_0_extend_0_29.wma
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ec9578d54d09b64bf112c2bf7a34b1ef3b93dbd3)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
5b38967745 avcodec/vc1: reset fcm/field_mode in non advanced header parsing
Fixes NULL pointer dereference
Fixes: signal_sigsegv_1ab8bf4_2847_cov_4254117347_SA10091.vc1
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b51e9354772de446e8196dabf9aad1567b22f74d)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
87c203ba26 avcodec/vmnc: Check that rectangles are within the picture
Prevents out of array accesses with CODEC_FLAG_EMU_EDGE

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 6ba02602aa7fc7d38db582e75b8b093fb3c1608d)

Conflicts:

	libavcodec/vmnc.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 7c17207ab9acfaa934e8feb8fba90765c9d0b989)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
617fecbe0f dnxhdenc: fix mb_rc size
Fixes out of array access with RC_VARIANCE set to 0

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f1caaa1c61310beba705957e6366f0392a0b005b)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
855fe65447 avcodec/jpeglsdec: check err value for ls_get_code_runterm()
Fixes infinite loop
Fixes Ticket3086

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit cc0e47b55096361723b364afa43b79a3f5619cdc)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
ba59ab5964 avcodec/avpacket/av_packet_split_side_data: ensure that side data padding is initialized
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 240fd8c96f59ebe9dcfc4152a1086cd3f63400c0)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1e48318802b3caa493a40c0584afc30cc866d9d0)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
6f9ae27f51 avcodec/parser: reset indexes on realloc failure
Fixes Ticket2982

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit f31011e9abfb2ae75bb32bc44e2c34194c8dc40a)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Michael Niedermayer
55ad69e851 avcodec/ffv1enc: update buffer check for 16bps
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3728603f1854b5c79d1a64dd3b41b80640ef1e7f)

Conflicts:

	libavcodec/ffv1enc.c
2014-03-08 16:25:20 +01:00
Michael Niedermayer
d7ce013de5 avcodec/truemotion2: Fix av_freep arguments
Fixes null pointer dereference
Fixes Ticket2944

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c54aa2fb0f869ec025933944cbd1634fffe95d09)

Conflicts:

	libavcodec/truemotion2.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2014-03-08 16:25:20 +01:00
Martin Storsjö
a4c804d0a2 arm: Don't clobber callee saved registers in scalarproduct
q4-q7/d8-d15 are supposed to not be clobbered by the callee.

CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
(cherry picked from commit d307e408d4a9ada22df443cc38be77cc5e492694)
2013-12-21 10:00:13 +01:00
Michael Niedermayer
e28d96024b avcodec/cabac: force get_cabac to be not inlined
works around bug in gccs inline asm register assignment
Fixes Ticket3177

gcc from 4.4 to 4.6 is affected at least, no non affected gccs known
clang seems not affected

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0538b29ae8002c44f27bae8a1a6fc6e646998be5)
2013-12-09 10:39:12 +01:00
Michael Niedermayer
946815aa09 avcodec/h264: reduce noisiness of "mmco: unref short failure"
Do not consider it an error if we have no frames and should discard one.
This condition can easily happen when decoding is started from an I frame

Fixes Ticket2811

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 08a89761964bdd0a023eff6d37a1131fb7e1d7a0)

Conflicts:
	libavcodec/h264_refs.c
2013-10-26 01:22:39 +02:00
Michael Niedermayer
e3b2b884fc avcodec/h264_refs: modify key frame detection heuristic to detect more cases
Fixes Ticket2968

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 5ac6b6028f17b64723884c9fa72cfcbd369a1ba2)

Conflicts:
	libavcodec/h264_refs.c
(cherry picked from commit 6636dd551fda4fac77f2caa25d24d81abcadcd71)
2013-10-24 11:27:03 +02:00
Michael Niedermayer
49237131e9 avcodec/mjpegdec: Add some sanity checks to ljpeg_decode_rgb_scan()
These prevent the rgb ljpeg code from being run on parameters that it doesnt
support. No testcase available but it seems possible to trigger these.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 61c68000eda643dfce96dc46b488d39fd5c4e309)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-09-05 05:48:15 +02:00
Michael Niedermayer
e12ada6fd1 avcodec/dsputil: fix signedness in sizeof() comparissions
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 454a11a1c9c686c78aa97954306fb63453299760)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-09-05 03:01:10 +02:00
Carl Eugen Hoyos
7599284ad3 Avoid a deadlock when decoding wma.
Fixes ticket #2925.
(cherry picked from commit ec8a4841f7e81040f9a2757f23e70dff5e6b33a4)
2013-09-02 09:41:14 +02:00
Michael Niedermayer
5bd2b24db3 avcodec/pngdsp: fix (un)signed type in end comparission
Fixes out of array accesses
Fixes Ticket2919

Found_by: ami_stuff
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 86736f59d6a527d8bc807d09b93f971c0fe0bb07)
2013-08-30 23:40:31 +02:00
Michael Niedermayer
f229c59893 jpeg2000: check log2_cblk dimensions
Fixes out of array access
Fixes Ticket2895

Found-by: Piotr Bandurski <ami_stuff@o2.pl>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9a271a9368eaabf99e6c2046103acb33957e63b7)

Conflicts:

	libavcodec/jpeg2000dec.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-08-29 01:31:18 +02:00
Michael Niedermayer
7549bcb295 avcodec/rpza: Perform pointer advance and checks before using the pointers
Fixes out of array accesses
Fixes Ticket2850

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3819db745da2ac7fb3faacb116788c32f4753f34)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-08-29 01:29:35 +02:00
Michael Niedermayer
65753697ef avcodec/flashsv: check diff_start/height
Fixes out of array accesses
Fixes Ticket2844

Found-by: ami_stuff
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 880c73cd76109697447fbfbaa8e5ee5683309446)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-08-29 01:29:35 +02:00
Michael Niedermayer
24cff71d02 avcodec/kmvc: fix MV checks
Fixes Ticket2813
Fixes regression since 70b5583

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 3cd8aaa2b2e78faf039691e1c31ff4f8d94e3bc6)
2013-07-31 03:01:17 +02:00
Michael Niedermayer
b896c1c98e update all trac links to use the trac subdomain
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-07-21 01:13:10 +02:00
Michael Niedermayer
ec268c24a7 mpeg12dec: avoid reinitialization on PS changes when possible.
Fixes Ticket2574

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 970c8df73528659925819dec31c4c8c0887f0321)

Conflicts:
	libavcodec/mpeg12dec.c
2013-07-09 00:45:38 +02:00
Michael Niedermayer
8c5c4adebb libavcodec/x86/mpegvideo: Move mmx functions under HAVE_MMX_INLINE
should fix ticket2755

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 707b2135fda3687a6eeb91411f801e437f633f30)
2013-07-07 18:45:03 +02:00
Hendrik Leppkes
491f2e517d mathops/x86: work around inline asm miscompilation with GCC 4.8.1
The volatile is not required here, and prevents a miscompilation with GCC
4.8.1 when building on x86 with --cpu=i686

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 659df32a9d8984081ccd54adc3aee7daeb33388d)
2013-06-24 08:47:20 +02:00
Michael Niedermayer
194590dc56 alacenc: Fix missing sign_extend()
Fixes ticket #2497

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8aea2f05dc56f7e7d60767dd27ba8e846a05e8ae)
2013-06-13 00:08:02 +02:00
Claudio Freire
be66531ee2 AAC encoder: Fix rate control on twoloop.
Fixes a case where multichannel bitrate isn't accurately
targetted by psy model alone, never achieving the target bitrate.
Now fixed.

Fixes ticket #2625.

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
Signed-off-by: Carl Eugen Hoyos <cehoyos@ag.or.at>
2013-06-02 16:28:35 +02:00
Michael Niedermayer
d70043ea0f h264_cavlc: fix reading skip run
Fixes Ticket2606

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 826b3a75cd295c03720e00d3de83e1abcbedd4b9)

Conflicts:
	libavcodec/h264_cavlc.c
2013-05-30 23:18:11 +02:00
Michael Niedermayer
0428594f47 smacker: remove av_clip_int16()
Fixes Ticket2425

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 2211c76287e073a9e176fde7dbb9a63ceb2af8d1)
2013-05-21 00:01:11 +02:00
Michael Niedermayer
2c66058737 avcodec/cdgraphics: check buffer size before use
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ad002e1a13a8df934bd6cb2c84175a4780ab8942)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:30:16 +02:00
Michael Niedermayer
813b3d11e0 sanm: Check dimensions before use
Fixes integer overflow and out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 9dd04f6d8cdd1c10c28b2cb4252c1a41df581915)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:30:11 +02:00
Michael Niedermayer
d814b9b51d vmdav: Try to fix unpack_rle()
This fixes out of array accesses
The code prior to this commit could not have worked, thus obviously
was untested. I was also not able to find a valid sample that uses this
code.
This fix is thus only based on the description of the format

If someone has a sample that uses unpack_rle(), please mail me.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c1f2c4c3b49277d65b71ccdd3b6b2878f1b593eb)

Conflicts:

	libavcodec/vmdav.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 0baa0a5a02e16ef097ed9f72bc8a7d7b585c7652)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:29:16 +02:00
Michael Niedermayer
1d285db8a3 mmvideo/mm_decode_intra: check horizontal coordinate too
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit ae2132ac90f02330b0988e6e26ee0d53e41cd196)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:29:11 +02:00
Michael Niedermayer
cd24fdad47 mmvideo/mm_decode_inter: check horizontal coordinate too
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 8d3c99e825317b7efda5fd12e69896b47c700303)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:29:05 +02:00
Michael Niedermayer
ee57bfc47a mjpegdec: fix overlapping memcpy with upscale_v
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit b39fd7d63648442c20671c3e4b357268ec5c49f2)

Conflicts:

	libavcodec/mjpegdec.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:22:27 +02:00
Michael Niedermayer
4a442ac117 avcodec/mpegvideo: Fix edge emu with lowres
Fixes a few green artifacts at the top
Fixes rest of Ticket 2535

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c67bca2b5a94efb8fb3c2ba2c7d9b1b413468838)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:21:02 +02:00
Michael Niedermayer
d8c4ee6fea avcodec/mpegvideo: Fix block height for lowres 3 interlaced blocks
Fixes green trash
Fixes part of Ticket2535

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit bca50e5cd52240d885afeb3edb0fa2c74b352add)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:20:57 +02:00
Michael Niedermayer
e32acc455d avcodec/lcldec: Check that dimensions are a multiple of the subsample factors
Other dimensions would not work correctly currently,
also ask for a sample for files that fail this check.

This fixes an integer overflow leading to out of array
accesses.

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 1e00bbb10cbde3da03a1e744265ce6def9ae4c56)

Conflicts:

	libavcodec/lcldec.c

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 01:05:59 +02:00
Michael Niedermayer
f072b69f39 xbmdec: fix off by one error in scanf()
Fixes out of array access

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-05-14 00:50:54 +02:00
Carl Eugen Hoyos
6e2fb5cd73 Fix type of shared flac table ff_flac_blocksize_table[].
Fixes ticket #2533.
(cherry picked from commit a07ac1f7888fd08e42da2bed0421e74f1cfac177)
2013-05-05 20:39:50 +02:00
Hendrik Leppkes
9b481654e1 h264: fix parsing of old lossless profile (profile_idc == 144)
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 572781b25f98f99f2255404810ac33168bf28482)
2013-03-21 23:57:25 +01:00
ArnoB
becde6ab1c dpxenc: fix data offset
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit 361319d0f49475bc14c744194870f9bab78a8a83)

Conflicts:

	tests/ref/lavf/dpx
2013-03-20 23:11:26 +01:00
Michael Niedermayer
f84ddb0c0f msrledec: fix output_end checks
Fixes out of array accesses

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit e398990eb87785e20e065cd3f14d1dbb69df4392)

Conflicts:

	libavcodec/msrledec.c
2013-03-20 19:54:53 +01:00
Michael Niedermayer
bbefdb3938 msrledec: merge switches
More speedup and fixes 'may be used uninitialized in this function' warnings

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit d2e0a276d593ded94401e687f60bee266f3e725e)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-20 19:47:03 +01:00
Michael Niedermayer
cc0662daef msrledec: move loop into switch
speeds up code and allows more simplifications

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit dbaae33c2c71862b8eaea978ed6dccc5ec03db89)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-20 19:46:58 +01:00
Michael Niedermayer
10d59d29bc msrledec: move output pointer test up
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
(cherry picked from commit c2992b705381e082e33633e62e151887da67b285)

Signed-off-by: Michael Niedermayer <michaelni@gmx.at>
2013-03-20 19:46:52 +01:00