Commit Graph

34324 Commits

Author SHA1 Message Date
Michael Niedermayer
2140858524 avcodec/hevc: Fix integer overflow of entry_point_offset
Fixes out of array read
Fixes: d41d8cd98f00b204e9800998ecf8427e/signal_sigsegv_321165b_7641_077dfcd8cbc80b1c0b470c8554cd6ffb.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 22:34:30 +01:00
Rostislav Pehlivanov
dcbe8d8abc aacenc_ltp: use an AR filter for LTP encoding as well
Seems to work better. Information on why the decoder does this is
lacking.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2015-12-05 19:06:39 +00:00
Rostislav Pehlivanov
3112501daf aacenc: fix aac_pred option triggering an error
Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2015-12-05 18:43:17 +00:00
Michael Niedermayer
a08681f1e6 avcodec/dirac_parser: Check that there is a previous PU before accessing it
Fixes out of array read
Fixes: 99d142c47e6ba3510a74b872a1a2ae72/asan_heap-oob_11b36f4_3811_0f5c69e7609a88a580135678de1df844.dxa

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 17:42:45 +01:00
Michael Niedermayer
c7d6ec947c avcodec/dirac_parser: Add basic validity checks for next_pu_offset and prev_pu_offset
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 17:42:45 +01:00
Michael Niedermayer
79798f7c57 avcodec/dirac_parser: Fix potential overflows in pointer checks
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 17:42:45 +01:00
Rostislav Pehlivanov
d9791a8656 aacenc: remove the experimental flag
Thiss commit removes the experimental flag from the native AAC Encoder
and thus makes it the default.

After a lot of work, done by myself and Claudio Freire, the quality of
this encoder rivals and surpasses libfdk_aac in some situations. The
encoder had instability issues earlier which prevented it from having
its experimental flag removed, however the last commits done by Claudio
removed the last known source of instability and solved a lot of
problems which were previously observed. The issues were caused by the
various coding tools interfering with the scalefactor indices. Thus,
with these problems solved, it should now be possible to declare this
encoder as the default and recommend that the users should use it
instead of others provided by external libraries, as it is both faster
and has a subjectively higher quality with selected tracks.
The encoder has still yet to be fine tuned for every possible audio file
type like music or voice, so it is hoped that with the experimental flag
removed the users should be able to provide feedback and make the
encoder better than the alternatives for every type of audio and at
every bitrate.

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2015-12-05 15:41:25 +00:00
Rostislav Pehlivanov
b270ec9a10 aacenc: mark coders other than twoloop as experimental
ANMR has some interesting things coming up but is currently not in a
shape fit for non-experimental usage. Same with "FAST".

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2015-12-05 15:41:25 +00:00
Rostislav Pehlivanov
3a6e020861 aacenc: mark the "faac"-like coder for removal
This coder produces a much lower quality audio than the rest, is much
slower and is unstable. Hasn't been updated for a very long time as
well, hence it is more appropriate to remove it since it also depends on
a big burden of a code (the encode_window_bands_info function which is
just as old, just as unstable and bad and in no way modifiable or
fixable).

Signed-off-by: Rostislav Pehlivanov <atomnuker@gmail.com>
2015-12-05 15:41:25 +00:00
Michael Niedermayer
66e05f6ff5 avcodec/wmaprodec: Check bits per sample to be within the range not causing integer overflows
Fixes: 549d5aab1480d10f2a775ed90b0342f1/signal_sigabrt_7ffff6ae7cc9_5643_96bbb0cfe3e28be1dadfce1075016345.wma

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 14:35:37 +01:00
Michael Niedermayer
15d14ce47c avcodec/utils: Fix overflow in get_bit_rates computations
Fixes: 129ca3e28d73af7b1e24a9d4118e7a2d/signal_sigabrt_7ffff6ae7cc9_836_762b310fc3ef6087bd7771e5d8e90b9b.asf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 13:20:03 +01:00
Michael Niedermayer
2de8bfd2ef avcodec/pcm: Fix overflow in bitrate computation
Fixes: 129ca3e28d73af7b1e24a9d4118e7a2d/signal_sigabrt_7ffff6ae7cc9_836_762b310fc3ef6087bd7771e5d8e90b9b.asf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 13:20:03 +01:00
Michael Niedermayer
0c56f8303e avcodec/wmaprodec: Fix overflow of cutoff
Fixes: 129ca3e28d73af7b1e24a9d4118e7a2d/signal_sigabrt_7ffff6ae7cc9_836_762b310fc3ef6087bd7771e5d8e90b9b.asf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-05 13:20:03 +01:00
Claudio Freire
293c170f59 AAC encoder: ANMR, avoid empty search ranges
Avoid generating broken paths when some bands have an
empty search space.
2015-12-04 22:56:34 -03:00
Andreas Cadhalpun
7a4652dd5d aaccoder: prevent crash of anmr coder
If minq is negative, the range of sf_idx can be larger than
SCALE_MAX_DIFF allows, causing assertion failures later in
encode_scale_factors.

Reviewed-by: Claudio Freire <klaussfreire@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
2015-12-05 01:50:08 +01:00
Michael Niedermayer
4f2419888b avcodec/pgssubdec: Fix left shift of 255 by 24 places cannot be represented in type int
Fixes: b293a6479bb4b5286cff24d356bfd955/asan_generic_225c3c9_7819_cc526b657450c6cdef1371b526499626.mkv

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-04 21:39:14 +01:00
Michael Niedermayer
a85b02dcf7 avcodec/jpeg2000dec: Check bpno in decode_cblk()
Fixes: undefined shift
Fixes: c409ef86f892335a0a164b5871174d5a/asan_heap-oob_1dff564_2159_162b7234616deab02b544410455eb07b.mov

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-04 18:54:39 +01:00
Clément Bœsch
60532348d2 avcodec/mpegvideo: use constants for rc_strategy 2015-12-04 15:44:45 +01:00
Clément Bœsch
fb99ef0bd3 avcodec: use AV_OPT_TYPE_BOOL in a bunch of places 2015-12-04 15:37:05 +01:00
Hendrik Leppkes
3c4e49187f libavcodec/aacsbr_tablegen: fix tablegen on windows
Including these headers is not needed and breaks building on Windows as it
tries to activate the full compat tools, which are not needed for host
tools.
2015-12-04 14:49:40 +01:00
Hendrik Leppkes
42a9aead85 avcodec: include tablegen compat header before the tablegen functionality 2015-12-04 14:49:38 +01:00
Michael Niedermayer
2ff61c3c1a avcodec/dnxhddec: Fix runtime error: left shift of negative value
Fixes: 2abd25478c62a675f335fac00b467023/asan_static-oob_10aff98_1227_8811480c6ef1e970a7977ceb7e5a9958.mxf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-04 13:44:22 +01:00
Michael Niedermayer
18268f761b avcodec/vp3: Fix "runtime error: left shift of negative value"
Fixes: 5c6129154b356b80bcab86f9e3ee5d29/signal_sigabrt_7ffff6ae7cc9_7322_d26ac6d7cb6567db1b8be0159b387d0b.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-04 13:37:00 +01:00
Vicente Olivert Riera
a27401a05b mips: rename mipsdspr1 to mipsdsp
Signed-off-by: Vicente Olivert Riera <Vincent.Riera@imgtec.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-04 02:35:42 +01:00
Michael Niedermayer
77a644e6fa avcodec/h264_direct: Check CO-POCs before use
Fixes harmless integer overflow
Fixes mozilla bug 1230286

Found-by: Tyson Smith
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-04 01:55:11 +01:00
Rostislav Pehlivanov
ed08cbd7b1 aacenc_ltp: fix out of bounds memory access
Discovered by Coverity.
2015-12-03 23:12:37 +00:00
Michael Niedermayer
0afdfbe116 avcodec/jpeg2000: fix type of arguments of tag_tree_size
Fixes: out of array read
Fixes: 36b8096fefab16c4c9326a508053e95c/signal_sigsegv_1d9ce18_3233_1a55196b018106dfabeace071a432d9e.r3d

Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-03 02:28:00 +01:00
Michael Niedermayer
26379d4fdd avcodec/vp3: ensure header is parsed successfully before tables
Fixes assertion failure
Fixes: 266ee543812e934f7b4a72923a2701d4/signal_sigabrt_7ffff6ae7cc9_7322_85218d61759d461bdf7387180e8000c9.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-03 00:13:27 +01:00
Michael Niedermayer
cd7524fdd1 avcodec/apedec: Check length in long_filter_high_3800()
Fixes out of array read
Fixes: 0a7ff0c1d93da9cef28a315ec91b692a/asan_heap-oob_4a52e5_3604_9c56dbb20e308f4faeef7b35f688521a.ape

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-02 21:38:11 +01:00
Ronald S. Bultje
1ac89869db vp9: always keep s->bytesperpixel and ctx->pix_fmt in sync.
Fixes mozilla bug 1229128.
2015-12-02 14:53:46 -05:00
Claudio Freire
ca203e9985 AAC encoder: improve SF range utilization
This patch does 4 things, all of which interact and thus it
woudln't be possible to commit them separately without causing
either quality regressions or assertion failures.

Fate comparison targets don't all reflect improvements in
quality, yet listening tests show substantially improved quality
and stability.

1. Increase SF range utilization.

The spec requires SF delta values to be constrained within the
range -60..60. The previous code was applying that range to
the whole SF array and not only the deltas of consecutive values,
because doing so requires smarter code: zeroing or otherwise
skipping a band may invalidate lots of SF choices.

This patch implements that logic to allow the coders to utilize
the full dynamic range of scalefactors, increasing quality quite
considerably, and fixing delta-SF-related assertion failures,
since now the limitation is enforced rather than asserted.

2. PNS tweaks

The previous modification makes big improvements in twoloop's
efficiency, and every time that happens PNS logic needs to be
tweaked accordingly to avoid it from stepping all over twoloop's
decisions. This patch includes modifications of the sort.

3. Account for lowpass cutoff during PSY analysis

The closer PSY's allocation is to final allocation the better
the quality is, and given these modifications, twoloop is now
very efficient at avoiding holes. Thus, to compute accurate
thresholds, PSY needs to account for the lowpass applied
implicitly during twoloop (by zeroing high bands).

This patch makes twoloop set the cutoff in psymodel's context
the first time it runs, and makes PSY account for it during
threshold computation, making PE and threshold computations
closer to the final allocation and thus achieving better
subjective quality.

4. Tweaks to RC lambda tracking loop in relation to PNS

Without this tweak some corner cases cause quality regressions.
Basically, lambda needs to react faster to overall bitrate
efficiency changes since now PNS can be quite successful in
enforcing maximum bitrates, when PSY allocates too many bits
to the lower bands, suppressing the signals RC logic uses to
lower lambda in those cases and causing aggressive PNS.

This tweak makes PNS much less aggressive, though it can still
use some further tweaks.

Also update MIPS specializations and adjust fuzz

Also in lavc/mips/aacpsy_mips.h: remove trailing whitespace
2015-12-02 07:47:37 -03:00
Michael Niedermayer
e1057babdf avcodec/mpegaudio_tablegen: Add (), fix fate
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-12-02 03:42:44 +01:00
Ganesh Ajjanagadde
2f5075f551 avcodec/cbrt_tablegen: speed up dynamic table creation
On systems having cbrt, there is no reason to use the slow pow function.

Sample benchmark (x86-64, Haswell, GNU/Linux):
new:
5124920 decicycles in cbrt_tableinit,       1 runs,      0 skips

old:
12321680 decicycles in cbrt_tableinit,       1 runs,      0 skips

Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
2015-12-01 19:05:19 -05:00
Ganesh Ajjanagadde
1d0c94ddae avcodec/mpegaudio_tablegen: more dynamic table creation speedups
This further speeds up runtime initialization, with identical generated tables.

Sample benchmark (x86-64, Haswell, GNU/Linux):

old:
34441423 decicycles in mpegaudio_tableinit,    8192 runs,      0 skips

new:
10776291 decicycles in mpegaudio_tableinit,    8192 runs,      0 skips

Most low hanging fruit is taken care of here. For some idea, note that
83,064 array elements totalling 233,722 bytes need to be initialized.
Thus, with this patch, we average ~ 12.9 cycles per element or ~ 4.6
cycles per byte.

Reviewed-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
2015-12-01 19:05:19 -05:00
Ganesh Ajjanagadde
16d4c28cbb avcodec/mpegaudio_tablegen: speed up dynamic table creation
This does some miscellaneous stuff mainly avoiding the usage of pow to
achieve significant speedups. This is not speed critical, but is
unnecessary latency and cycles wasted for a user.

All tables tested and are identical to the old ones
(bit-exact even in floating point case).

Sample benchmark (x86-64, Haswell, GNU/Linux):
old:
102329530 decicycles in mpegaudio_tableinit,       1 runs,      0 skips

new:
34111900 decicycles in mpegaudio_tableinit,       1 runs,      0 skips

Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
2015-12-01 19:05:19 -05:00
Ganesh Ajjanagadde
789ef087c6 avcodec/ppc/fdctdsp: use more accurate constants
Whoever wrote this stuff had a pretty bad libm - digits differ pretty
quickly.

Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
2015-12-01 19:05:19 -05:00
Ganesh Ajjanagadde
7b11eead1b avcodec/ac3: always use hardcoded tables
The table in question is a 253 byte one. In fact, it turns out that
dynamic generation of the table results in an increased binary size.

Code compiled with GCC 5.2.0, x86-64 (size in bytes), before and after
patch:
old: 62321064 libavcodec/libavcodec.so.57
new: 62320536 libavcodec/libavcodec.so.57

Thus, it always make sense to statically allocate this.

Tested with FATE with/without --enable-hardcoded-tables.

Reviewed-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
2015-11-30 18:29:57 -05:00
Will Kelleher
0eac93da0f qsvenc: write a53 caption data to SEI
Signed-off-by: Will Kelleher <wkelleher@gogoair.com>
Previous version reviewed-by: Ivan Uskov <ivan.uskov@nablet.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-30 23:31:12 +01:00
Michael Niedermayer
a814f1d364 avcodec/vp3: always set pix_fmt in theora_decode_header()
Fixes assertion failure
Fixes: d0bb0662da342ec65f8f2a081222e6b9/signal_sigabrt_7ffff6ae7cc9_5471_82964f0a9ac2f4d3d59390c15473f6f7.ogg

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-30 04:25:00 +01:00
Michael Niedermayer
891dc8f875 avcodec/mpeg4videodec: Check available data before reading custom matrix
Fixes: out of array read
Fixes: 76c515fc3779d1b838667c61ea13ce92/asan_heap-oob_1fc0d07_8913_794a4629a264ebdb25b58d3a94ed1785.bit

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-29 23:45:31 +01:00
Michael Niedermayer
55711e6fd0 avcodec/dnxhddec: Disable 12bit by default
The DC VLC table used is too small, fixing this requires a sample,
thus request a sample.
Some samples are said to work even though the table has the wrong size, thus
this is left enabled if the user enables experimental features.

Fixes: 2abd25478c62a675f335fac00b467023/asan_static-oob_10aff98_1227_8811480c6ef1e970a7977ceb7e5a9958.mxf

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Approved-by: kurosu
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
2015-11-29 21:30:37 +01:00
Claudio Freire
88e498a87e AAC encoder: make pe.min a local minimum
As noted in a comment, pe.min in the reference encoder
is centered around current pe. The bit reservoir algo
needs pe.min to be a local minimum, because it can only
account for local PE variations. If it's set to a global
minimum as was being done, bit reservoir logic doesn't
work as efficiently.

This patch tries to forget old minimums and converge to
a local minimum without losing the stability of the
previous solution. Listening tests until now suggest this
solves numerous RC issues.
2015-11-29 15:03:45 -03:00
James Almer
27ce53b52d avcodec/libdcadec: require first public release
Reviewed-by: Hendrik Leppkes <h.leppkes@gmail.com>
Signed-off-by: James Almer <jamrial@gmail.com>
2015-11-29 12:53:14 -03:00
Hendrik Leppkes
503153a967 Merge commit 'fdd5c48ebdec489ec9e84eee547fefa50c3ad53c'
* commit 'fdd5c48ebdec489ec9e84eee547fefa50c3ad53c':
  texturedsp: Explicitly cast RGBA parameters to unsigned

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 16:07:59 +01:00
Hendrik Leppkes
c754591677 Merge commit 'eef38316cab68aff080cf2b83c2427a734194e8d'
* commit 'eef38316cab68aff080cf2b83c2427a734194e8d':
  texturedspenc: Avoid using separate variables

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 16:05:50 +01:00
Hendrik Leppkes
f01a276b50 Merge commit '7831fb90503142e32cc3c9be43bc3f9d342ded6b'
* commit '7831fb90503142e32cc3c9be43bc3f9d342ded6b':
  textureencdsp: cosmetics: Use normal static const for tables

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 16:05:19 +01:00
Hendrik Leppkes
2876569395 Merge commit '99cb833fc2d9874c62fffbcd3347fae660de0fe5'
* commit '99cb833fc2d9874c62fffbcd3347fae660de0fe5':
  sgi: Correctly propagate meaningful error values

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 16:04:46 +01:00
Hendrik Leppkes
7c56b8792c Merge commit '4a0918cae6394e503b17c71f8f171b4a795eb849'
* commit '4a0918cae6394e503b17c71f8f171b4a795eb849':
  sgienc: Support encoding high bit depth images with RLE

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 15:49:31 +01:00
Hendrik Leppkes
81b73f1f97 Merge commit '59e3f4e598ae381600ef54e1f6e6a8a5270ac245'
* commit '59e3f4e598ae381600ef54e1f6e6a8a5270ac245':
  sgienc: Use a local RLE encoding function

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 15:45:47 +01:00
Hendrik Leppkes
e9675ed13a Merge commit '62cc8f4d79dad119e8efeaae080a58a8dcb1e89d'
* commit '62cc8f4d79dad119e8efeaae080a58a8dcb1e89d':
  sgienc: Port to bytestream2

Merged-by: Hendrik Leppkes <h.leppkes@gmail.com>
2015-11-29 15:44:44 +01:00