All MDCT outputs must be checked in case of 128point MDCTs
Fixes: out of array read
Fixes: 04442da73d935b776d2236282588d4f9/signal_sigsegv_2625a69_351_52ca6226eb83547a2d26e322ce84ed84.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes runtime error: null pointer passed as argument 2, which is
declared to never be null
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
This is used to check if the input buffer is larger enough, so if this
overflows it can cause a false negative leading to a segmentation fault
in bytestream2_get_bufferu.
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
More don't fit into the integer output.
Also use get_bits_long, since get_bits only supports reading up to 25
bits, while get_bits_long supports the full integer range.
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Signed-off-by: Anton Khirnov <anton@khirnov.net>
Use the ability to invert phase with ms_mask instead of changing
the codebook when possible, to avoid having to switch codebooks
if some bands are INTENSTY_BT and others are INTENSITY_BT2, since
usually a set ms_mask uses less bits that a codebook change. While
it may not always be a win (ie: if it causes an ms_mask bitmap
to be sent when it wouldn't have been otherwise), it's unlikely
since the ms_mask bitmap will almost always be there already for
M/S itself.
The relative error between two encoding strategies is the simple
difference of rate-distortion values, and not the absolute
difference. An absolute measure would allow worsening of the
quantization error as well as improving.
1. Fix sf_idx and band_type addressing to address only the first
subwindow in the group (others could hold garbage values)
2. Don't step on ms_mask when is_mask is set. I/S selection
already sets the ms_mask properly and shouldn't be overridden.
3. Use mid/sid cb/sf when computing coding error, as should be
since those are the cb/sfs that will eventually be set.
4. Fix distortion computation on multi-subwindow groups (was
subtracting the bits terms multiple times)
5. Clear ms_mask when one side uses PNS and the other doesn't.
When using PNS, ms_mask signals correlated noise, which can be
detected just like regular M/S detection, so we don't skip
noise bands, but when only one side uses PNS setting the flag
can confuse some encoders, so avoid that.
Correct addressing of sf_idx and band_type arrays in I/S code.
Both arrays don't guarantee valid values for subwindows at all
times, depending on which coder is being used, which could result
in reading of garbage values. Instead, only the first subwindow
in the window group has to be used, ie: use w*16+g instead of
(w+w2)*16+g
Change the condition for application of the M/S transform to match
that of the decoder. Namely, that no special coding books must be
in use in either channel. While the condition ought to be
equivalent to the current one when the invariant of is_mask is
kept, matching the decoder's condition is safer and easier to
maintain.
Fixes an assertion error reported in #2686 that happens when
using prediction (either explicitly or implicitly by setting
the AAC main profile), since prediction code would allow
creating new zeroes or removing existing ones, without
properly checking for SF delta violations.
This patch forbids creating/removing zeroes, perhaps an
overly conservative approach, but a safe one. More permissive
and sophisticated approaches may be attempted in the future.
Fixes out of array read
Fixes: 03c406ec9530e594a074ce2979f8a1f0/asan_heap-oob_7dec26_4664_37c52495b2870a2eaac65f53958e76c1.flac
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
since:
cbcc88c libvpx: Support setting color range for vp9.
Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: James Zern <jzern@google.com>
Use av_realloc() rather than av_malloc() when normalizing noncompliant
private data in get_qt_codec().
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
When there is a non-zero decoding delay due to reordering, the first dts
should be lower than the first pts (since the first packet fed to the
decoder does not produce any output).
Use the same scheme used in mpegvideo_enc (which comes from x264
originally) -- wait for first two timestamps and extrapolate linearly to
the past to produce the first dts value.
When B-frames are enabled and the encoder returns success, all currently
pending buffers immediately become valid and can be returned to the
caller. We can only return one packet at a time, so all the other
pending buffers should be transferred to a new 'ready' fifo, from where
they can be returned in subsequent calls (in which the encoder does not
produce any new output). This bug was hidden by the incorrect testing of
the encoder return value (the return value was overwritten before it was
tested).
Fixes assertion failure
Fixes out of memory access
Fixes: test_casex.ivf
Found-by: Tyson Smith <twsmith@mozilla.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Either disabling or init'ing secure memory is required after the use
of gcry_check_version. From a look at the functions rtmpdh uses, I
noticed none require the use of secure memory, so we disable it [1][2].
This resolves some errors returned by rtmpdh code with uninitialized
gcrypt, especifically:
Fatal: failed to create the RNG lock: Invalid argument
FATAL: failed to acquire the FSM lock in libgrypt: Invalid argument
Version "1.5.4" was arbitrarily chosen. An older version probably works
as well, but I couldn't compile older versions to test on my machine.
[1]
https://gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
[2]
https://www.gnupg.org/documentation/manuals/gcrypt/Controlling-the-library.html
Signed-off-by: Ricardo Constantino <wiiaboo@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The transformation to bytes must happen after alignment to get the same
resulting pointers as before.
This fixes segmentation faults in the assembler code.
The regression was introduced in commit 9553689.
Reviewed-by: Kieran Kunhya <kierank@obe.tv>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
This exploits an approach based on the sieve of Eratosthenes, a popular
method for generating prime numbers.
Tables are identical to previous ones.
Tested with FATE with/without --enable-hardcoded-tables.
Sample benchmark (Haswell, GNU/Linux+gcc):
prev:
7860100 decicycles in cbrt_tableinit, 1 runs, 0 skips
7777490 decicycles in cbrt_tableinit, 2 runs, 0 skips
[...]
7582339 decicycles in cbrt_tableinit, 256 runs, 0 skips
7563556 decicycles in cbrt_tableinit, 512 runs, 0 skips
new:
2099480 decicycles in cbrt_tableinit, 1 runs, 0 skips
2044470 decicycles in cbrt_tableinit, 2 runs, 0 skips
[...]
1796544 decicycles in cbrt_tableinit, 256 runs, 0 skips
1791631 decicycles in cbrt_tableinit, 512 runs, 0 skips
Both small and large run count given as this is called once so small run
count may give a better picture, small numbers are fairly consistent,
and there is a consistent downward trend from small to large runs,
at which point it stabilizes to a new value.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
It is used to store the difference between pointers, so ptrdiff_t is the
correct type.
This prevents potential overflows.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
This fixes segmentation faults due to out of bounds writes, when
color_start is interpreted as negative number.
This regression was introduced in commit 57631f.
Reviewed-by: Mats Peterson <matsp888@yahoo.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
