Doing that doesn't make sense, because the only purpose of sbr_dequant
is to process the data from read_sbr_data.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
If videotoolbox_common_end_frame failed, then the AVFrame was returned
to the API user with the dummy buffer (in AVFrame.buf[0]) still set, and
the decode call indicating success.
These "half-set" AVFrames with dummy buffer are a videotoolbox specific
hack, because the decoder requires an allocated AVFrame for its internal
logic. Videotoolbox on the other hand allocates its frame itself
internally, and outputs it only on end_frame. At this point, the dummy
buffer is replaced with the real frame (unless decoding fails).
The replaced check should have become redundant
Found-by: Andreas Cadhalpun <andreas.cadhalpun@googlemail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
It is used as size argument of ff_canopus_parse_info_tag, which uses it
as size argument to bytestream2_init, which only supports sizes up to
INT_MAX.
Changing it's type to unsigned simplifies the check.
Reviewed-by: Vittorio Giovara <vittorio.giovara@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
This allows removing a special case for the fixed point decoder and will
make error checks simpler
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
In case of bitstream errors the deblock filter and slices can access uninitialized
top_borders from previous slices which did not fill them as they stoped halfway due
to error or where entirely missing.
This also makes code using these tables deterministic in case of missing or damaged
slices
Found-by: Tyson Smith
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes potential integer overflows
Fixes: 03e0abe721b1174856d41a1eb5d6a896/signal_sigabrt_7ffff6ae7cc9_3813_e71bf3541abed3ccba031cd5ba0269a4.avi
This fix is choosen to be simple to backport, better solution
for master is planed
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes assertion failure
Fixes: 03e0abe721b1174856d41a1eb5d6a896/signal_sigabrt_7ffff6ae7cc9_3813_e71bf3541abed3ccba031cd5ba0269a4.avi
This fix is choosen to be simple to backport, better solution
for master is planed
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This guarantees a "best effort precision".
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
Fixes out of array access
Fixes: ce19e41f0ef1e52a23edc488faecdb58/asan_heap-oob_2504e97_4202_ffa0df1baed14022b9bfd4f8ac23d0cb.smk
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Otherwise the h263p decoder can try to copy overread bytes, even though
buffer is NULL.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Fixes out of array read
Fixes: cb3f38b08b4541523974667c7d1eee9e/asan_heap-oob_2659e18_9838_021fd5cd635bf76cede6398cd9ecbcdd.tak
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes out of array read
Fixes: c50c4aa6cefda71b19a31ea12302980c/asan_heap-oob_12be5fd_7011_33ebd015a74976215934add72b9c8352.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The height of tables can be rounded up for MBAFF but this does not imply that is also true
for the previous frames
Fixes out of array reads
Fixes: c106b36fa36db8ff8f3ed0c82be7bea2/asan_heap-oob_32699f0_6321_467b9a1d7e03d7cfd310b7e65dc53bcc.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Having both is not valid and can cause a NULL pointer dereference of
frame->data[1] later.
Reviewed-by: Vittorio Giovara <vittorio.giovara@gmail.com>
Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun@googlemail.com>
Fixes out of array read
Fixes: 7fcd09eadd046e326d8ea0af66f166c8/asan_heap-oob_4a52e5_2273_fa6078a10dd575df266fb1e0b4114cd5.avi
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
M_SQRT2 is defined in math.h, or in avutil/mathematics.h for
compatibility hacks. This uses this value instead of a floating literal.
Fixed point values produced by C_FIX(), R_FIX() remain identical.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
M_SQRT2 is defined in math.h, or in avutil/mathematics.h for
compatibility hacks. This uses this value instead of an ad-hoc define.
Fixed point values produced by FIX() remain identical.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
M_PI is defined in math.h, or in avutil/mathematics.h for compatibility
hacks. This uses this value instead of an ad-hoc floating literal.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
M_E is defined in math.h, or in avutil/mathematics.h for compatibility
hacks. This uses this value instead of an ad-hoc define.
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Ganesh Ajjanagadde <gajjanagadde@gmail.com>
Fixes out of array read
Fixes: 61cf123c081ee2bb774d307c75bdb99e/asan_heap-oob_1224f76_5546_bee833ffae73f752b489b9eeaac52db7.dpx
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes out of array read
Fixes: 3c857d4d90365731524716e6d051e43a/signal_sigsegv_7f4f59bcc29e_1386_20abd2c8e655cb9c75b24368e65fe3b1.flv
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Fixes out of array read
Fixes: 20dd01398dee0f6d83d7e5410a2ae8eb/signal_sigsegv_39eeb1f_4001_62efbdf1c60748dabf1ec310b59525fd.mov
Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
