Add option, for newmv-last, to limit the rd-threshold update for early exit,
under a source varianace condition.
This can improve visual quality in low texture moving areas,
like forehead/faces.
Also add bias against golden to improve the speed/fps,
will little/negligible loss in quality.
Only affects CBR mode, non-svc, non-screen-content.
Change-Id: I3a5229eee860c71499a6fd464c450b167b07534d
Changes the default recode rule for Speed 0 and best quality
from ALLOW_RECODE to ALLOW_RECODE_KFARFGF.
Tested on the NF, hdres, midres and lowres test sets, this setting
when combined with patch I40cb559... now performs "as well" in
metrics terms (in fact it came out a tiny amount better overall)
but encode time is 9.6% faster (measured as the average
from 27 mid rate local encodes on clips in the derf/lowres set.
Change-Id: I8c781c0cdfa3a9929cd9406d15582fce47d6ae3b
Allow recodes for the first inter frame in each arf group
even when the recode rule is set to ALLOW_RECODE_KFARFGF.
Small gains of 0.05%.
Change-Id: I40cb559d36a2bf0ebf5cf758c3f92e452b480577
This patch fixed a motion vector out of range bug:
vpxenc: ../libvpx/vp9/encoder/vp9_mcomp.c:69:
mv_cost: Assertion `mv->col >= -((1 << (11 + 1 + 2)) - 1) &&
mv->col < ((1 << (11 + 1 + 2)) - 1)' failed.
For blocks that returned without having full-pixel search, the original
MV limits were not restored, which caused the failure. Moved the set
MV limit function down to fix the bug.
Change-Id: Id7d798fc7214e95c6e4846c588f0233fcf1a4223
This patch fixed a motion vector(MV) out of range bug, which was caused
by not restoring the original values of the MV min/max thresholds after
the sub8x8 full pixel motion search. It occurred rarely and only was seen
while encoding a 4k clip for 200 frames.
BUG=webm:1271
Change-Id: Ibc4e0de80846f297431923cef8a0c80fe8dcc6a5
* changes:
Use common transpose for vpx_idct32x32_1024_add_neon
Use common transpose for vpx_idct8x8_[12|64]_add_neon
Use common transpose for vp9_iht8x8_add_neon
Use common transpose for vpx_idct16x16_[10|256]_add_neon
Increase the minimum distance.
Reduces the overshoot somewhat on some clips,
small gain in avgPSNR (~0.1%) on ytlive set.
Change-Id: Id5ddde20c2907dbdb536e79542eff775019c142b
Move best index into the token state. Shrink it down to one byte. This
is more cache friendly (access are group together) and uses less total
memory.
Results in 4% fewer cycles in optimize_b().
Change-Id: I75db484fb3dc82f59928d54b659d79c80ee40452
Shutdown all threads before reclaiming any memory. The frame-level
parallel decoder may access data from another worker.
BUG=webm:1259
Change-Id: I26856ebd1f77cc4a4545331baa19bbf3e01c4ea4
This commit changes the call in vp9 encoder from vp9_deblock() to
vp9_post_proc_frame() to ensure the data structures used in the call
are properly allocated. This fixes an encoder crash when configured
with --enable-internal-stats.
Change-Id: I2393b336c0f566665336df4f1ba91c405eb56764
Allow usage of lookahead for VBR in real-time mode, for 1 pass vbr.
Current usage is for fast checking of future scene cuts/changes,
and adjusting rate control (gf interval and active_worst/target size).
Added unittests (datarate) for 1 pass vbr mode, with non-zero lag.
Added an experimental option to limit QP based on lookahead.
Overall positive gain in metrics on ytlive set:
avgPNSR/SSIM up on average ~1-3%; several clips up by 5, 7%.
Change-Id: I960d57dfc89de121c4824b9a9bf88d2814e74b56
This change eliminates redundant computation in the two stage
downscaling, which saves ~1% encoding time in 3-layer svc encoding.
Change-Id: Ib4b218811b68499a740af1f9b7b5a5445e28d671
Modify the gfu_boost and af_ratio setting based on the
average frame motion level.
Change only affects 1 pass vbr.
Metrics overall positive on ytlive set.
On average up by ~1%, several clips up by 2-4%.
Change-Id: Ic18c49eb2df74cb4986b63cdb11be36d86ab5e8d
Use the precise context to estimate the zero token cost in trellis
optimization process. This improves the speed 0 coding performance
by 0.15% for lowres and 0.1% for midres. It improves the speed 1
coding performance by 0.2% for midres and hdres.
Change-Id: I59c7c08702fc79dc4f8534b64ca594da909e2c91
This commit allows the inter prediction residual to use uniform
quantization followed by trellis coefficient optimization in
speed 0. It improves the coding performance by
lowres 0.79%
midres 1.07%
hdres 1.44%
Change-Id: I46ef8cfe042a4ccc7a0055515012cd6cbf5c9619
Move the operations that update the context buffers outside this
function. The coeff_cost() takes all input as const value and returns
the coefficient cost.
This makes preparation for the next coefficient optimization CLs.
Change-Id: I850eec6e5470b91ea84646ff26b9231b09f70a0c
Replace the existing mv bias with a bias only for
NEWMV, and based on the motion vector difference of
its top/left neighbors.
For cbr non-screen-content mode.
Change-Id: I8a8cf56347cfa23e9ffd8ead69eec8746c8f9e09
Use a measure of noise energy to adjust Q estimate and
arf filter strength.
Gains 0.3-0.5% on Lowres and |Netflix sets.
Hdres and Midres neutral.
Change-Id: Ic0de552e7b6763e70eeeaa3651619831b423e151
Use pixel domain distortion metric in speed 0. This improves the
compression performance by 0.3% for both low and high resolution
test sets.
Change-Id: I5b5b7115960de73f0b5e5d0c69db305e490e6f1d
Safer to have the decoder operate normally and have
better-hw-compatibility only implement encoding changes.
Fixes some test failures.
Change-Id: I0dd70d002e4e893992f0cd59774b9363e6f7fe76
For real time CBR mode, use model_rd_for_sb_y for 32x32 if the sb is
a skin sb to avoid visual regression on the slowly moving face.
Refer to the cl: https://chromium-review.googlesource.com/#/c/356020/
Change-Id: I42c36666b2b474ce5ee274239d52ae8ab400fd46
The transform block row and column positions are always available
outside the callees. There is no need to re-compute these values
again. This approach has been used by the decoder. This commit
removes txfrm_block_to_raster_xy() function.
Change-Id: I5b90f91a0d8b7c35cfa7d171da9edf8202630108
these are debug-only modules that can be added in manually when needed.
leave a reference in vp8_common.mk / vp9_common.mk for easy addition.
quiets -Wmissing-prototypes warning
BUG=b/29584271
Change-Id: Ifc8637d877edfbd562b34dc5c540428bba7951fc
Moved the API patch from NextGenv2. An example was included.
To try it, for example, run the following command:
$ examples/vpx_cx_set_ref vp9 352 288 in.yuv out.ivf 4 30
Change-Id: I4cf8f23b86d7ebd85ffd2630dcfbd799c0b88101
The scaling of the threshold for 10 and 12 bit here appears
to be in the wrong direction. For 10 and 12 bit we expect sse
values to be higher and hence the threshold used should be
scaled up not down.
Change-Id: I2678116652b539aef48100e0f22873edd4f5a786
This function seems to scale the threshold for testing an
SSE value in the wrong direction for 10 and 12 bit inputs.
Also for a true SSE the scalings should probably be << 4 and 8
Change-Id: Iba8047b3f70d04aa46d9688a824f3d49c1c58e90
For real time CBR mode, use model_rd_for_sb_y for 32x32 if the mode is
newmv last, which is less aggressive in skipping transform and
quantization, to avoid quality regression in some conditions.
Change-Id: Ifa30be587f2a8a4a7f182a172de6ce277c0f8556
For forced key frames in particular this helps to make them
blend better with the surrounding frames where noise tends
to be suppressed by a combination of quantization and alt
ref filtering.
Currently disabled by default under and IFDEF flag pending
wider testing.
Change-Id: I971b5cc2b2a4b9e1f11fe06c67ef073f01b25056
Bug fix: The crash is caused by not allocating buffer for prev_mip in
postproc_state and prev_mip in postproc_state is only used for MFQE,
ohter postproc modules, deblocking and etc., should not use it.
BUG=webm:1251
Change-Id: I3120d2f50603b4a2d400e92d583960a513953a28
add a trailing ':', though it's optional with the tools we support, it's
more common to use it to mark a label. this also quiets the
orphan-labels warning with nasm/yasm.
BUG=b/29583530
Change-Id: I46e95255e12026dd542d9838e2dd3fbddf7b56e2
For real-time mode, increase variance threshold for 32x32 blocks in
var-based partitioning for resolution >= 720p, so that it is more
likely to stay at 32x32 for high resolution which accelerates the
encoding speed with little/no PSNR drop.
PSNR effect on different speed settings:
speed 8 rtc: 0.02 overall PSNR drop, 0.285% SSIM drop
speed 7 rtc: 0.196% overall PSNR increase, 0.066% SSIM increase
speed 5 rtc_derf: no effect.
Speed up:
gips_motion_WHD, 1mbps: 2.5% faster on speed 7, 2.6% faster on speed8
gips_stat_WHD, 1mbps: 4.6% faster on speed 7, 5.6% faster on speed8
Change-Id: Ie7c33c4d2dd7d09294917e031357fc5476c3a4bb
Avoids a segfault in high-bitdepth builds.
This restores the condition to its state prior to:
7991241 vp9: Change the scheme for modeling rd for bsize 32x32.
BUG=webm:1250
Change-Id: I6183d5b34cb89dfbf27b7bb589812148a72cd7de
For real-time CBR mode, use model_rd_for_sb_y_large instead of
model_rd_for_sb_y for 32x32 block. In the former model, transform
might be skipped more aggressively in some condtions, which speeds
up encoding time with only a little PSNR/SSIM drop on rtc test set.
No obvious visual quality regression.
PSNR effect on different speed settings:
speed 8 rtc: 0.129% overall PSNR drop, 0.137% SSIM drop
speed 7 rtc: 0.135% overall PSNR drop, 0.062% SSIM drop
speed 5 rtc_derf: 0.105% overall PSNR drop, 0.095% SSIM drop
Speed up:
gips_motion_WHD, 1mbps: 3.29% faster on speed 7, 2.56% faster on speed8
gips_stat_WHD, 1mbps: 2.17% faster on speed 7, 1.62% faster on speed8
BUG=webm:1250
Change-Id: I818babce5b8549b4b1a7c3978df8591bffde7173
decoder_peek_si_internal could potentially read more bytes than
what actually exists in the input buffer. We check for the buffer
size to be at least 8, but we try to read up to 10 bytes in the
worst case. A well crafted file could thus cause a segfault.
Likely change that introduced this bug was:
https://chromium-review.googlesource.com/#/c/70439 (git hash:
7c43fb6)
BUG=chromium:621095
Change-Id: Id74880cfdded44caaa45bbdbaac859c09d3db752
In vp9_pick_inter_mode(), instead of using
vp9_get_pred_context_switchable_interp(xd) to assign filter_ref,
we use a less strict condition on assigning filter_ref.
This is to reduce the probabily of entering the flow of not
assigning filter_ref and then skipping filter search.
Overall PSNR gain 0.074% for rtc dataset
Details:
Low Mid High
0.185% -0.008% -0.082%
Change-Id: Id5c5ab38d3766c213d5681e17b4d1afd1529e676
For real-time CBR mode, use model_rd_for_sb_y_large instead of
model_rd_for_sb_y for 32x32 block. In the former model, transform
might be skipped more aggressively in some condtions, which speeds
up encoding time with only a little PSNR/SSIM drop on rtc test set.
No obvious visual quality regression.
PSNR effect on different speed setting:
speed 8 rtc: 0.129% overall PSNR drop, 0.137% SSIM drop
speed 7 rtc: 0.135% overall PSNR drop, 0.062% SSIM drop
speed 5 rtc_derf: 0.105% overall PSNR drop, 0.095% SSIM drop
Speed up:
gips_motion_WHD, 1mbps: 3.29% faster on speed 7, 2.56% faster on speed8
gips_stat_WHD, 1mbps: 2.17% faster on speed 7, 1.62% faster on speed8
Change-Id: I902f62def225ea01c145d7e5a93497398b8f5edf
This commit adds an encoder workaround to support better
compatibility with a non-compliant hardware vp9 profile 2 decoder.
The known issue with this decoder is:
The decoder assumes a wrong value, 127 instead of the correct
value of 511 and 2047, for any assumed top-left corner pixel in
UV planes for 10 and 12 bit, respectively. Such assumed
top-left corner pixel is used for INTRA prediction when a real
decoded/reconstructed pixel is not avalable, e.g. when it is
located inside the row above the top row or inside the column
left to the leftest column of a video image.
Change-Id: Ic15a938a3107e1b85e96cb7903a5c4220986b99d
development has moved to the nextgenv2 branch and a snapshot from here
was used to seed aomedia
BUG=b/29457125
Change-Id: Iedaca11ec7870fb3a4e50b2c9ea0c2b056a0d3c0
This commit refactors the trellis coefficient optimization process.
It saves multiplications used to generate the final dequantized
coefficients. It removes two memset operations on quantized
and dequantized coefficient sets. This improves the unit speed
by 10%.
Change-Id: I23f47c6e14582520a7f952f03ce8f72183e7f0e6
Since combining VPX_DL_REALTIME with VPX_RC_FIRST_PASS is basically
nonsense, ignore the user's pass setting when this happens and
behave as if the requested encode is a single pass encode.
BUG=webm:1233
Change-Id: I5ee4c4e5838c4ca6d24988890aae490b10826db2
For VBR: (1) allow newmv mode for golden ref to
select interpolation filter (as in last ref case), and
(2) don't use the more aggressive tx-skip testing logic for large blocks.
Only affects 1 pass real-time vbr mode (speed >= 5).
PSNR/SSIM metrics on ytlive set are all positive, ~0.5-2% gain.
Change-Id: I0ffbb0a9755563a5acd6230c58236e4f19a47266
This change is only for real-time mode if short_circuit_low_temp_var
is on. Add bias to last frame in choosing ref frame for partitioning,
when y_sad and y_sad_g are close. It speeds up real-time encoding by
0.5% on some clips with less than 0.1% overall PSNR drop on rtc test set.
Change-Id: I2a2110fe36455f3d8f0fc404aef2228f512e8df8
1. Skip golden non-zeromv and newmv-last for bsize >= 16x16 if the
temporal variance obtained from choose_partitioning is very low.
2. Skip horz and vert INTRA mode for speed 8.
This change works best on the clips with little noise and with some
motion (e.g. gips_motion which has > 5% speed up). PSNR drop is 1.78%
on rtc test set, no obvious visual quality regression found.
Change-Id: Ib43b5b20e67809d03c5a6890818ddff59e1fc94a
Move initialization of a some new "twopass" values
to the function vp9_init_second_pass() and some other
small changes.
Remove #if GROUP_ADAPTIVE_MAXQ as this is always
enabled now.
Change-Id: I1dbec2fd7c419779848aa987c4cd7824d4df8456
On scene-cut detected frames (i.e., high_source_sad = 1), use
nonrd_pick_partition (over choose_part + select_part), as
the nonrd_pick partitioning is generally better.
Small positive increase in metrics on ytlive set (~0.5 - 1%).
Negligle overall speed decrease, as its only used on scene-cut frames.
Only affects 1 pass vbr mode, speed = 5.
Change-Id: I07c89cbdc75f5bb16eb8e0e2773ead0980d2de5c
This reverts commit be12fefa4b
and commit 057c1c4034.
Also, the mismatch between the avx version and the
c version has been fixed.
BUG=https://bugs.chromium.org/p/webm/issues/detail?id=1168
For a rt encode using 1080p@60fps material, up to 11% performance
improvement overall was seen.
Change-Id: Icd1f216209ebc6fc0b8da885f32f356fa4355ed0
The eob of a block is not perperly set when skip_recode is true,
thus triggering assert(eob <= default_eob) to fail.
Change-Id: Ifecbe33dce2dc4903e0a80bd384dc09bf0dd8a44
Code cleaup, use existing rolling_actual/target metrics instead,
set threshold to get same/similar effect.
Little/no change in metrics on ytlive set.
Change-Id: I74f3c3d0a143a9cf20dc9c3dee54c0f7e6a97a51
Add a max condition and lower the min value.
No change in behavior (metrics for yt live set) for the
default min/max_gf_interval=4/16 settings.
Small positive change when min/max_gf_interval=7/16
(for 60fps clips on ytlive set).
Change-Id: I1c1d72425c86c69419ea43fb9730130e81062f91
Error messages:
..\vp9\common\vp9_loopfilter.c(1312): warning C4244: 'function' :
conversion from 'uint64_t' to 'unsigned int', possible loss of data
[.build-x86_64-win64-vs10\vpx.vcxproj]
..\vp9\common\vp9_loopfilter.c(1313): warning C4244: 'function' :
conversion from 'uint64_t' to 'unsigned int', possible loss of data
[.build-x86_64-win64-vs10\vpx.vcxproj]
..\vp9\common\vp9_loopfilter.c(1312): error C2220: warning treated as
error - no 'object' file generated
[.build-x86_64-win64-vs10\vpx.vcxproj]
Change-Id: Ia69260611997cd2ba41c7184a85ecead740a7c07
Increase in the damping used in adjusting the active Q range.
This does hurt rate accuracy a little in a few extreme cases
especially if the clip is very short*, but helps metrics.
* Note that the adjustment is applied at the GF/ARF group level based
on what happened in the last group. Hence for very short clips where
the length of a single group may be a significant % of the clip length
there is still scope for some drift that cannot be accommodated.
In practice most data points in our test sets are now much closer to target
than was previously the case with default settings and in some cases are
better even than they were with the command line undershoot and overshoot
parameter was set very low (e.g. 2%). For example in bridge_close at high rates
the old mechanism was unable to adapt enough to prevent extreme overshoot.
Change-Id: I634f8f0e015b5ee64a9f0ccaa2bcfdbc1d360489
Change to the calculation of the error divisor used in
get_twopass_worst_quality(). This follows on from other
changes to the rate control that impact the output of this
function.
Change-Id: I414fa9aa1e6a68a64dccea17c3712f44b8a0c10c
Changes to the function the redistributes bits from overshoot
or undershoot throughout the rest of the clip to respond more
quickly.
Change-Id: I90f10900cdd82cf2ce1d8da4b6f91eb5934310da
Added a factor based on the bit spend in the last arf group vs the
target to adjust the choice of the active worst quality in subsequent
groups.
Helps clips where previously there was a big overshoot or undershoot
to adapt and get closer to the target rate.
Change-Id: I67034b801679b99024409489a2273ea6fe23b8e6
The use of this value is preventing rate adjustment on clips
or sections that have very little motion but high noise and
this can give rise to some sections with massive overshoot.
Change-Id: I9a65c7c1148dc5d3a7d8b23e50fc1733f3661621
Skip intra-mode and some inter-modes (newmv, nearmv, nearestmv) for
golden frame if the variance got from choose_partitioning is very low.
Only for 1 pass real-time CBR mode and bsize >= 32x32, it has ~2.5%
speed up with less than 0.1% PSNR drop for rtc test set. Don't see
visual regression.
Change-Id: I70efbc95a1007231ae36f02c5b2fbf6cd35077ad
Reduce operations and jumps. perf shows CPU time reduced from 1.9% to
1.6% when decoding fdJc1_IBKJA.248.webm on Xeon E5.
Will apply the changes to vp10 after code review.
Change-Id: I9351509922855d8896ddef1ed093b3ca12619a61
For non-rd pickmode:
best_pred_sad, computed for NEWMV-last, is only used for
skipping golden non-zero modes. Add condition to avoid this
computation if not used (i.e, if golden nonzero modes are not used).
And remove code for computing best_pred_sad for NEWMV-golden,
since that sad is not used.
No change in behavior; small speed gain (~1%) for svc encodes.
Change-Id: Ic2cbdef6c4e9a233a57c0db0eeac8ad5fcead366
Function level timing test shows about 27% time saving on
a Xeon E5-2680 v2 desktop.
Rename vp9_dct_sse2.c to vp9_dct_intrin_sse2.c for vp9 and
rename dct_sse2.c to dct_intrin_sse2.c for vp10 to avoid
duplicate basenames.
Actually vp9_fwht4x4_mmx/sse2() and vp10_fwht4x4_mmx/sse2()
are identical. TODO: They should be unified later if there is
no intention to keep a duplicate.
Change-Id: I3e537b7bbd9ba417c606cd7c68c4dbbfa583f77d
- Avoid excessive copying
- Don't both searching if no update can possibly offer savings
- Simplify the interface
- Remove the confusing vp9_cost_upd256 macro
Change-Id: Id9d9676a361fd1203b27e930cd29c23b2813ce59
vp9_diamond_search_sad_avx was disabled in:
057c1c4 disable vp9_diamond_search_sad_avx
this removes a missing prototype warning as the prototype is no longer
included in vp9_rtcd.h. the file can be restored if someone gets around
to fixing the issue.
BUG=https://bugs.chromium.org/p/webm/issues/detail?id=1168
Change-Id: Ia9fda4b81c53dc5fba7c31d780d761f886940b52
Move the logic for rechecking zeromv on denoised block out to simplify
the function. To simplify the param passing, add a new structure
VP9_PICKMODE_CTX_DEN which is only used when denoiser is enabled.
Change-Id: Iaa9b4396dfcb8147236c02d4a1868a09103a4476
The inlining mirrors what was done with the low bit depth
inter_predictor. And the new highbd_inter_predictor name is more
consistent with other high bit depth functions.
Change-Id: I96437f745759aeec6260c6e39a974bf36f1c211c
Rename and change to how its updated.
Only affects 1 pass vbr.
Small change in metrics (< ~0.1%) on ytlive set.
Change-Id: Ibb1fe485699b6c4a8194951c8f229abe2f64b9a5
Added actual and absolute rate miss values to the opsnr.stt
stats output line.
Changes to the borg graphing may be needed before merge.
Change-Id: I1e9d548ce445d29002f0c59ebfd3957a6f15e702
Bug found by Yunqing relating to the correction for size at 8K and
above in get_twopass_worst_quality().
The basis for the correction was changed to the linear size relative to
1080P as a baseline and the adjustment has been clamped to prevent
problems at extreme images sizes.
For 1080P the results on our test sets were neutral but the low res and
mid res sets saw a small gain (0.1%-0.2% average).
I would also expect some gains on 4k and larger content where the
previous correction was overly aggressive.
Change-Id: I30b026b5f4535e9601e3178d738066459d19c8fb
Add control API VP9E_SET_TARGET_LEVEL that allows the encoder to
control the output bitstream level and/or keep level related
statistics.
Usage:
255 do not care about level (default)
0 keep level related stats only
10 target for level 1
11 target for level 1.1
.
.
.
62 target for level 6.2
Usage for vpxenc:
--target-level=0/255/10/11...
Change-Id: I31d1aeca19358b893e7577b4e63748c8e614034a
block_variance: This operates on 8x8s and would be safe with a int32 *
int32 to uint32 multiply, but this is potentially unsafe for 12-bit
input. Unfortunately the code already segfaults on 12-bit input:
https://bugs.chromium.org/p/webm/issues/detail?id=1223
calculate_variance: This operates on up to a 32x32 of 8x8s and can
overflow even with 8-bit input (log2((256*32*32)**2) == 36).
BUG=https://bugs.chromium.org/p/webm/issues/detail?id=1220
Change-Id: I1ca4ff6092db9a7580da371ee9a21f403fdadc40
Reduce factor for setting base-qp for active_best_quality (for inter-frames).
Small increase in metrics on yt live set.
Change-Id: I9cf0ac797783aeddbfaf1ff510696c9035d7c5ee
* changes:
vp9_frame_scale_ssse3.c: make 2 functions static
vp9_pickmode.c: make function static
vp9_noise_estimate.c: make function static
vp9_aq_360.c: add missing include
vp9_idct_intrin_sse2: add missing vp9_rtcd.h include
vpx_dsp/*.[hc]: add missing vpx_dsp_rtcd.h include
Makes the delta-qp stop little earlier on areas that have been refreshed enough.
This helps to reduce some pulsing artifact on noisy flat areas observed in some
noisy vc-clips.
Threshold changes only take effect for sources where noise level is estimated to
be >= medium level.
Only affects 1 pass CBR, non-screen content case.
Change-Id: Iacf557f6aa8abbcd6782c02ff2e6c14891960850
For 1 pass vbr mode:
Refactor to move the logic for gf setting based on up-coming
key frames to a separate function, so same logic can be used for
scene-cuts/changes.
Change-Id: Ic4ede308e08ba869bb62e4566e19ea31222c5229
Makes the noise estimation react little faster.
Little/no change in metrics.
Change only affects 1 pass cbr.
Change-Id: I13f0daa90ecbf9d49eb1cf2e48febd9d92292940
"qc" in vp{9,10}_token_state is used to save quantized coefficients, this
commit changes the type from short to tran_low_t to properly reflect
the value range for highbitdepth build.
This fixes an out-of-range bug when optimize_b is used in highbitdepth
build.
Change-Id: Ibf330879e6ac6ae8f099e085caa9d3d9a889fde8
This is an actual overflow where the result of the calculation is
materially changed, not just a negative value that is stored in an
unsigned.
Caught with fsanitize=integer on the VP9/AqSegmentTest.TestNoMisMatchAQ2/1 test.
Change-Id: I514b0ef4ae7ad50e3e08c0079aa204d59fa679aa
Move skin superblock force split out of this function as well
as some minor code refactors. Checked bitexact for different speed
settings and different resolutions.
Change-Id: I6078cbe88dd9ce6c0b69470a8a0a8f8d2274161b
First, we only set use_4x4_partition for key frame where we don't
denoise; second, envision we have small partitions, we should pass the
actual block size to denoiser and make an early termination if needed.
Change-Id: I331f42046d792b17360723d17ff817d601394658
Wrap around behavior is enforced manually and we use the values in
arithmetic involving negative integers.
Change-Id: I199706b6f3af91f4fb6fe2ef302fbbc6d0cf5785
Block size passed into denoiser filter is always >= BLOCK_8X8 (in
vp9_pick_inter_mode), it is not necessary to check smaller block
size. Passed the bitexact test on clips with different resolutions and
noise levels.
Change-Id: I19fa3195d18c27d9e5de60dc11cff1522ef3714e
Fix will reset the consec_zero_mv map on non-skipped blocks with non-zero mv.
Adjust thresholds on consec_zero_mv in noise estimation and skin detection,
as more possible reset on map means lower thresholds should be used.
Change-Id: Ibe8520057472b3609585260b51b6f95a38fb777d
In VP9 internal denoiser, motion magnitude is computed from
best_sse_mv, which should be set to 0 at the begining. This bug may
cause visual aritifact in denoiser. Also, delete two improper comments.
Change-Id: I8710d2acba23320bc85cf72af17d65245c19438b
Need to check that sse for non-zero mv has been set for the current block
(i.e., check that nonzero-mv is tested as a mode, so newmv_sse != UINT_MAX)
before forcing to not use zero-mv for denoising.
Also increase some thresholds (sse and sse_diff) for high noise case,
and use shift operaton instead of multiplication on a threshold computation.
Change-Id: Iae7339475d57240316b7fa8b887c4ee3c0d0dbec
