748 Commits

Author SHA1 Message Date
Jingning Han
6515afc6b9 Merge "Add min_tx_size variable to recursive transform block partition system" into nextgenv2 2016-11-08 19:14:33 +00:00
Yaowu Xu
f6e958b604 Merge "Fix the bug that PVQ commit broke dering" into nextgenv2 2016-11-08 18:00:53 +00:00
Angie Chiang
13ea019574 Merge changes Ib9428dc9,Ide04717a,If1dba7d8,I6da97880 into nextgenv2
* changes:
  Merge rd_stats only when it is valid
  Let parentheses in handle_inter_mode be symmetric
  Add RD_STATS into MB_MODE_INFO
  Add txb_coeff_cost_map
2016-11-08 17:42:04 +00:00
Jingning Han
e67b38aa7c Add min_tx_size variable to recursive transform block partition system
Replace max_tx_size with min_tx_size for transform type decision.

Change-Id: I64e39923a67903d52b381bd93eaac33b3400a201
2016-11-08 09:36:54 -08:00
Yushin Cho
48f84dbd1c Fix the bug that PVQ commit broke dering
Since PVQ's max block size equals to the max transform size,
daala's definition of OD_BSIZE_MAX was changed from 5 down to 4 to
use AV1's max trasform size 32x32. However, dering also uses
OD_BSIZE_MAX and assumes its value is 5, which caused dering
not working.

Change-Id: I9d82bb24adc7d57552a8e0a8a7e798e77d96fd4b
2016-11-08 08:15:57 -08:00
Brennan Shacklett
e0b5ae8c4e Remove multiple coefficient buffers from PICK_MODE_CONTEXT
This commit is a manual cherry-pick from aom/master:
45592a39d3b00aee4d6bd70da669400017b7a5d8

Only part of the changes apply in nextgenv2

Change-Id: I1e22514c6fe5af556710254278f2f8a5805db999
2016-11-08 03:50:10 +00:00
Yaowu Xu
fc1b213af1 Use block_size for max_scan_line in pvq decoding
Change-Id: I642bc205a7d2c4d472385fbeb4323e62e17984b4
2016-11-08 00:55:23 +00:00
Yaowu Xu
856c55e93d Add transform parameter initialization
The initialization of transform parameters was missing, that led to a
crash in encoder.

Change-Id: I9e35830d5f24e771c845f0d8881671d6b7228c5e
2016-11-08 00:36:30 +00:00
Tristan Matthews
cc37e36683 pvq: drop unused declaration
Change-Id: I5d95bb897d335dc17aa0ae5e873ba7dee46c6fda
2016-11-07 22:14:50 +00:00
Yaowu Xu
f591782085 Fix compiler warning of out-of-bound array access
Change-Id: I00f147cd372cedc5038708b0f23f6fae68918528
2016-11-07 22:14:24 +00:00
Yaowu Xu
dc9720433f Merge "Fix compiler warning of un-used variables" into nextgenv2 2016-11-07 22:03:12 +00:00
Yaowu Xu
4007fa6852 Merge "change to call fwd_txfm()" into nextgenv2 2016-11-07 22:03:01 +00:00
Yaowu Xu
c4c21734d6 Merge "Resolve merge issues with --enable-pvq" into nextgenv2 2016-11-07 20:31:33 +00:00
Yaowu Xu
02d4c3b780 Fix compiler warning of un-used variables
Change-Id: I17d05bbf75a201fd010fc17e2d9bd0db8ef36d41
2016-11-07 19:56:13 +00:00
Yaowu Xu
3442b4b159 change to call fwd_txfm()
The transform functions have been refactored in nextgenv2, this commit
resolves the calls in pvq patch to use this new scheme.

Change-Id: I1b56e75106a3357bb19bd7df2b4ba305eb9ed185
2016-11-07 10:40:41 -08:00
Yaowu Xu
d6ea71cf73 Resolve merge issues with --enable-pvq
This commit resolves some compiling issues due to merge.

Change-Id: I0eef8aa36c404e185e0b0004948a49307c360d3e
2016-11-07 10:35:55 -08:00
Debargha Mukherjee
03dc29bdf3 Merge "Fix bug in bicubic filter in warped_motion.c" into nextgenv2 2016-11-07 17:58:47 +00:00
Yaowu Xu
00a0e010f7 Merge "New experiment: Perceptual Vector Quantization from Daala" into nextgenv2 2016-11-07 16:00:32 +00:00
David Barker
f23bdca6a8 Fix bug in bicubic filter in warped_motion.c
Previously, do_cubic_filter would return results with the
wrong precision if the sample point was exactly aligned to
a pixel.

Change-Id: I40139f9a6701a8e72e691f37bb352f7814a7f306
2016-11-07 13:47:13 +00:00
Yushin Cho
77bba8d30a New experiment: Perceptual Vector Quantization from Daala
PVQ replaces the scalar quantizer and coefficient coding with a new
design originally developed in Daala. It currently depends on the
Daala entropy coder although it could be adapted to work with another
entropy coder if needed:
./configure --enable-experimental --enable-daala_ec --enable-pvq

The version of PVQ in this commit is adapted from the following
revision of Daala:
fb51c1ade6

More information about PVQ:
- https://people.xiph.org/~jm/daala/pvq_demo/
- https://jmvalin.ca/papers/spie_pvq.pdf

The following files are copied as-is from Daala with minimal
adaptations, therefore we disable clang-format on those files
to make it easier to synchronize the AV1 and Daala codebases in the future:
 av1/common/generic_code.c
 av1/common/generic_code.h
 av1/common/laplace_tables.c
 av1/common/partition.c
 av1/common/partition.h
 av1/common/pvq.c
 av1/common/pvq.h
 av1/common/state.c
 av1/common/state.h
 av1/common/zigzag.h
 av1/common/zigzag16.c
 av1/common/zigzag32.c
 av1/common/zigzag4.c
 av1/common/zigzag64.c
 av1/common/zigzag8.c
 av1/decoder/decint.h
 av1/decoder/generic_decoder.c
 av1/decoder/laplace_decoder.c
 av1/decoder/pvq_decoder.c
 av1/decoder/pvq_decoder.h
 av1/encoder/daala_compat_enc.c
 av1/encoder/encint.h
 av1/encoder/generic_encoder.c
 av1/encoder/laplace_encoder.c
 av1/encoder/pvq_encoder.c
 av1/encoder/pvq_encoder.h

Known issues:
- Lossless mode is not supported, '--lossless=1' will give the same result as
'--end-usage=q --cq-level=1'.
- High bit depth is not supported by PVQ.

Change-Id: I1ae0d6517b87f4c1ccea944b2e12dc906979f25e
2016-11-06 22:18:01 -08:00
Angie Chiang
616990d607 Merge rd_stats only when it is valid
Change-Id: Ib9428dc9b6e224fdb5d410368c5b92042c96f68a
2016-11-06 15:25:37 -08:00
Angie Chiang
78a3bc165c Let parentheses in handle_inter_mode be symmetric
Change-Id: Ide04717a8ce2a7c1245f9614485647e296e96abd
2016-11-06 13:01:16 -08:00
Angie Chiang
9a44f5fbc8 Add RD_STATS into MB_MODE_INFO
With RD_STATS in MB_MODE_INFO, we will be able to compare the results
from rate-distortion loop and the results from bitstream packing.

Change-Id: If1dba7d87126577a6f369ac087d4517f7cebb0c5
2016-11-06 12:21:34 -08:00
Angie Chiang
85279f6668 Add txb_coeff_cost_map
The txb_coeff_cost_map is a 16x16 map which records each single
transform block's cost from the transform block's location in 4-pixel
unit in recursive transform experiment.

Change-Id: I6da97880c457680594bca56617084010891beaa2
2016-11-06 11:55:17 -08:00
Debargha Mukherjee
5f305854e6 Increase gm precision from 16 to 32 bit ints
Change-Id: I7117a6c14dc8438e4225b50bd2d3ebbaa7f850cc
2016-11-05 16:50:08 -07:00
James Zern
99ff89b6fb Merge "rdopt: clear maybe-uninitialized variable warning" into nextgenv2 2016-11-05 03:19:33 +00:00
Sarah Parker
70c4fab569 rdopt: clear maybe-uninitialized variable warning
av1/encoder/rdopt.c:9533 ‘zeromv[1].as_int’ may be used
uninitialized in this function [-Wmaybe-uninitialized]
this was spurious given the logic in the if

Change-Id: I8ddfe7e46d1bf5593cc8624f05c9f181243a87d4
2016-11-04 17:56:23 -07:00
Jingning Han
713b56121a Merge "Clean up write_tx_type()" into nextgenv2 2016-11-04 23:03:53 +00:00
Jingning Han
0880b5466f Merge "Refactor tx_type reader" into nextgenv2 2016-11-04 23:03:47 +00:00
Jingning Han
05abee1530 Merge "Factor out common tx_type writing codes from inter/intra frame" into nextgenv2 2016-11-04 23:03:38 +00:00
Angie Chiang
7a77169a35 Merge changes Ia37f170d,Ie3082db5 into nextgenv2
* changes:
  Record YUV planes' txfm block coeff cost in handle_inter_mode()
  Separate coefficient cost of U/V planes in write_modes_b()
2016-11-04 22:58:58 +00:00
Angie Chiang
59526ead45 Merge changes I3bc782d6,I8359e849,Iae50d0b0,Id1704d88,Ia69f13c4, ... into nextgenv2
* changes:
  Add av1_ prefix on ###_rd_stats functions
  Use init_rd_stats() in encodeframe.c
  Add transform block coefficient cost in RD_STATS for debugging
  Add helper functions to modify RD_STATS
  Add mi_row and mi_col into mbmi to facilitate rd_debug process
  Add token cost comparison in write_modes_b()
2016-11-04 22:43:30 +00:00
James Zern
653fdd6d55 Merge changes I139808f4,I3d97d8db into nextgenv2
* changes:
  warped_motion.c: delete unused filter_4tap[]
  warped_motion.c: quiet float-conversion warnings
2016-11-04 22:34:11 +00:00
Angie Chiang
628d7c915b Record YUV planes' txfm block coeff cost in handle_inter_mode()
Change-Id: Ia37f170d8fd961d78a751d84b9525ab7e973b81a
2016-11-04 11:12:44 -07:00
Angie Chiang
c0feea8a0c Add av1_ prefix on ###_rd_stats functions
Change-Id: I3bc782d68bcd9b52b38210eec9eecb21146fde75
2016-11-04 11:12:44 -07:00
Angie Chiang
75f45814ba Separate coefficient cost of U/V planes in write_modes_b()
Change-Id: Ie3082db5b0fead8c322b2aeede4eff7cd723ea12
2016-11-04 11:12:44 -07:00
Angie Chiang
4695b97030 Use init_rd_stats() in encodeframe.c
Change-Id: I8359e8498efd301ff81eea1d7466d0f3fec5e006
2016-11-04 11:11:27 -07:00
Angie Chiang
d81fdb41e6 Add transform block coefficient cost in RD_STATS for debugging
Change-Id: Iae50d0b0c4f8f383ab4f91d2c1c2fa4e799c7250
2016-11-04 11:11:27 -07:00
Angie Chiang
d7246171b5 Add helper functions to modify RD_STATS
Those functions includes
init_rd_stats()
invalid_rd_stats()
merge_rd_stats()

This CL help simplify the code.

Change-Id: Id1704d883bd21a039b0478a940994ca14184ae1c
2016-11-04 11:11:27 -07:00
Angie Chiang
394c337754 Add mi_row and mi_col into mbmi to facilitate rd_debug process
Change-Id: Ia69f13c47f2dd34fabd220652691049166a06a68
2016-11-04 11:09:24 -07:00
Angie Chiang
d402282f69 Add token cost comparison in write_modes_b()
This is just partial implementation
Compare token cost of pack_mb_tokens/pack_txb_tokens with token cost
from rate-distortion loop. If there is any difference, dump out mode
info.

Change-Id: I46b373ee2522c5047f799f36baf7cec5fbc06f06
2016-11-04 11:09:24 -07:00
Jingning Han
4be3214fec Merge "Properly schedule the transform block recursion order" into nextgenv2 2016-11-04 17:53:53 +00:00
Jingning Han
641b1ad5ad Clean up write_tx_type()
Remove repeated mbmi->tx_size calls.

Change-Id: I3e4e03b69b2efffd860cc1ea34e150f4257bf081
2016-11-04 10:36:20 -07:00
Jingning Han
ab7163db08 Refactor tx_type reader
Factor out common codes. Remove repeated mbmi->tx_size calls.

Change-Id: Id5de35e88f1a5f16223eaa06fc2c9f69124061ef
2016-11-04 10:35:34 -07:00
Jingning Han
2a4da9476b Factor out common tx_type writing codes from inter/intra frame
Change-Id: Id2626bd19db2504756d9a1dee709c2d08c79f771
2016-11-04 10:33:12 -07:00
Yue Chen
95a3898cbd Merge "Remove duplicated variables in EXT_INTER" into nextgenv2 2016-11-04 17:11:08 +00:00
Jingning Han
98d6a1f247 Properly schedule the transform block recursion order
This commit replaces the offset based block index calculation with
incremental based one. It does not change the coding statistics.

Change-Id: I3789294eb45416bd0823e773ec30f05ed41ba0dc
2016-11-04 09:06:49 -07:00
Jingning Han
137b2671eb Fix format issue in handle_inter_mode()
Change-Id: I681fd799cf46991de419cc867ccb649a6990c19d
2016-11-04 08:31:24 -07:00
Debargha Mukherjee
68d695b7ca Merge "Further work on 64x64 fwd/inv transform support" into nextgenv2 2016-11-04 09:32:07 +00:00
Angie Chiang
e89ea0ceb7 Merge "Refactor: Replace rate dist sse skip by RD_STATS in VAR_TX" into nextgenv2 2016-11-04 05:42:59 +00:00