generic-library/vpx
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
15879fb457
vpx/vp9/common/vp9_default_coef_probs.h

785 lines
41 KiB
C
Raw Normal View History

Faster vp8_default_coef_probs Copies from a generated table instead of building the default coeff probabilities during runtime. Change-Id: I4d9551ea3a2d7d4a4f7ce9eda006495221a8de50
2011-08-16 22:21:21 +02:00
/*
* Copyright (c) 2010 The WebM project authors. All Rights Reserved.
*
* Use of this source code is governed by a BSD-style license
* that can be found in the LICENSE file in the root of the source
* tree. An additional intellectual property rights grant can be found
* in the file PATENTS. All contributing project authors may
* be found in the AUTHORS file in the root of the source tree.
*/
Add vp9_ prefix to all vp9 files Support for gyp which doesn't support multiple objects in the same static library having the same basename. Change-Id: Ib947eefbaf68f8b177a796d23f875ccdfa6bc9dc
2012-11-27 22:59:17 +01:00
/*Generated file, included by vp9_entropy.c*/
Faster vp8_default_coef_probs Copies from a generated table instead of building the default coeff probabilities during runtime. Change-Id: I4d9551ea3a2d7d4a4f7ce9eda006495221a8de50
2011-08-16 22:21:21 +02:00
Modeling default coef probs with distribution Replaces the default tables for single coefficient magnitudes with those obtained from an appropriate distribution. The EOB node is left unchanged. The model is represeted as a 256-size codebook where the index corresponds to the probability of the Zero or the One node. Two variations are implemented corresponding to whether the Zero node or the One-node is used as the peg. The main advantage is that the default prob tables will become considerably smaller and manageable. Besides there is substantially less risk of over-fitting for a training set. Various distributions are tried and the one that gives the best results is the family of Generalized Gaussian distributions with shape parameter 0.75. The results are within about 0.2% of fully trained tables for the Zero peg variant, and within 0.1% of the One peg variant. The forward updates are optionally (controlled by a macro) model-based, i.e. restricted to only convey probabilities from the codebook. Backward updates can also be optionally (controlled by another macro) model-based, but is turned off by default. Currently model-based forward updates work about the same as unconstrained updates, but there is a drop in performance with backward-updates being model based. The model based approach also allows the probabilities for the key frames to be adjusted from the defaults based on the base_qindex of the frame. Currently the adjustment function is a placeholder that adjusts the prob of EOB and Zero node from the nominal one at higher quality (lower qindex) or lower quality (higher qindex) ends of the range. The rest of the probabilities are then derived based on the model from the adjusted prob of zero. Change-Id: Iae050f3cbcc6d8b3f204e8dc395ae47b3b2192c9
2013-03-13 19:03:17 +01:00
// NOTE: When the CONFIG_MODELCOEFPROB experiment is on, only the first
// 2 or 3 from each row is actually used depending on whether
// UNCONSTRAINDED_NODES is 2 or 3. If this experiment is merged
// the tables below should be shortened accordingly.
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
static const vp9_coeff_probs default_coef_probs_4x4[BLOCK_TYPES] = {
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
{ /* block Type 0 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 208, 32, 178, 198, 161, 167, 196, 147, 244, 194, 210 },
{ 102, 43, 132, 185, 148, 162, 185, 141, 237, 181, 215 },
{ 15, 36, 68, 143, 119, 151, 169, 133, 230, 173, 214 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 71, 91, 178, 226, 169, 176, 232, 170, 252, 219, 231 },
{ 72, 88, 174, 226, 168, 176, 232, 170, 252, 219, 234 },
{ 40, 79, 154, 222, 161, 174, 231, 169, 251, 219, 238 },
{ 21, 68, 126, 211, 144, 167, 230, 167, 252, 219, 236 },
{ 7, 49, 84, 175, 121, 152, 223, 151, 251, 218, 237 },
{ 1, 20, 32, 100, 97, 140, 163, 116, 237, 186, 222 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 108, 110, 206, 237, 182, 183, 239, 181, 252, 221, 245 },
{ 72, 98, 191, 236, 180, 182, 240, 183, 252, 223, 239 },
{ 26, 77, 152, 230, 166, 179, 239, 181, 252, 222, 241 },
{ 7, 57, 106, 212, 141, 167, 236, 173, 252, 223, 243 },
{ 1, 35, 60, 171, 110, 149, 225, 155, 251, 218, 240 },
{ 1, 14, 22, 90, 86, 134, 163, 116, 238, 181, 233 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 105, 139, 222, 245, 196, 192, 245, 195, 253, 229, 255 },
{ 76, 118, 205, 245, 192, 192, 247, 198, 254, 230, 255 },
{ 21, 88, 164, 240, 175, 186, 246, 197, 255, 232, 255 },
{ 5, 63, 118, 222, 149, 172, 242, 185, 255, 230, 254 },
{ 1, 42, 74, 186, 120, 157, 227, 161, 253, 220, 250 },
{ 1, 18, 30, 97, 92, 136, 163, 118, 244, 184, 244 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 143, 117, 233, 251, 207, 201, 250, 210, 255, 239, 128 },
{ 99, 104, 214, 249, 200, 199, 251, 211, 255, 238, 255 },
{ 26, 81, 170, 245, 183, 192, 250, 206, 255, 242, 255 },
{ 6, 60, 116, 226, 151, 176, 242, 187, 255, 235, 255 },
{ 1, 38, 65, 178, 114, 153, 224, 157, 254, 224, 255 },
{ 1, 15, 26, 86, 88, 133, 163, 110, 251, 197, 252 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 155, 74, 238, 252, 215, 206, 252, 223, 255, 255, 128 },
{ 152, 64, 223, 250, 205, 201, 254, 219, 255, 255, 128 },
{ 67, 55, 182, 246, 187, 192, 251, 210, 255, 240, 128 },
{ 27, 44, 127, 227, 155, 176, 244, 186, 255, 240, 255 },
{ 9, 27, 69, 176, 115, 152, 227, 154, 255, 229, 255 },
{ 2, 11, 28, 91, 84, 133, 177, 115, 254, 210, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 207, 112, 234, 244, 192, 193, 246, 194, 255, 237, 255 },
{ 145, 120, 212, 233, 178, 183, 232, 177, 252, 216, 228 },
{ 77, 114, 177, 214, 164, 174, 210, 159, 245, 199, 230 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 93, 174, 243, 248, 205, 200, 245, 195, 255, 232, 255 },
{ 100, 144, 231, 248, 204, 200, 244, 193, 255, 232, 255 },
{ 28, 101, 186, 247, 194, 199, 244, 194, 255, 232, 255 },
{ 9, 73, 132, 238, 155, 186, 245, 197, 255, 232, 250 },
{ 2, 44, 76, 187, 112, 151, 240, 172, 255, 235, 249 },
{ 1, 19, 33, 98, 92, 138, 176, 113, 252, 208, 249 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 116, 175, 246, 250, 212, 202, 248, 198, 255, 238, 255 },
{ 78, 142, 231, 250, 208, 203, 249, 200, 255, 241, 255 },
{ 14, 93, 177, 245, 186, 196, 248, 198, 255, 241, 255 },
{ 4, 65, 122, 227, 148, 177, 244, 186, 255, 241, 243 },
{ 1, 38, 69, 180, 111, 152, 235, 162, 255, 237, 247 },
{ 1, 18, 30, 101, 89, 133, 190, 116, 255, 219, 246 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 138, 183, 249, 253, 220, 209, 252, 210, 255, 251, 128 },
{ 93, 147, 237, 252, 213, 209, 253, 213, 255, 251, 128 },
{ 21, 104, 187, 247, 185, 196, 252, 210, 255, 249, 128 },
{ 6, 73, 131, 225, 147, 174, 248, 190, 255, 248, 128 },
{ 1, 47, 83, 189, 119, 155, 239, 167, 255, 246, 128 },
{ 1, 26, 44, 130, 96, 139, 209, 129, 255, 235, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 188, 143, 252, 255, 228, 218, 253, 218, 255, 209, 128 },
{ 137, 124, 241, 253, 215, 211, 254, 221, 255, 255, 128 },
{ 32, 89, 188, 248, 186, 198, 254, 216, 255, 253, 128 },
{ 7, 61, 122, 231, 146, 176, 252, 201, 255, 250, 128 },
{ 1, 34, 66, 186, 103, 149, 246, 176, 255, 249, 128 },
{ 1, 18, 34, 115, 91, 134, 217, 124, 255, 233, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 198, 92, 253, 255, 231, 222, 255, 230, 128, 128, 128 },
{ 189, 79, 244, 254, 220, 217, 255, 237, 255, 255, 128 },
{ 78, 61, 200, 252, 196, 207, 255, 231, 255, 255, 128 },
{ 34, 50, 146, 242, 161, 187, 255, 222, 255, 255, 128 },
{ 11, 38, 93, 215, 122, 159, 253, 202, 255, 255, 128 },
{ 1, 31, 55, 143, 102, 143, 227, 148, 255, 238, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
Restyle code Approximate the Google style guide[1] so that that there's a written document to follow and tools to check compliance[2]. [1]: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml [2]: http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py Change-Id: Idf40e3d8dddcc72150f6af127b13e5dab838685f
2012-07-14 00:21:29 +02:00
}
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
}, { /* block Type 1 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 207, 35, 219, 243, 195, 192, 243, 188, 251, 232, 238 },
{ 126, 46, 182, 230, 177, 182, 228, 171, 248, 214, 232 },
{ 51, 47, 125, 196, 147, 166, 206, 151, 245, 199, 229 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 114, 124, 220, 244, 197, 192, 242, 189, 253, 226, 255 },
{ 142, 116, 213, 243, 194, 191, 241, 188, 252, 226, 255 },
{ 81, 101, 190, 242, 188, 190, 242, 190, 253, 229, 255 },
{ 42, 83, 155, 235, 166, 183, 241, 190, 253, 227, 246 },
{ 16, 62, 104, 205, 133, 161, 238, 176, 254, 227, 250 },
{ 6, 40, 60, 132, 109, 145, 190, 128, 248, 202, 239 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 139, 149, 228, 248, 205, 198, 244, 196, 255, 223, 255 },
{ 115, 127, 221, 248, 202, 198, 245, 198, 255, 228, 255 },
{ 43, 100, 189, 246, 195, 195, 244, 196, 254, 234, 228 },
{ 13, 77, 141, 238, 168, 187, 243, 191, 255, 232, 255 },
{ 3, 49, 88, 203, 125, 160, 237, 178, 253, 227, 251 },
{ 1, 23, 41, 118, 97, 136, 191, 127, 250, 207, 247 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 119, 185, 236, 251, 216, 205, 249, 202, 253, 237, 255 },
{ 89, 140, 224, 251, 211, 205, 250, 208, 255, 241, 255 },
{ 34, 105, 189, 248, 195, 197, 250, 208, 255, 245, 255 },
{ 14, 78, 142, 235, 166, 182, 246, 194, 255, 242, 255 },
{ 5, 49, 90, 196, 128, 160, 235, 165, 255, 237, 255 },
{ 1, 22, 41, 114, 97, 139, 180, 124, 252, 201, 249 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 162, 142, 244, 254, 228, 215, 255, 230, 128, 128, 128 },
{ 129, 120, 231, 253, 216, 210, 255, 228, 255, 255, 128 },
{ 44, 90, 189, 249, 195, 199, 253, 217, 255, 240, 128 },
{ 14, 65, 132, 234, 158, 181, 249, 203, 255, 248, 128 },
{ 3, 38, 72, 188, 112, 154, 239, 171, 255, 243, 128 },
{ 1, 17, 39, 110, 86, 141, 201, 123, 255, 240, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 167, 96, 247, 255, 230, 218, 249, 231, 255, 255, 128 },
{ 163, 84, 234, 253, 214, 209, 255, 231, 255, 255, 128 },
{ 70, 63, 185, 249, 189, 197, 255, 230, 255, 255, 128 },
{ 30, 44, 132, 238, 157, 180, 251, 210, 255, 220, 128 },
{ 13, 30, 80, 195, 121, 153, 243, 179, 255, 224, 128 },
{ 5, 13, 38, 103, 109, 128, 196, 147, 255, 255, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 242, 90, 246, 244, 200, 192, 242, 189, 255, 234, 255 },
{ 186, 102, 228, 233, 187, 182, 231, 172, 254, 225, 252 },
{ 102, 108, 203, 228, 181, 180, 218, 167, 243, 201, 223 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 152, 169, 250, 253, 223, 209, 251, 208, 255, 250, 128 },
{ 164, 149, 242, 253, 222, 209, 249, 207, 253, 238, 255 },
{ 63, 108, 204, 252, 215, 211, 251, 211, 255, 242, 128 },
{ 39, 83, 153, 248, 175, 199, 250, 214, 255, 245, 128 },
{ 31, 66, 108, 214, 130, 161, 251, 196, 255, 237, 128 },
{ 27, 65, 71, 150, 112, 149, 213, 133, 255, 230, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 161, 174, 250, 254, 226, 215, 254, 226, 255, 230, 128 },
{ 133, 150, 239, 254, 222, 213, 254, 225, 255, 255, 128 },
{ 32, 105, 197, 252, 206, 207, 253, 220, 255, 255, 128 },
{ 10, 78, 147, 245, 173, 193, 253, 212, 255, 255, 128 },
{ 2, 49, 99, 221, 133, 164, 250, 198, 255, 252, 128 },
{ 1, 26, 53, 154, 96, 135, 234, 142, 255, 240, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 160, 187, 251, 255, 234, 223, 255, 233, 128, 128, 128 },
{ 131, 155, 241, 255, 228, 222, 255, 232, 255, 255, 128 },
{ 42, 108, 198, 253, 207, 212, 255, 234, 255, 255, 128 },
{ 18, 81, 151, 246, 176, 194, 254, 222, 255, 255, 128 },
{ 9, 60, 112, 225, 144, 167, 252, 199, 255, 255, 128 },
{ 5, 35, 49, 163, 113, 150, 237, 118, 255, 255, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 195, 141, 253, 255, 242, 232, 255, 255, 128, 128, 128 },
{ 169, 128, 245, 255, 235, 227, 255, 248, 128, 128, 128 },
{ 62, 91, 204, 255, 216, 220, 255, 233, 128, 128, 128 },
{ 23, 70, 150, 248, 178, 202, 255, 223, 128, 128, 128 },
{ 2, 44, 78, 220, 110, 164, 255, 209, 128, 128, 128 },
{ 1, 1, 128, 255, 255, 128, 128, 128, 128, 128, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 195, 104, 253, 255, 246, 246, 255, 171, 128, 128, 128 },
{ 197, 92, 248, 255, 239, 228, 255, 239, 128, 128, 128 },
{ 88, 71, 214, 255, 219, 220, 255, 244, 128, 128, 128 },
{ 39, 56, 160, 250, 187, 204, 255, 255, 128, 128, 128 },
{ 18, 28, 90, 217, 81, 137, 255, 128, 128, 128, 128 },
{ 128, 128, 128, 128, 128, 128, 128, 128, 128, 128, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
}
Restyle code Approximate the Google style guide[1] so that that there's a written document to follow and tools to check compliance[2]. [1]: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml [2]: http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py Change-Id: Idf40e3d8dddcc72150f6af127b13e5dab838685f
2012-07-14 00:21:29 +02:00
}
Faster vp8_default_coef_probs Copies from a generated table instead of building the default coeff probabilities during runtime. Change-Id: I4d9551ea3a2d7d4a4f7ce9eda006495221a8de50
2011-08-16 22:21:21 +02:00
};
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
static const vp9_coeff_probs default_coef_probs_8x8[BLOCK_TYPES] = {
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
{ /* block Type 0 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 196, 40, 199, 180, 158, 161, 172, 135, 226, 183, 140 },
{ 83, 38, 128, 153, 142, 157, 155, 128, 222, 164, 202 },
{ 10, 29, 55, 116, 113, 146, 150, 122, 223, 169, 200 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 33, 114, 160, 211, 155, 169, 223, 162, 248, 212, 215 },
{ 69, 107, 155, 210, 154, 169, 224, 163, 248, 212, 216 },
{ 30, 91, 138, 207, 150, 168, 223, 162, 248, 212, 216 },
{ 12, 74, 115, 200, 140, 164, 222, 160, 249, 212, 219 },
{ 4, 52, 80, 172, 121, 153, 216, 149, 249, 212, 226 },
{ 1, 27, 40, 105, 101, 141, 157, 120, 231, 177, 210 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 38, 159, 190, 227, 171, 177, 229, 172, 250, 214, 237 },
{ 34, 130, 182, 229, 173, 180, 231, 174, 249, 215, 234 },
{ 10, 97, 153, 226, 164, 178, 232, 175, 250, 215, 241 },
{ 3, 71, 115, 213, 145, 170, 230, 171, 251, 217, 235 },
{ 1, 41, 68, 172, 114, 152, 219, 154, 250, 212, 235 },
{ 1, 16, 27, 88, 90, 135, 155, 113, 235, 180, 216 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 41, 184, 214, 238, 187, 186, 235, 180, 252, 217, 236 },
{ 24, 142, 199, 241, 188, 189, 237, 184, 252, 220, 235 },
{ 6, 97, 159, 235, 172, 184, 239, 185, 252, 221, 243 },
{ 1, 63, 110, 214, 144, 170, 234, 174, 253, 223, 243 },
{ 1, 32, 58, 166, 109, 149, 218, 152, 251, 215, 238 },
{ 1, 12, 21, 78, 85, 131, 152, 109, 236, 180, 224 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 54, 207, 231, 245, 201, 193, 238, 186, 252, 221, 220 },
{ 32, 156, 213, 246, 198, 195, 242, 192, 252, 224, 245 },
{ 7, 98, 164, 240, 177, 187, 243, 193, 252, 227, 244 },
{ 2, 62, 108, 216, 143, 170, 237, 177, 254, 227, 248 },
{ 1, 32, 57, 165, 108, 148, 219, 152, 252, 217, 243 },
{ 1, 13, 22, 79, 87, 132, 153, 109, 240, 182, 232 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 89, 208, 239, 250, 216, 200, 240, 190, 255, 222, 219 },
{ 53, 155, 223, 250, 209, 202, 245, 199, 253, 225, 246 },
{ 12, 102, 170, 243, 183, 192, 246, 198, 254, 230, 255 },
{ 3, 67, 111, 218, 144, 171, 239, 180, 254, 231, 248 },
{ 1, 38, 60, 164, 108, 148, 221, 152, 253, 220, 246 },
{ 1, 18, 26, 81, 88, 132, 157, 108, 245, 188, 241 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 205, 121, 244, 237, 187, 188, 229, 174, 248, 215, 228 },
{ 140, 120, 211, 219, 174, 177, 207, 158, 241, 195, 214 },
{ 51, 100, 152, 198, 155, 168, 199, 148, 240, 193, 207 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 66, 196, 236, 247, 202, 197, 243, 193, 254, 228, 246 },
{ 99, 164, 223, 246, 199, 196, 243, 193, 254, 226, 255 },
{ 29, 122, 187, 244, 187, 194, 244, 193, 255, 227, 239 },
{ 14, 95, 145, 234, 156, 181, 244, 194, 254, 229, 246 },
{ 6, 68, 97, 190, 123, 155, 240, 168, 254, 232, 245 },
{ 3, 43, 50, 112, 105, 143, 170, 118, 245, 195, 230 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 66, 202, 238, 248, 206, 199, 245, 196, 254, 233, 244 },
{ 45, 155, 218, 248, 200, 199, 245, 197, 254, 229, 208 },
{ 6, 96, 163, 242, 178, 191, 245, 196, 254, 233, 228 },
{ 2, 64, 110, 224, 142, 175, 242, 185, 254, 232, 247 },
{ 1, 34, 61, 172, 103, 147, 232, 164, 254, 226, 244 },
{ 1, 13, 24, 82, 85, 133, 165, 105, 248, 199, 242 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 66, 204, 242, 251, 213, 204, 248, 204, 255, 236, 255 },
{ 38, 158, 222, 251, 206, 205, 249, 206, 255, 238, 255 },
{ 6, 95, 166, 244, 178, 194, 249, 205, 255, 236, 255 },
{ 2, 61, 111, 223, 141, 173, 244, 187, 255, 237, 255 },
{ 1, 31, 59, 171, 104, 149, 230, 158, 255, 230, 252 },
{ 1, 12, 22, 82, 79, 128, 171, 111, 251, 203, 249 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 63, 214, 245, 252, 219, 208, 249, 206, 255, 241, 128 },
{ 38, 164, 228, 252, 210, 208, 251, 212, 255, 245, 255 },
{ 5, 101, 174, 246, 182, 196, 251, 207, 255, 244, 255 },
{ 1, 64, 116, 224, 142, 174, 246, 190, 255, 241, 228 },
{ 1, 34, 63, 172, 105, 148, 233, 160, 255, 235, 237 },
{ 1, 14, 26, 88, 85, 130, 177, 110, 252, 210, 250 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 91, 214, 246, 254, 226, 213, 251, 210, 255, 239, 255 },
{ 55, 162, 233, 253, 215, 210, 253, 216, 255, 244, 128 },
{ 10, 104, 179, 247, 184, 196, 252, 212, 255, 247, 255 },
{ 2, 67, 119, 226, 143, 173, 249, 195, 255, 245, 255 },
{ 1, 37, 66, 175, 106, 149, 237, 164, 255, 240, 255 },
{ 1, 16, 30, 96, 87, 132, 188, 113, 255, 222, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
Entropy coding for hybrid transform Separates the entropy coding context models for 4x4, 8x8 and 16x16 ADST variants. There is a small improvement for HD (hd/std-hd) by about 0.1-0.2%. Results on derf/yt are about the same, probably because there is not enough statistics. Results may improve somewhat once the initial probability tables are updated for the hybrid transforms which is coming soon. Change-Id: Ic7c0c62dacc68ef551054fdb575be8b8507d32a8
2012-09-10 07:42:35 +02:00
}
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
}, { /* block Type 1 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 211, 32, 212, 235, 185, 184, 223, 167, 239, 210, 182 },
{ 121, 47, 171, 224, 171, 180, 211, 162, 238, 195, 221 },
{ 40, 51, 118, 203, 145, 168, 211, 160, 246, 200, 236 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 71, 129, 209, 244, 192, 194, 242, 188, 255, 230, 255 },
{ 118, 122, 206, 244, 192, 192, 241, 187, 254, 227, 255 },
{ 53, 104, 184, 241, 186, 190, 241, 184, 254, 232, 255 },
{ 20, 81, 148, 234, 168, 183, 240, 183, 254, 231, 240 },
{ 3, 47, 82, 197, 127, 160, 234, 166, 254, 228, 251 },
{ 1, 18, 28, 96, 88, 134, 174, 116, 247, 194, 247 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 86, 162, 220, 247, 203, 198, 245, 193, 255, 237, 255 },
{ 84, 134, 216, 247, 201, 197, 244, 192, 255, 233, 255 },
{ 26, 102, 186, 243, 190, 192, 244, 192, 255, 232, 255 },
{ 7, 75, 135, 231, 163, 181, 240, 183, 255, 234, 255 },
{ 1, 46, 79, 193, 121, 157, 233, 168, 255, 225, 242 },
{ 1, 20, 35, 113, 94, 136, 191, 123, 252, 209, 250 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 89, 191, 232, 250, 211, 203, 248, 202, 255, 230, 128 },
{ 67, 148, 223, 250, 207, 201, 250, 207, 255, 247, 255 },
{ 19, 105, 183, 245, 189, 193, 249, 202, 255, 244, 255 },
{ 5, 72, 127, 228, 156, 177, 245, 186, 255, 238, 255 },
{ 1, 44, 76, 190, 119, 156, 234, 167, 255, 231, 255 },
{ 1, 21, 36, 116, 92, 138, 195, 128, 250, 208, 241 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 94, 210, 236, 252, 215, 206, 253, 209, 255, 247, 128 },
{ 68, 153, 224, 251, 209, 204, 251, 213, 255, 240, 128 },
{ 14, 103, 178, 246, 188, 195, 251, 209, 255, 239, 128 },
{ 2, 70, 122, 230, 154, 177, 247, 194, 255, 239, 128 },
{ 1, 42, 72, 189, 115, 153, 234, 166, 255, 229, 255 },
{ 1, 19, 34, 104, 98, 143, 180, 124, 252, 200, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 87, 200, 238, 254, 226, 214, 250, 212, 255, 226, 128 },
{ 55, 151, 225, 253, 217, 212, 253, 217, 255, 233, 128 },
{ 11, 106, 179, 249, 193, 200, 252, 213, 255, 247, 128 },
{ 2, 72, 124, 232, 155, 180, 246, 195, 255, 230, 128 },
{ 1, 42, 70, 182, 114, 153, 232, 163, 255, 236, 255 },
{ 1, 17, 28, 95, 92, 137, 170, 115, 252, 208, 228 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 238, 66, 250, 245, 205, 193, 232, 180, 254, 228, 255 },
{ 178, 84, 226, 237, 192, 185, 230, 176, 253, 217, 251 },
{ 76, 83, 168, 218, 166, 173, 225, 162, 252, 220, 243 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 137, 176, 246, 252, 218, 207, 251, 208, 255, 238, 128 },
{ 176, 160, 237, 252, 217, 206, 249, 209, 255, 247, 128 },
{ 68, 128, 205, 251, 209, 207, 251, 207, 255, 248, 128 },
{ 40, 105, 167, 246, 172, 192, 252, 215, 255, 247, 128 },
{ 22, 84, 131, 214, 144, 164, 249, 185, 255, 250, 255 },
{ 11, 60, 91, 161, 130, 155, 194, 133, 253, 214, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 124, 192, 247, 253, 223, 210, 254, 215, 255, 255, 128 },
{ 103, 161, 234, 253, 218, 209, 253, 214, 255, 255, 128 },
{ 19, 108, 190, 250, 202, 202, 251, 213, 255, 241, 128 },
{ 6, 74, 131, 242, 165, 191, 251, 207, 255, 244, 128 },
{ 1, 41, 72, 198, 111, 151, 249, 185, 255, 248, 128 },
{ 1, 14, 24, 82, 90, 140, 185, 96, 254, 224, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 118, 200, 248, 254, 228, 216, 254, 222, 255, 213, 128 },
{ 91, 166, 235, 254, 220, 212, 254, 223, 255, 233, 128 },
{ 16, 110, 186, 251, 197, 201, 255, 225, 255, 255, 128 },
{ 3, 72, 124, 239, 160, 186, 253, 209, 255, 239, 128 },
{ 1, 39, 66, 198, 106, 151, 248, 191, 255, 247, 128 },
{ 1, 14, 19, 94, 74, 124, 209, 109, 255, 245, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 112, 213, 248, 255, 231, 218, 255, 234, 255, 255, 128 },
{ 80, 172, 234, 254, 220, 216, 255, 233, 255, 255, 128 },
{ 11, 112, 182, 251, 195, 204, 255, 231, 255, 224, 128 },
{ 2, 73, 126, 241, 159, 186, 254, 219, 255, 255, 128 },
{ 1, 40, 69, 207, 111, 159, 249, 191, 255, 255, 128 },
{ 1, 16, 24, 83, 78, 138, 230, 134, 255, 239, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 100, 209, 245, 255, 236, 225, 248, 231, 255, 192, 128 },
{ 65, 164, 232, 255, 226, 221, 255, 240, 255, 255, 128 },
{ 11, 117, 186, 253, 203, 209, 255, 240, 255, 255, 128 },
{ 2, 83, 136, 245, 167, 191, 253, 222, 255, 255, 128 },
{ 1, 55, 88, 213, 122, 157, 248, 182, 255, 255, 128 },
{ 1, 10, 38, 58, 85, 43, 198, 107, 255, 255, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
Use 8x8 DCT transform for I8X8 prediction mode Apply 2D-DCT transform of dimension 8x8 to encode prediction residuals of I8X8 mode. Brought back block type 3 probability context model for 8x8 tokens, which is used for the coefficients of Y blocks in I8x8 modes. The coefficient costs estimate of I8X8 mode in rate-distortion is also changed appropriately. Performance results: derf: 0.246 yt: 0.114 std-hd: 0.730 hd: 0.670 Change-Id: If1d970eeb4e1827c9f0d2c5b27d33089b347ea27
2012-08-01 19:18:25 +02:00
}
}
Entropy coding for hybrid transform Separates the entropy coding context models for 4x4, 8x8 and 16x16 ADST variants. There is a small improvement for HD (hd/std-hd) by about 0.1-0.2%. Results on derf/yt are about the same, probably because there is not enough statistics. Results may improve somewhat once the initial probability tables are updated for the hybrid transforms which is coming soon. Change-Id: Ic7c0c62dacc68ef551054fdb575be8b8507d32a8
2012-09-10 07:42:35 +02:00
};
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
static const vp9_coeff_probs default_coef_probs_16x16[BLOCK_TYPES] = {
Entropy coding for hybrid transform Separates the entropy coding context models for 4x4, 8x8 and 16x16 ADST variants. There is a small improvement for HD (hd/std-hd) by about 0.1-0.2%. Results on derf/yt are about the same, probably because there is not enough statistics. Results may improve somewhat once the initial probability tables are updated for the hybrid transforms which is coming soon. Change-Id: Ic7c0c62dacc68ef551054fdb575be8b8507d32a8
2012-09-10 07:42:35 +02:00
{ /* block Type 0 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 8, 26, 101, 170, 141, 159, 166, 138, 205, 164, 158 },
{ 2, 25, 67, 119, 124, 152, 121, 123, 189, 145, 175 },
{ 1, 15, 28, 67, 102, 139, 95, 107, 191, 136, 187 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 22, 73, 118, 160, 137, 157, 175, 132, 242, 184, 229 },
{ 43, 73, 116, 160, 137, 157, 177, 132, 242, 185, 231 },
{ 24, 66, 105, 158, 134, 156, 175, 133, 242, 185, 232 },
{ 9, 54, 85, 150, 126, 153, 175, 132, 242, 185, 231 },
{ 2, 34, 54, 123, 109, 145, 168, 124, 242, 183, 231 },
{ 1, 14, 22, 63, 93, 134, 108, 103, 214, 149, 206 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 34, 123, 149, 186, 148, 163, 195, 143, 245, 195, 233 },
{ 34, 106, 147, 189, 149, 164, 198, 146, 246, 197, 234 },
{ 10, 81, 123, 186, 143, 162, 200, 147, 246, 198, 235 },
{ 2, 56, 87, 170, 127, 156, 201, 143, 248, 202, 234 },
{ 1, 35, 56, 138, 109, 146, 187, 133, 246, 196, 233 },
{ 1, 17, 27, 80, 93, 135, 136, 109, 229, 168, 215 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 27, 159, 171, 208, 161, 171, 211, 155, 249, 205, 239 },
{ 17, 119, 162, 213, 160, 172, 218, 160, 250, 210, 238 },
{ 3, 81, 128, 207, 149, 168, 220, 161, 250, 213, 238 },
{ 1, 53, 87, 183, 128, 158, 217, 153, 251, 214, 239 },
{ 1, 31, 52, 143, 106, 145, 199, 137, 249, 205, 235 },
{ 1, 14, 24, 77, 89, 133, 142, 109, 234, 174, 215 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 24, 189, 200, 224, 177, 178, 221, 164, 250, 212, 234 },
{ 14, 136, 184, 230, 176, 181, 228, 172, 252, 215, 231 },
{ 2, 87, 140, 222, 159, 176, 230, 172, 252, 218, 238 },
{ 1, 54, 90, 193, 130, 161, 223, 160, 252, 217, 241 },
{ 1, 28, 49, 142, 103, 144, 202, 139, 250, 208, 233 },
{ 1, 12, 21, 73, 87, 132, 141, 106, 234, 176, 209 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 32, 220, 227, 242, 199, 190, 234, 180, 251, 220, 232 },
{ 12, 155, 200, 242, 190, 191, 240, 187, 252, 225, 230 },
{ 1, 90, 144, 231, 164, 180, 240, 184, 253, 229, 239 },
{ 1, 53, 90, 198, 130, 162, 230, 165, 253, 226, 238 },
{ 1, 28, 50, 145, 103, 144, 207, 140, 251, 213, 236 },
{ 1, 13, 22, 74, 88, 132, 142, 107, 233, 176, 216 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 5, 61, 234, 230, 183, 183, 212, 164, 241, 199, 205 },
{ 3, 65, 184, 199, 164, 170, 182, 145, 232, 175, 223 },
{ 1, 56, 104, 154, 137, 158, 156, 131, 221, 165, 210 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 46, 183, 210, 229, 181, 182, 222, 165, 252, 214, 251 },
{ 122, 166, 202, 228, 179, 181, 223, 164, 252, 217, 250 },
{ 49, 125, 177, 225, 172, 179, 223, 163, 252, 215, 253 },
{ 22, 99, 142, 216, 155, 173, 222, 164, 252, 215, 250 },
{ 8, 69, 95, 180, 127, 156, 220, 153, 252, 214, 250 },
{ 2, 38, 51, 112, 109, 144, 159, 118, 243, 184, 232 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 56, 196, 218, 236, 187, 185, 231, 172, 254, 223, 239 },
{ 38, 141, 195, 235, 182, 185, 233, 174, 254, 225, 232 },
{ 7, 93, 147, 225, 164, 178, 233, 173, 255, 226, 248 },
{ 2, 63, 101, 201, 137, 165, 227, 162, 254, 225, 248 },
{ 1, 39, 61, 159, 110, 148, 213, 146, 254, 218, 247 },
{ 1, 20, 33, 98, 95, 136, 166, 115, 247, 192, 231 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 44, 206, 223, 240, 193, 189, 235, 177, 255, 231, 224 },
{ 27, 147, 200, 240, 188, 189, 238, 181, 255, 229, 239 },
{ 4, 93, 147, 230, 165, 180, 238, 180, 255, 231, 237 },
{ 1, 58, 95, 201, 134, 164, 229, 164, 255, 228, 254 },
{ 1, 32, 52, 152, 105, 146, 212, 142, 254, 221, 255 },
{ 1, 14, 23, 81, 87, 133, 156, 109, 248, 191, 236 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 39, 216, 227, 244, 200, 194, 237, 179, 255, 231, 255 },
{ 22, 152, 204, 243, 192, 193, 240, 186, 255, 231, 240 },
{ 2, 92, 148, 232, 167, 183, 239, 182, 255, 232, 255 },
{ 1, 55, 91, 200, 132, 164, 229, 164, 255, 230, 255 },
{ 1, 28, 47, 144, 99, 142, 211, 141, 255, 222, 251 },
{ 1, 13, 21, 75, 86, 131, 152, 103, 249, 193, 242 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 34, 228, 234, 249, 213, 201, 246, 194, 255, 239, 255 },
{ 13, 161, 208, 247, 198, 197, 248, 197, 255, 243, 255 },
{ 1, 95, 148, 234, 166, 183, 246, 190, 255, 243, 236 },
{ 1, 55, 90, 199, 128, 161, 237, 168, 255, 239, 255 },
{ 1, 30, 51, 147, 102, 144, 218, 142, 255, 232, 254 },
{ 1, 16, 25, 86, 88, 131, 168, 109, 252, 207, 245 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
}
}, { /* block Type 1 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 204, 33, 217, 233, 185, 184, 199, 165, 204, 163, 162 },
{ 93, 48, 151, 209, 157, 171, 193, 161, 203, 167, 189 },
{ 18, 43, 86, 173, 126, 156, 203, 149, 231, 193, 200 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 43, 121, 184, 233, 173, 182, 235, 187, 248, 211, 237 },
{ 93, 117, 177, 232, 170, 180, 235, 182, 246, 204, 224 },
{ 33, 101, 158, 229, 165, 179, 235, 182, 245, 207, 236 },
{ 11, 81, 129, 221, 153, 173, 233, 179, 246, 203, 229 },
{ 2, 51, 82, 188, 124, 158, 224, 162, 248, 206, 228 },
{ 1, 18, 29, 88, 93, 137, 141, 116, 222, 161, 217 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 63, 154, 199, 239, 184, 187, 236, 187, 248, 209, 221 },
{ 53, 128, 191, 239, 182, 188, 236, 188, 251, 209, 255 },
{ 14, 99, 160, 235, 172, 184, 235, 187, 249, 207, 240 },
{ 4, 75, 122, 219, 150, 173, 226, 177, 250, 204, 240 },
{ 1, 47, 77, 176, 121, 154, 207, 153, 245, 197, 237 },
{ 1, 18, 30, 84, 95, 136, 138, 112, 229, 167, 228 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 48, 193, 210, 245, 194, 194, 241, 196, 252, 213, 255 },
{ 26, 145, 201, 245, 194, 196, 240, 195, 251, 215, 240 },
{ 6, 104, 165, 241, 179, 190, 239, 191, 253, 222, 255 },
{ 1, 73, 120, 218, 151, 174, 227, 172, 251, 219, 248 },
{ 1, 42, 69, 167, 118, 153, 205, 146, 251, 206, 245 },
{ 1, 16, 27, 84, 89, 133, 148, 112, 240, 179, 238 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 47, 213, 225, 248, 203, 199, 240, 194, 254, 211, 255 },
{ 32, 153, 212, 248, 201, 199, 241, 196, 251, 226, 255 },
{ 6, 102, 168, 240, 181, 190, 240, 187, 251, 225, 238 },
{ 1, 66, 111, 211, 146, 169, 229, 167, 255, 224, 244 },
{ 1, 36, 60, 157, 110, 148, 209, 143, 252, 215, 255 },
{ 1, 16, 27, 83, 90, 133, 152, 111, 244, 184, 250 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 46, 225, 232, 252, 219, 208, 247, 204, 254, 233, 255 },
{ 24, 162, 214, 250, 208, 204, 247, 201, 254, 236, 255 },
{ 3, 106, 165, 242, 182, 191, 245, 196, 255, 231, 255 },
{ 1, 66, 108, 213, 142, 169, 235, 175, 255, 226, 247 },
{ 1, 35, 59, 158, 108, 147, 216, 146, 254, 220, 255 },
{ 1, 16, 27, 85, 90, 131, 159, 110, 248, 191, 252 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 229, 28, 245, 227, 195, 182, 200, 145, 253, 186, 255 },
{ 151, 44, 210, 214, 180, 175, 193, 146, 247, 185, 254 },
{ 55, 48, 131, 183, 148, 163, 194, 138, 249, 201, 246 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 126, 165, 239, 250, 206, 204, 248, 193, 255, 255, 128 },
{ 199, 158, 231, 248, 206, 198, 247, 200, 243, 255, 255 },
{ 102, 136, 209, 248, 203, 197, 247, 201, 255, 244, 128 },
{ 64, 116, 181, 245, 185, 196, 248, 201, 255, 233, 128 },
{ 44, 98, 151, 233, 162, 179, 248, 195, 255, 242, 128 },
{ 44, 81, 119, 204, 140, 165, 222, 163, 252, 217, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 108, 185, 239, 252, 216, 209, 248, 205, 255, 230, 128 },
{ 91, 155, 224, 252, 211, 205, 251, 211, 255, 230, 128 },
{ 20, 116, 185, 248, 194, 196, 252, 206, 255, 255, 128 },
{ 8, 86, 141, 239, 168, 185, 248, 196, 255, 247, 128 },
{ 3, 50, 92, 206, 125, 164, 242, 176, 255, 246, 128 },
{ 1, 21, 40, 131, 85, 141, 200, 131, 247, 236, 255 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 94, 198, 243, 254, 226, 215, 254, 220, 255, 255, 128 },
{ 67, 164, 228, 253, 217, 208, 250, 216, 255, 213, 128 },
{ 14, 120, 185, 250, 196, 205, 248, 205, 255, 255, 128 },
{ 4, 83, 134, 238, 161, 181, 250, 202, 255, 233, 128 },
{ 1, 48, 82, 196, 119, 157, 248, 178, 255, 255, 128 },
{ 1, 26, 38, 96, 84, 132, 221, 110, 255, 209, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 82, 210, 245, 255, 230, 215, 246, 221, 255, 255, 128 },
{ 55, 170, 231, 254, 222, 213, 255, 220, 255, 255, 128 },
{ 8, 118, 184, 251, 200, 207, 255, 219, 255, 255, 128 },
{ 2, 78, 126, 239, 156, 185, 251, 216, 255, 255, 128 },
{ 1, 43, 68, 189, 108, 151, 247, 187, 255, 228, 128 },
{ 1, 34, 40, 121, 114, 102, 205, 96, 255, 255, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 65, 228, 241, 255, 231, 214, 253, 222, 255, 255, 128 },
{ 33, 173, 226, 254, 222, 216, 255, 231, 255, 255, 128 },
{ 5, 120, 180, 251, 197, 205, 251, 226, 255, 233, 128 },
{ 1, 81, 130, 240, 159, 187, 251, 206, 255, 205, 128 },
{ 1, 51, 78, 198, 119, 168, 238, 181, 255, 171, 128 },
{ 1, 18, 49, 183, 119, 160, 255, 171, 128, 128, 128 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
16x16 DCT blocks. Set on all 16x16 intra/inter modes Features: - Butterfly fDCT/iDCT - Loop filter does not filter internal edges with 16x16 - Optimize coefficient function - Update coefficient probability function - RD - Entropy stats - 16x16 is a config option Have not tested with experiments. hd: 2.60% std-hd: 2.43% yt: 1.32% derf: 0.60% Change-Id: I96fb090517c30c5da84bad4fae602c3ec0c58b1c
2012-08-03 02:03:14 +02:00
}
}
};
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
static const vp9_coeff_probs default_coef_probs_32x32[BLOCK_TYPES] = {
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
{ /* block Type 0 */
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 37, 34, 137, 205, 154, 170, 151, 159, 109, 172, 44 },
{ 3, 26, 60, 113, 123, 154, 100, 124, 152, 131, 144 },
{ 1, 13, 23, 54, 102, 139, 71, 106, 146, 123, 148 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 26, 77, 122, 152, 144, 160, 143, 129, 216, 158, 201 },
{ 43, 76, 123, 152, 142, 159, 145, 129, 218, 160, 204 },
{ 25, 67, 112, 150, 141, 159, 144, 128, 218, 159, 204 },
{ 9, 54, 90, 143, 134, 156, 144, 127, 218, 159, 204 },
{ 2, 32, 52, 116, 114, 148, 138, 123, 217, 158, 207 },
{ 1, 10, 15, 44, 91, 133, 75, 99, 172, 128, 169 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 32, 122, 143, 163, 145, 161, 162, 131, 226, 171, 206 },
{ 46, 105, 143, 168, 148, 161, 165, 133, 228, 174, 204 },
{ 17, 79, 116, 164, 142, 161, 166, 134, 229, 174, 206 },
{ 4, 53, 78, 143, 125, 153, 163, 129, 232, 175, 213 },
{ 1, 29, 44, 105, 105, 142, 147, 120, 228, 168, 211 },
{ 1, 12, 18, 52, 91, 133, 92, 100, 193, 140, 183 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 33, 157, 160, 182, 149, 163, 185, 141, 236, 185, 218 },
{ 20, 116, 152, 188, 152, 165, 191, 144, 238, 188, 217 },
{ 4, 74, 114, 180, 141, 162, 192, 143, 240, 191, 219 },
{ 1, 44, 69, 148, 119, 151, 183, 134, 243, 192, 227 },
{ 1, 25, 40, 110, 101, 141, 162, 121, 238, 181, 223 },
{ 1, 12, 18, 56, 89, 132, 103, 101, 206, 148, 196 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 25, 183, 174, 207, 159, 171, 205, 156, 243, 194, 228 },
{ 13, 124, 159, 209, 157, 171, 213, 160, 243, 200, 228 },
{ 2, 75, 117, 199, 143, 166, 215, 158, 246, 205, 230 },
{ 1, 45, 73, 165, 119, 153, 204, 144, 248, 205, 231 },
{ 1, 26, 43, 120, 101, 141, 178, 127, 242, 192, 226 },
{ 1, 12, 19, 59, 89, 132, 112, 102, 215, 154, 201 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 13, 232, 223, 239, 196, 188, 225, 172, 248, 209, 226 },
{ 4, 155, 187, 237, 184, 187, 233, 180, 250, 216, 232 },
{ 1, 86, 131, 222, 156, 175, 233, 176, 251, 218, 237 },
{ 1, 49, 79, 181, 123, 157, 218, 155, 251, 214, 237 },
{ 1, 26, 43, 125, 100, 141, 188, 130, 246, 199, 231 },
{ 1, 12, 20, 62, 88, 131, 119, 102, 222, 161, 209 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 51, 37, 227, 237, 205, 184, 200, 162, 231, 187, 207 },
{ 9, 36, 172, 204, 176, 173, 171, 145, 217, 167, 197 },
{ 21, 26, 112, 162, 145, 162, 155, 133, 215, 165, 191 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 79, 169, 219, 223, 176, 177, 222, 161, 248, 213, 244 },
{ 177, 166, 216, 222, 175, 178, 222, 161, 246, 212, 226 },
{ 119, 141, 196, 222, 174, 176, 220, 163, 250, 212, 236 },
{ 63, 117, 165, 217, 163, 175, 218, 161, 248, 209, 231 },
{ 30, 87, 117, 192, 138, 162, 216, 157, 247, 211, 224 },
{ 14, 56, 60, 119, 111, 146, 156, 123, 227, 171, 220 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 88, 195, 225, 229, 181, 181, 229, 171, 252, 212, 221 },
{ 66, 145, 202, 229, 177, 180, 230, 172, 253, 220, 255 },
{ 12, 97, 152, 221, 162, 174, 230, 169, 253, 218, 249 },
{ 3, 66, 103, 198, 138, 165, 223, 159, 253, 219, 251 },
{ 1, 38, 61, 158, 110, 148, 209, 146, 252, 212, 238 },
{ 1, 19, 30, 94, 94, 136, 160, 114, 244, 185, 236 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 79, 211, 228, 235, 186, 184, 233, 176, 255, 225, 255 },
{ 50, 151, 205, 235, 182, 185, 237, 177, 254, 223, 255 },
{ 7, 95, 149, 225, 162, 176, 236, 177, 254, 229, 219 },
{ 1, 62, 98, 198, 134, 164, 228, 162, 254, 224, 238 },
{ 1, 35, 57, 156, 108, 148, 211, 143, 253, 215, 238 },
{ 1, 17, 26, 87, 89, 135, 161, 113, 246, 189, 237 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 68, 225, 230, 239, 190, 187, 238, 180, 252, 234, 255 },
{ 39, 156, 206, 239, 185, 187, 241, 187, 254, 231, 255 },
{ 4, 94, 147, 229, 163, 178, 242, 183, 255, 236, 224 },
{ 1, 58, 94, 200, 132, 163, 232, 166, 254, 230, 255 },
{ 1, 32, 52, 153, 104, 146, 214, 144, 253, 222, 236 },
{ 1, 15, 24, 84, 89, 131, 159, 109, 247, 192, 240 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 45, 248, 234, 248, 208, 198, 244, 193, 255, 233, 255 },
{ 19, 169, 204, 246, 195, 195, 246, 199, 255, 233, 255 },
{ 2, 98, 145, 235, 166, 183, 245, 192, 255, 235, 255 },
{ 1, 59, 92, 205, 131, 164, 236, 172, 254, 231, 250 },
{ 1, 33, 52, 152, 103, 145, 216, 144, 253, 221, 240 },
{ 1, 15, 24, 83, 87, 133, 156, 110, 246, 191, 242 }
Split coefficient token tables intra vs. inter. Change-Id: I5416455f8f129ca0f450d00e48358d2012605072
2013-02-19 22:36:38 +01:00
}
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
}
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* block Type 1 */
{ /* Intra */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 179, 23, 200, 222, 180, 182, 150, 152, 148, 135, 125 },
{ 60, 33, 113, 185, 143, 166, 168, 144, 189, 168, 152 },
{ 8, 31, 59, 137, 114, 150, 163, 132, 206, 171, 169 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 27, 103, 158, 215, 157, 174, 209, 165, 239, 191, 233 },
{ 90, 101, 159, 213, 156, 173, 212, 164, 230, 185, 237 },
{ 39, 91, 146, 212, 155, 169, 212, 165, 232, 186, 207 },
{ 16, 75, 120, 203, 144, 169, 210, 161, 233, 189, 227 },
{ 3, 48, 76, 167, 120, 154, 199, 146, 236, 190, 218 },
{ 1, 18, 26, 72, 95, 137, 113, 109, 197, 146, 186 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 45, 137, 177, 218, 166, 174, 206, 163, 234, 184, 214 },
{ 47, 117, 167, 218, 166, 176, 206, 164, 234, 182, 229 },
{ 16, 90, 136, 211, 153, 172, 205, 162, 236, 192, 231 },
{ 6, 65, 100, 188, 136, 162, 193, 155, 237, 177, 228 },
{ 1, 37, 58, 137, 113, 150, 166, 134, 229, 167, 234 },
{ 1, 13, 19, 55, 90, 132, 93, 103, 196, 137, 202 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 36, 171, 194, 227, 177, 179, 208, 165, 244, 196, 245 },
{ 19, 129, 178, 227, 175, 184, 214, 165, 246, 188, 255 },
{ 5, 90, 139, 217, 158, 174, 213, 166, 246, 198, 255 },
{ 1, 59, 93, 182, 134, 162, 193, 150, 242, 188, 241 },
{ 1, 31, 49, 122, 108, 145, 160, 127, 235, 172, 229 },
{ 1, 10, 18, 54, 89, 132, 101, 99, 213, 144, 217 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 37, 197, 210, 233, 187, 186, 216, 172, 250, 202, 255 },
{ 20, 142, 191, 234, 183, 186, 219, 170, 249, 207, 246 },
{ 3, 93, 144, 222, 163, 176, 219, 170, 249, 204, 224 },
{ 1, 56, 88, 179, 130, 159, 199, 148, 246, 197, 243 },
{ 1, 29, 47, 123, 104, 144, 172, 127, 244, 185, 234 },
{ 1, 14, 22, 66, 91, 130, 120, 103, 225, 158, 221 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 19, 227, 223, 245, 203, 194, 238, 187, 251, 225, 217 },
{ 6, 152, 192, 242, 189, 190, 241, 190, 253, 225, 255 },
{ 1, 89, 138, 228, 161, 177, 239, 181, 254, 224, 248 },
{ 1, 52, 84, 188, 127, 157, 224, 159, 253, 222, 247 },
{ 1, 29, 47, 132, 102, 140, 196, 132, 251, 208, 244 },
{ 1, 14, 23, 71, 90, 133, 134, 103, 239, 174, 233 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}
}, { /* Inter */
{ /* Coeff Band 0 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 205, 14, 245, 235, 216, 189, 190, 146, 249, 201, 255 },
{ 97, 19, 213, 210, 194, 174, 176, 139, 241, 183, 250 },
{ 31, 20, 144, 183, 160, 167, 171, 132, 240, 184, 253 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 1 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 137, 182, 245, 254, 221, 216, 255, 160, 128, 128, 128 },
{ 231, 185, 242, 251, 218, 205, 255, 233, 128, 128, 128 },
{ 170, 175, 229, 252, 205, 209, 255, 211, 128, 128, 128 },
{ 107, 157, 213, 250, 199, 205, 251, 207, 255, 255, 128 },
{ 77, 126, 183, 243, 182, 183, 252, 206, 255, 255, 128 },
{ 69, 96, 149, 229, 157, 170, 247, 169, 255, 255, 128 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 2 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 107, 196, 241, 252, 211, 208, 255, 210, 128, 128, 128 },
{ 92, 162, 221, 249, 203, 195, 255, 199, 128, 128, 128 },
{ 20, 108, 181, 244, 190, 191, 250, 200, 255, 255, 128 },
{ 7, 80, 132, 241, 172, 197, 253, 191, 255, 255, 128 },
{ 2, 43, 75, 219, 122, 150, 255, 203, 128, 128, 128 },
{ 1, 15, 48, 98, 51, 192, 255, 160, 128, 128, 128 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 3 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 107, 202, 244, 254, 226, 215, 255, 192, 128, 128, 128 },
{ 77, 167, 224, 252, 215, 212, 255, 235, 128, 128, 128 },
{ 14, 117, 179, 249, 191, 196, 255, 212, 128, 128, 128 },
{ 3, 84, 134, 237, 160, 194, 248, 216, 255, 255, 128 },
{ 1, 57, 84, 216, 145, 136, 255, 161, 128, 128, 128 },
{ 1, 1, 1, 255, 128, 255, 128, 128, 128, 128, 128 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 4 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 88, 219, 248, 255, 239, 225, 255, 255, 128, 128, 128 },
{ 61, 178, 234, 255, 227, 227, 255, 217, 128, 128, 128 },
{ 6, 127, 188, 252, 201, 211, 255, 244, 128, 128, 128 },
{ 1, 83, 130, 248, 173, 197, 255, 175, 128, 128, 128 },
{ 1, 61, 66, 211, 121, 188, 255, 213, 128, 128, 128 },
{ 128, 128, 128, 128, 128, 128, 128, 128, 128, 128, 128 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}, { /* Coeff Band 5 */
Redo banding for all transforms. Now that the first AC coefficient in both directions use the same DC as their context, there no longer is a purpose in letting both have their own band. Merging these two bands allows us to split bands for some of the very high-frequency AC bands. In addition, I'm redoing the banding for the 1D-ADST col/row scans. I don't think the old banding made any sense at all (it merged the last coefficient of the first row/col in the same band as the first two of the second row/col), which was clearly an oversight from the band being applied in scan-order (rather than in their actual position). Now, coefficients at the same position will be in the same band, regardless what scan order is used. I think this makes most sense for the purpose of banding, which is basically "predict energy for this coefficient depending on the energy of context coefficients" (i.e. pt). After full re-training, together with previous patch, derf gains about 1.2-1.3%, and hd/stdhd gain about 0.9-1.0%. Change-Id: I7a0cc12ba724e88b278034113cb4adaaebf87e0c
2013-03-25 20:28:24 +01:00
{ 73, 243, 250, 255, 244, 220, 255, 205, 128, 128, 128 },
{ 42, 197, 242, 255, 237, 227, 242, 166, 255, 255, 128 },
{ 10, 137, 197, 252, 214, 199, 255, 238, 128, 128, 128 },
{ 2, 85, 134, 242, 163, 185, 224, 238, 255, 255, 128 },
{ 1, 70, 69, 199, 110, 64, 255, 213, 128, 128, 128 },
{ 1, 1, 1, 1, 128, 128, 255, 1, 128, 128, 128 }
Make superblocks independent of macroblock code and data. Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-03-04 23:12:17 +01:00
}
}
New default coefficient/band probabilities. Gives 0.5-0.6% improvement on derf and stdhd, and 1.1% on hd. The old tables basically derive from times that we had only 4x4 or only 4x4 and 8x8 DCTs. Note that some values are filled with 128, because e.g. ADST ever only occurs as Y-with-DC, as does 32x32; 16x16 ever only occurs as Y-with-DC or as UV (as complement of 32x32 Y); and 8x8 Y2 ever only has 4 coefficients max. If preferred, I can add values of other tables in their place (e.g. use 4x4 2nd order high-frequency probabilities for 8x8 2nd order), so that they make at least some sense if we ever implement a larger 2nd order transform for the 8x8 DCT (etc.), please let me know Change-Id: I917db356f2aff8865f528eb873c56ef43aa5ce22
2012-12-12 19:25:58 +01:00
}
};
Coding con-zero count rather than EOB for coeffs This patch revamps the entropy coding of coefficients to code first a non-zero count per coded block and correspondingly remove the EOB token from the token set. STATUS: Main encode/decode code achieving encode/decode sync - done. Forward and backward probability updates to the nzcs - done. Rd costing updates for nzcs - done. Note: The dynamic progrmaming apporach used in trellis quantization is not exactly compatible with nzcs. A suboptimal approach has been used instead where branch costs are updated to account for changes in the nzcs. TODO: Training the default probs/counts for nzcs Change-Id: I951bc1e22f47885077a7453a09b0493daa77883d
2013-02-20 19:16:24 +01:00
End of orientation zero group experiment Adds an experiment that codes an end-of-orientation symbol for every eligible zero encountered in scan order. This cleans out various other sub-experiments that were part of the origiinal patch, which will be later included if found useful. Results are slightly positive on all sets (0.1 - 0.2% range). Change-Id: I57765c605fefc7fb9d1b57f1b356843602abefaf
2013-03-28 18:42:23 +01:00
#if CONFIG_CODE_ZEROGROUP
// There are two probs: the first is the prob(0) of the isolated zero bit,
// the second is the prob(0) of the end of orientation symbol [if 0 that
// indicates a zerotree root].
static const vp9_zpc_probs default_zpc_probs_4x4 = {
{ /* Intra */
{ /* Coeff Band 0 */
{ 1, }, { 1, }, { 1, },
}, { /* Coeff Band 1 */
{ 1, }, { 1, }, { 1, },
}, { /* Coeff Band 2 */
{ 1, }, { 1, }, { 1, },
}
}, { /* Inter */
{ /* Coeff Band 0 */
{ 1, }, { 1, }, { 1, },
}, { /* Coeff Band 1 */
{ 1, }, { 1, }, { 1, },
}, { /* Coeff Band 2 */
{ 1, }, { 1, }, { 1, },
}
}
};
static const vp9_zpc_probs default_zpc_probs_8x8 = {
{ /* Intra */
{ /* ZPC Band 0 */
{ 4, }, { 2, }, { 1, },
}, { /* ZPC Band 1 */
{ 4, }, { 2, }, { 1, },
}, { /* ZPC Band 2 */
{ 4, }, { 2, }, { 1, },
}
}, { /* Inter */
{ /* ZPC Band 0 */
{ 4, }, { 2, }, { 1, },
}, { /* ZPC Band 1 */
{ 4, }, { 2, }, { 1, },
}, { /* ZPC Band 2 */
{ 4, }, { 2, }, { 1, },
}
}
};
static const vp9_zpc_probs default_zpc_probs_16x16 = {
{ /* Intra */
{ /* ZPC Band 0 */
{ 57, }, { 30, }, { 13, },
}, { /* ZPC Band 1 */
{ 46, }, { 23, }, { 4, },
}, { /* ZPC Band 1 */
{ 36, }, { 11, }, { 2, },
},
}, { /* Inter */
{ /* ZPC Band 0 */
{ 45, }, { 21 }, { 10, },
}, { /* ZPC Band 1 */
{ 24, }, { 14, }, { 3, },
}, { /* ZPC Band 2 */
{ 16, }, { 6, }, { 1, },
},
},
};
static const vp9_zpc_probs default_zpc_probs_32x32 = {
{ /* Intra */
{ /* ZPC Band 0 */
{ 132, }, { 60, }, { 19, },
}, { /* ZPC Band 1 */
{ 64, }, { 32, }, { 8, },
}, { /* ZPC Band 2 */
{ 25, }, { 11, }, { 1, },
},
}, { /* Inter */
{ /* ZPC Band 0 */
{ 134, }, { 39, }, { 25, },
}, { /* ZPC Band 1 */
{ 64, }, { 24, }, { 12, },
}, { /* ZPC Band 2 */
{ 21, }, { 10, }, { 1, },
},
},
};
#endif // CONFIG_CODE_ZEROGROUP
Reference in New Issue Copy Permalink