generic-library/webrtc
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
02e749f848
webrtc/tools/perf_expectations/perf_expectations.json

42 lines
7.3 KiB
JSON
Raw Normal View History

Adjusted net_50_5_plr_5 on Linux, removed all gilbert_elliot metrics (too flaky), added mac expectations. BUG= Review URL: https://webrtc-codereview.appspot.com/1075006 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3453 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-02-01 14:28:58 +01:00
{"linux-large-tests/audio_e2e_test/audio_e2e_score/e2e_score": {"reva": 3380, "revb": 3451, "type": "absolute", "better": "higher", "improve": 4.6977, "regress": 3.77055, "sha1": "b76f22a2"},
Add regression monitoring for audioproc and iSAC fixed-point tests. BUG=none TEST=none Review URL: https://webrtc-codereview.appspot.com/1094011 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3513 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-02-14 13:27:17 +01:00
"linux-large-tests/audioproc_perf/audioproc/time_per_10ms_frame": {"reva": 3435, "revb": 3510, "type": "absolute", "improve": 64, "regress": 75, "sha1": "b9b3d6a3"},
"linux-large-tests/isac_fixed_perf/isac/time_per_10ms_frame": {"reva": 3435, "revb": 3510, "type": "absolute", "improve": 102, "regress": 120, "sha1": "0922ca06"},
Fixed stale regression values and calibrated some vie_auto_test values. BUG= Review URL: https://webrtc-codereview.appspot.com/1066010 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3441 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-01-31 11:06:44 +01:00
"linux-large-tests/vie_auto_test/psnr/net_delay_0_0_plr_0": {"reva": 3361, "revb": 3389, "type": "absolute", "better": "higher", "improve": 36.75, "regress": 34.47265, "sha1": "75dc797e"},
"linux-large-tests/vie_auto_test/psnr/net_delay_100_10_plr_0": {"reva": 3361, "revb": 3389, "type": "absolute", "better": "higher", "improve": 36.75, "regress": 34.4907, "sha1": "82050834"},
"linux-large-tests/vie_auto_test/psnr/net_delay_50_5_plr_5": {"reva": 3361, "revb": 3389, "type": "absolute", "better": "higher", "improve": 36.75, "regress": 34.4128, "sha1": "93ca8ea5"},
"linux-large-tests/vie_auto_test/ssim/net_delay_0_0_plr_0": {"reva": 3381, "revb": 3389, "type": "absolute", "better": "higher", "improve": 1.0185, "regress": 0.9234, "sha1": "57fdf26c"},
"linux-large-tests/vie_auto_test/ssim/net_delay_100_10_plr_0": {"reva": 3381, "revb": 3389, "type": "absolute", "better": "higher", "improve": 1.0185, "regress": 0.9234, "sha1": "b0a62939"},
"linux-large-tests/vie_auto_test/ssim/net_delay_50_5_plr_5": {"reva": 3381, "revb": 3389, "type": "absolute", "better": "higher", "improve": 1.01745, "regress": 0.9234, "sha1": "580aea7a"},
"linux-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_0_0_plr_0": {"reva": 3337, "revb": 3389, "type": "absolute", "improve": 31.29034, "regress": 34.966995, "sha1": "250facd1"},
"linux-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_100_10_plr_0": {"reva": 3337, "revb": 3389, "type": "absolute", "improve": 31.15791, "regress": 34.936545, "sha1": "9ff072ce"},
"linux-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_50_5_plr_5": {"reva": 3337, "revb": 3389, "type": "absolute", "improve": 31.23353, "regress": 36.812685, "sha1": "8f5af7a5"},
Fixed more perf expectations. For Linux, the expectations just look a bit too tightly wound. On Windows there's a long-term increasing trend that we may want to have someone look at. http://www.corp.google.com/~webrtc-cb/perf//linux-large-tests/vie_auto_test/report.html?history=1500&rev=-1&graph=total_delay_incl_network http://www.corp.google.com/~webrtc-cb/perf//linux-large-tests/vie_auto_test/report.html?history=1500&rev=-1&graph=total_delay_incl_network BUG= R=kjellander@webrtc.org Review URL: https://webrtc-codereview.appspot.com/1472005 git-svn-id: http://webrtc.googlecode.com/svn/trunk@4025 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-05-14 13:26:14 +02:00
"linux-large-tests/vie_auto_test/total_delay_incl_network/net_delay_0_0_plr_0": {"reva": 3415, "revb": 4020, "type": "absolute", "improve": 24.329215, "regress": 36.22248, "sha1": "9698dfda"},
"linux-large-tests/vie_auto_test/total_delay_incl_network/net_delay_100_10_plr_0": {"reva": 3982, "revb": 4020, "type": "absolute", "improve": 124.5393, "regress": 212.1357, "sha1": "b182b10c"},
Adjusted net_50_5_plr_5 on Linux, removed all gilbert_elliot metrics (too flaky), added mac expectations. BUG= Review URL: https://webrtc-codereview.appspot.com/1075006 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3453 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-02-01 14:28:58 +01:00
"linux-large-tests/vie_auto_test/total_delay_incl_network/net_delay_50_5_plr_5": {"reva": 3337, "revb": 3451, "type": "absolute", "improve": 79.99912, "regress": 186.35715, "sha1": "377cb407"},
"mac-large-tests/vie_auto_test/psnr/net_delay_0_0_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "better": "higher", "improve": 38.2137, "regress": 34.0841, "sha1": "7f1bd1b0"},
"mac-large-tests/vie_auto_test/psnr/net_delay_100_10_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "better": "higher", "improve": 38.1759, "regress": 34.048, "sha1": "fe4facc2"},
"mac-large-tests/vie_auto_test/psnr/net_delay_50_5_plr_5": {"reva": 3394, "revb": 3452, "type": "absolute", "better": "higher", "improve": 38.1528, "regress": 34.06035, "sha1": "59b4f938"},
"mac-large-tests/vie_auto_test/ssim/net_delay_0_0_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "better": "higher", "improve": 1.0206, "regress": 0.92055, "sha1": "cf3861b8"},
"mac-large-tests/vie_auto_test/ssim/net_delay_100_10_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "better": "higher", "improve": 1.0206, "regress": 0.92055, "sha1": "5cfe0c92"},
"mac-large-tests/vie_auto_test/ssim/net_delay_50_5_plr_5": {"reva": 3394, "revb": 3452, "type": "absolute", "better": "higher", "improve": 1.0206, "regress": 0.9215, "sha1": "21e75d0c"},
"mac-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_0_0_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "improve": 31.933775, "regress": 38.39535, "sha1": "64644df0"},
"mac-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_100_10_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "improve": 31.9124, "regress": 35.920605, "sha1": "db0345c3"},
"mac-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_50_5_plr_5": {"reva": 3394, "revb": 3452, "type": "absolute", "improve": 31.972535, "regress": 36.81552, "sha1": "04b8f506"},
"mac-large-tests/vie_auto_test/total_delay_incl_network/net_delay_0_0_plr_0": {"reva": 3394, "revb": 3452, "type": "absolute", "improve": 26.810235, "regress": 35.407785, "sha1": "61f0eb7d"},
Adjusted perf expectations for mac large tests. BUG= R=kjellander@webrtc.org Review URL: https://webrtc-codereview.appspot.com/1472004 git-svn-id: http://webrtc.googlecode.com/svn/trunk@4024 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-05-14 12:51:13 +02:00
"mac-large-tests/vie_auto_test/total_delay_incl_network/net_delay_100_10_plr_0": {"reva": 3980, "revb": 4018, "type": "absolute", "improve": 131.59305, "regress": 205.2288, "sha1": "81299208"},
Adjusted net_50_5_plr_5 on Linux, removed all gilbert_elliot metrics (too flaky), added mac expectations. BUG= Review URL: https://webrtc-codereview.appspot.com/1075006 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3453 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-02-01 14:28:58 +01:00
"mac-large-tests/vie_auto_test/total_delay_incl_network/net_delay_50_5_plr_5": {"reva": 3394, "revb": 3452, "type": "absolute", "improve": 85.895485, "regress": 196.05705, "sha1": "8ac10284"},
"win-large-tests/vie_auto_test/psnr/net_delay_0_0_plr_0": {"reva": 3361, "revb": 3451, "type": "absolute", "better": "higher", "improve": 38.2431, "regress": 32.3, "sha1": "2b853279"},
Fixed stale regression values and calibrated some vie_auto_test values. BUG= Review URL: https://webrtc-codereview.appspot.com/1066010 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3441 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-01-31 11:06:44 +01:00
"win-large-tests/vie_auto_test/psnr/net_delay_100_10_plr_0": {"reva": 3361, "revb": 3389, "type": "absolute", "better": "higher", "improve": 37.8, "regress": 34.44225, "sha1": "d81a31e6"},
"win-large-tests/vie_auto_test/psnr/net_delay_50_5_plr_5": {"reva": 3361, "revb": 3389, "type": "absolute", "better": "higher", "improve": 36.75, "regress": 34.17625, "sha1": "b65327da"},
"win-large-tests/vie_auto_test/ssim/net_delay_0_0_plr_0": {"reva": 3381, "revb": 3389, "type": "absolute", "better": "higher", "improve": 0.9933, "regress": 0.92435, "sha1": "e6dbdfb6"},
"win-large-tests/vie_auto_test/ssim/net_delay_100_10_plr_0": {"reva": 3381, "revb": 3389, "type": "absolute", "better": "higher", "improve": 1.01955, "regress": 0.9234, "sha1": "e5b7a3d8"},
"win-large-tests/vie_auto_test/ssim/net_delay_50_5_plr_5": {"reva": 3381, "revb": 3389, "type": "absolute", "better": "higher", "improve": 1.01745, "regress": 0.9215, "sha1": "318e79ed"},
Updating Perf numbers for Win Large Test. Due to a bug in the RTP module, which appeared during packet loss, we have had too short delay in the Win Large Test. When the bug was fixed we had a regression error that should be fixed with this update. Review URL: https://webrtc-codereview.appspot.com/1091005 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3466 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-02-04 14:25:11 +01:00
"win-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_0_0_plr_0": {"reva": 3455, "revb": 3463, "type": "absolute", "improve": 31.142425, "regress": 53.839275, "sha1": "b5dcfcf6"},
Fixed stale regression values and calibrated some vie_auto_test values. BUG= Review URL: https://webrtc-codereview.appspot.com/1066010 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3441 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-01-31 11:06:44 +01:00
"win-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_100_10_plr_0": {"reva": 3337, "revb": 3389, "type": "absolute", "improve": 31.1125, "regress": 35.68572, "sha1": "d21f281f"},
"win-large-tests/vie_auto_test/time_between_rendered_frames/net_delay_50_5_plr_5": {"reva": 3337, "revb": 3389, "type": "absolute", "improve": 31.616, "regress": 36.275295, "sha1": "51c950b1"},
Fixed more perf expectations. For Linux, the expectations just look a bit too tightly wound. On Windows there's a long-term increasing trend that we may want to have someone look at. http://www.corp.google.com/~webrtc-cb/perf//linux-large-tests/vie_auto_test/report.html?history=1500&rev=-1&graph=total_delay_incl_network http://www.corp.google.com/~webrtc-cb/perf//linux-large-tests/vie_auto_test/report.html?history=1500&rev=-1&graph=total_delay_incl_network BUG= R=kjellander@webrtc.org Review URL: https://webrtc-codereview.appspot.com/1472005 git-svn-id: http://webrtc.googlecode.com/svn/trunk@4025 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-05-14 13:26:14 +02:00
"win-large-tests/vie_auto_test/total_delay_incl_network/net_delay_0_0_plr_0": {"reva": 3935, "revb": 4022, "type": "absolute", "improve": 28.04761, "regress": 119.67795, "sha1": "7f4d427c"},
Adjusted net delay perf expectations slightly. This should fix the flakiness we see on the Windows vie_auto_test bot. TBR=kjellander@webrtc.org BUG=1871 Review URL: https://webrtc-codereview.appspot.com/1845004 git-svn-id: http://webrtc.googlecode.com/svn/trunk@4378 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-07-22 15:15:59 +02:00
"win-large-tests/vie_auto_test/total_delay_incl_network/net_delay_100_10_plr_0": {"reva": 4343, "revb": 4377, "type": "absolute", "improve": 124.58205, "regress": 149.44755, "sha1": "620d25b7"},
Updating perf R=niklas.enbom@webrtc.org Review URL: https://webrtc-codereview.appspot.com/1447004 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3995 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-05-09 19:42:58 +02:00
"win-large-tests/vie_auto_test/total_delay_incl_network/net_delay_50_5_plr_5": {"reva": 3974, "revb": 3983, "type": "absolute", "improve": 153.1894, "regress": 205.27185, "sha1": "9e52c01a"},
Added perf expectations and corrected existing tests to remove spaces from series names. The make_expectations.py script is copied from Chrome. It can look at a test's result history and automagically compute good regress/improve values. This patch comes into play together with https://chromereviews.googleplex.com/6242014/. NOTE TO SELF: save history on master when deploying! Review URL: https://webrtc-codereview.appspot.com/1019005 git-svn-id: http://webrtc.googlecode.com/svn/trunk@3339 4adac7df-926f-26a2-2b94-8c16560cd09d
2013-01-07 13:05:53 +01:00
"load": true
}
Reference in New Issue Copy Permalink