This CL introduces a new directory for WebRTC utility scripts for Valgrind: tools/valgrind-webrtc

To be able to re-use as much as possible from Chromium's scripts, I've created two customized scripts for running tests:
- webrtc_tests.py: a customized version of chrome_tests.py with WebRTC tests instead.
- webrtc_tests.sh: a customized version of the chrome_tests.sh wrapper script, to launch the above script.

The webrtc_tests.sh script is setting up PYTHONPATH so that tools/valgrind is available for the webrtc_tests.py script.
The webrtc_tests.py script inherits the chrome_tests.py script as much as possible, to minimize maintenance and maximize readability.
Having this mirrored setup of directories, allows us to use the same directory hierarchy for suppression files too. This CL only adds suppression files for memcheck, but we can add files for tsan later easily.

The suppression file is currently empty for Linux. For Mac I copied all the Chromium third-party suppressions. We will need a lot more added for Mac before Valgrind runs becomes usable for that platform.
The platform specific naming of the suppression files are handled automatically when webrtc_tests.sh  is used to launch the tests.

Example:
Plain memcheck (default tool): tools/valgrind-webrtc/webrtc_tests.sh -t system_wrappers
Run ThreadSanitizer: tools/valgrind-webrtc/webrtc_tests.sh  --tool=tsan -t system_wrappers

Previously mentioned AddressSanitizer requires additional scripts to be added and it not covered in this CL.

BUG=
TEST=

Review URL: http://webrtc-codereview.appspot.com/322010

git-svn-id: http://webrtc.googlecode.com/svn/trunk@1240 4adac7df-926f-26a2-2b94-8c16560cd09d
This commit is contained in:
kjellander@webrtc.org 2011-12-20 08:53:41 +00:00
parent 9e4f3830a7
commit b5b155b8a9
5 changed files with 612 additions and 0 deletions

1
.gitignore vendored
View File

@ -36,6 +36,7 @@
/resources
/src/supplement.gypi
/testing
/third_party/asan
/third_party/cygwin
/third_party/expat
/third_party/google-gflags/src

View File

@ -0,0 +1,30 @@
# There are four kinds of suppressions in this file.
# 1. third party stuff we have no control over
#
# 2. intentional unit test errors, or stuff that is somehow a false positive
# in our own code, or stuff that is so trivial it's not worth fixing
#
# 3. Suppressions for real WebRTC bugs that are not yet fixed.
# These should all be in WebRTC's bug tracking system.
# Periodically we should sweep this file and the bug tracker clean by
# running overnight and removing outdated bugs/suppressions.
#
# 4. issues that happen only on Google workstations.
#-----------------------------------------------------------------------
# 1. third party stuff we have no control over
#-----------------------------------------------------------------------
# 2. intentional unit test errors, or stuff that is somehow a false positive
# in our own code, or stuff that is so trivial it's not worth fixing
#-----------------------------------------------------------------------
# 3. Suppressions for real chromium bugs that are not yet fixed.
# These should all be in WebRTC's bug tracking system.
#-----------------------------------------------------------------------
# 4. These only occur on our Google workstations

View File

@ -0,0 +1,323 @@
# There are three kinds of suppressions in this file:
# 1. Third party stuff we have no control over.
#
# 2. Intentional unit test errors, stuff that is somehow a false positive
# in our own code, or stuff that is so trivial it's not worth fixing.
#
# 3. Suppressions for real WebRTC bugs that are not yet fixed.
# These should all be in WebRTC's bug tracking system.
# Periodically we should sweep this file and the bug tracker clean by
# running overnight and removing outdated bugs/suppressions.
#-----------------------------------------------------------------------
# 1. Third party stuff we have no control over.
{
FIXME mac kevent libevent probably needs valgrind hooks
Memcheck:Param
kevent(changelist)
fun:kevent
fun:event_base_new
}
{
# CoreAudio leak. See http://crbug.com/9351
bug_9351
Memcheck:Leak
...
fun:_ZN12HALCADClient19AddPropertyListenerEmPK26AudioObjectPropertyAddressPFlmmS2_PvES3_
fun:_ZN16HALDefaultDevice22InstallServerListenersEv
fun:_ZN16HALDefaultDevice10InitializeEv
fun:_ZN9HALSystem16CheckOutInstanceEv
}
{
# Mac test_shell_tests. See http://crbug.com/11134
# Doesn't happen on bots, but happens like crazy on the smo
# test machine 'caliban'. Don't delete just because it
# doesn't happen on the bots.
bug_11134
Memcheck:Uninitialized
fun:vCMMVectorConvert8BitRGBToRGB
fun:_ZNK15CMMConvRGBToRGB7ConvertER8CMM8BitsP14CMMRuntimeInfomm
}
{
# Mac system library bug? See http://crbug.com/11327
bug_11327
Memcheck:Uninitialized
fun:_ZN19AudioConverterChain5ResetEv
fun:AudioConverterReset
obj:/System/Library/Components/CoreAudio.component/Contents/MacOS/CoreAudio
}
{
# Mac system library bug? See http://crbug.com/11327
bug_11327b
Memcheck:Uninitialized
fun:AUNetSendEntry
fun:AUNetSendEntry
obj:/System/Library/Components/CoreAudio.component/Contents/MacOS/CoreAudio
}
{
# Filed with Apple as rdar://6915060; see http://crbug.com/11270
bug_11270
Memcheck:Leak
fun:calloc
fun:CMSSetLabCLUT
}
{
# Mac leak in CMOpenOrNewAccess in unit_tests PlatformCanvas_SkLayer_Test,
# ToolbarControllerTest_FocusLocation_Test. See http://crbug.com/11333.
bug_11333
Memcheck:Leak
fun:malloc
fun:stdSmartNewPtr
fun:stdSmartNewHandle
fun:IOCreateAndOpen
fun:ScratchInit
fun:CMOpenOrNewAccess
}
{
# suddenly very common as of 6 aug 2009
bug_11333b
Memcheck:Leak
fun:malloc
fun:stdSmartNewPtr
fun:stdSmartNewHandle
fun:IOCreateAndOpen
fun:ScratchInit
fun:CMNewAccessFromAnother
}
{
# Tiny one-time leak, widely seen by valgind users; everyone suppresses this.
# See related discussion at http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39366
plugin_bundle_global_leak
Memcheck:Leak
fun:malloc
fun:__cxa_get_globals
fun:__cxa_allocate_exception
fun:_ZN4dyld4loadEPKcRKNS_11LoadContextE
fun:dlopen
fun:dlopen
fun:_CFBundleDlfcnCheckLoaded
}
{
bug_18215
Memcheck:Uninitialized
fun:_DPSNextEvent
fun:-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]
fun:-[NSApplication run]
}
{
bug_18223
Memcheck:Uninitialized
fun:_ZNK8Security12UnixPlusPlus17StaticForkMonitorclEv
fun:_ZN12ocspdGlobals10serverPortEv
}
{
# Filed with Apple as rdar://7255382
bug_20459a
Memcheck:Leak
...
fun:_CFRuntimeCreateInstance
fun:CFRunLoopSourceCreate
fun:CFMachPortCreateRunLoopSource
fun:_ZN8Security12MachPlusPlus10CFAutoPort6enableEv
fun:_ZN8Security14SecurityServer14ThreadNotifierC2Ev
}
{
# Also filed with Apple as rdar://7255382
bug_20459b
Memcheck:Leak
fun:malloc_zone_malloc
fun:_CFRuntimeCreateInstance
fun:__CFArrayInit
fun:CFArrayCreateMutableCopy
fun:_ZN8Security12KeychainCore5Trust8evaluateEv
}
# See description of bug_20653a/b in suppressions.txt.
{
bug_20653a_mac
Memcheck:Param
write(buf)
fun:write$UNIX2003
fun:pager_write_pagelist
}
{
bug_20653b_mac
Memcheck:Param
write(buf)
fun:write$UNIX2003
...
fun:pager_write
}
# See http://www.openradar.me/8287193
{
Invalid redzone accesses in DKeyHas8Words
Memcheck:Unaddressable
fun:DKeyHas8Words
}
# See https://bugs.kde.org/show_bug.cgi?id=188572
# This suppression is missing in Valgrind on Mac 10.6
# TODO(glider): remove it once it arrives in the trunk.
{
Unavoidable leak in setenv()
Memcheck:Leak
fun:malloc_zone_malloc
fun:__setenv
fun:setenv$UNIX2003
}
{
# Reported to Apple as rdar://6915429
bug_12525
Memcheck:Leak
...
fun:-[CIContextImpl render:toBitmap:rowBytes:bounds:format:colorSpace:]
}
{
bug_69436
Memcheck:Leak
...
fun:-[CIKernel initWithCString:noCopy:]
...
fun:-[NSPopUpButtonCell _drawIndicatorWithFrame:inView:]
}
{
# Capturer on Mac uses OpenGL driver, which triggers several warnings.
# The check has to be quite generic, as different hardware graphics cards
# will cause different sets of warnings.
bug_75037
Memcheck:Uninitialized
...
fun:_ZN8remoting*CapturerMac*
}
{
# See also http://openradar.appspot.com/radar?id=1235407
bug_77063
Memcheck:Free
fun:_ZdlPv
fun:_ZN15THFSPlusCatalogD2Ev
fun:_ZN5TNode10SetCatalogEP15THFSPlusCatalog
fun:_ZN15TMountPointList9AddVolumeEsb
fun:_ZN15TMountPointList4FindEsPN5TNode12StPopulatingE
fun:_ZN15TMountPointList20SupportsInvisibleBitEsPN5TNode12StPopulatingEb
fun:_ZNK21THFSPlusPropertyStore4OpenEbb
fun:_ZNK21THFSPlusPropertyStore13GetPropertiesEb
fun:_ZN16TFSCopyOperation22GetSourcePropertyStoreERK11THFSPlusRef
fun:_ZN16TFSCopyOperation13DoMoveToTrashERK11THFSPlusRef
fun:_ZN16TFSCopyOperation3RunEv
fun:_FSOperation
fun:_FSOperateOnObjectSync
fun:FSMoveObjectToTrashSync
fun:_Z9TrashFuncRK8FilePath
}
{
# See also http://openradar.appspot.com/radar?id=1169404
bug_79533a
Memcheck:Uninitialized
...
fun:_Z*19cssm_DataAbortQuery17cssm_dl_db_handlel
fun:CSSM_DL_DataAbortQuery
fun:_ZN11SSDLSession14DataAbortQueryEll
fun:_Z*19cssm_DataAbortQuery17cssm_dl_db_handlel
fun:CSSM_DL_DataAbortQuery
fun:tpDbFindIssuerCrl
fun:tpVerifyCertGroupWithCrls
}
{
# See also http://openradar.appspot.com/radar?id=1169404
bug_79533b
Memcheck:Uninitialized
...
fun:_Z*19cssm_DataAbortQuery17cssm_dl_db_handlel
fun:CSSM_DL_DataAbortQuery
fun:_ZN11SSDLSession14DataAbortQueryEll
fun:_Z*19cssm_DataAbortQuery17cssm_dl_db_handlel
fun:CSSM_DL_DataAbortQuery
fun:tpDbFindIssuerCrl
fun:tpVerifyCertGroupWithCrls
}
{
bug_85213_a
Memcheck:Leak
...
fun:_CFBundleCopyDirectoryContentsAtPath
}
{
bug_85213_b
Memcheck:Leak
...
fun:_CFBundleCopyInfoDictionaryInDirectoryWithVersion
}
{
bug_85213_c
Memcheck:Leak
...
fun:_CFBundleURLLooksLikeBundleVersion
}
{
bug_85213_d
Memcheck:Leak
...
fun:_CFBundleCreate
fun:_ZN6webkit5npapi9PluginLib17ReadWebPluginInfoERK8FilePathPNS0_13WebPluginInfoE
}
{
bug_85213_e
Memcheck:Leak
...
fun:CFBundlePreflightExecutable
fun:_ZN6webkit5npapi9PluginLib17ReadWebPluginInfoERK8FilePathPNS0_13WebPluginInfoE
}
{
bug_85213_f
Memcheck:Leak
...
fun:CFBundleGetPackageInfo
fun:_ZN6webkit5npapi9PluginLib17ReadWebPluginInfoERK8FilePathPNS0_13WebPluginInfoE
}
{
bug_86927
Memcheck:Leak
fun:malloc
fun:CGSMapShmem
fun:CGSResolveShmemReference
fun:CGSScoreboard
fun:initCGDisplayState
fun:initCGDisplayMappings
fun:cgsInit
fun:pthread_once
fun:CGSInitialize
fun:CGSServerOperationState
fun:+[NSThemeFrame initialize]
fun:_class_initialize
}
{
# QTKit leak. See http://crbug.com/100772 and rdar://10319535.
bug_100772
Memcheck:Leak
fun:calloc
fun:QTMLCreateMutex
fun:WarholCreateGlobals
fun:INIT_QuickTimeLibInternal
fun:pthread_once
fun:INIT_QuickTimeLib
fun:EnterMovies_priv
fun:EnterMovies
fun:TundraUnitInputFromTSFileEntry
fun:TundraUnitVDIGInputEntry
fun:TundraUnitCreateFromDescription
fun:+[QTCaptureVDIGDevice _refreshDevices]
fun:+[QTCaptureVDIGDevice devicesWithIOType:]
fun:+[QTCaptureDevice devicesWithIOType:]
fun:+[QTCaptureDevice inputDevices]
fun:+[QTCaptureDevice inputDevicesWithMediaType:]
fun:+[VideoCaptureDeviceQTKit deviceNames]
fun:_ZN5media18VideoCaptureDevice14GetDeviceNamesEPSt4listINS0_4NameESaIS2_EE
fun:_ZN5media21VideoCaptureDeviceMac4InitEv
fun:_ZN5media18VideoCaptureDevice6CreateERKNS0_4NameE
fun:_ZN5media45VideoCaptureDeviceTest_OpenInvalidDevice_Test8TestBodyEv
}
# 2. Intentional unit test errors, stuff that is somehow a false positive
# in our own code, or stuff that is so trivial it's not worth fixing.
# 3. Suppressions for real WebRTC bugs that are not yet fixed.

View File

@ -0,0 +1,156 @@
#!/usr/bin/env python
# Copyright (c) 2011 The WebRTC project authors. All Rights Reserved.
#
# Use of this source code is governed by a BSD-style license
# that can be found in the LICENSE file in the root of the source
# tree. An additional intellectual property rights grant can be found
# in the file PATENTS. All contributing project authors may
# be found in the AUTHORS file in the root of the source tree.
''' Runs various WebRTC tests through valgrind_test.py.
This script inherits the chrome_tests.py in Chrome, replacing its tests with
our own in WebRTC instead.
'''
import optparse
import sys
import logging_utils
import chrome_tests
class WebRTCTests(chrome_tests.ChromeTests):
# WebRTC tests, similar functions for each tests as the Chrome tests in the
# parent class.
def TestSignalProcessing(self):
return self.SimpleTest("signal_processing", "signal_processing_unittests")
def TestResampler(self):
return self.SimpleTest("resampler", "resampler_unittests")
def TestVAD(self):
return self.SimpleTest("vad", "vad_unittests")
def TestCNG(self):
return self.SimpleTest("cng", "cng_unittests")
def TestG711(self):
return self.SimpleTest("g711", "g711_unittests")
def TestG722(self):
return self.SimpleTest("g722", "g722_unittests")
def TestPCM16B(self):
return self.SimpleTest("pcm16b", "pcm16b_unittests")
def TestNetEQ(self):
return self.SimpleTest("neteq", "neteq_unittests")
def TestAudioConferenceMixer(self):
return self.SimpleTest("audio_conference_mixer", "audio_conference_mixer_unittests")
def TestMediaFile(self):
return self.SimpleTest("media_file", "media_file_unittests")
def TestRTPRTCP(self):
return self.SimpleTest("rtp_rtcp", "rtp_rtcp_unittests")
def TestBWE(self):
return self.SimpleTest("test_bwe", "test_bwe")
def TestUDPTransport(self):
return self.SimpleTest("udp_transport", "udp_transport_unittests")
def TestWebRTCUtility(self):
return self.SimpleTest("webrtc_utility", "webrtc_utility_unittests")
def TestVP8(self):
return self.SimpleTest("vp8", "vp8_unittests")
def TestVideoCoding(self):
return self.SimpleTest("video_coding", "video_coding_unittests")
def TestVideoProcessing(self):
return self.SimpleTest("video_processing", "video_processing_unittests")
def TestSystemWrappers(self):
return self.SimpleTest("system_wrappers", "system_wrappers_unittests")
def TestTestSupport(self):
return self.SimpleTest("test_support", "test_support_unittests")
def _main(_):
parser = optparse.OptionParser("usage: %prog -b <dir> -t <test> "
"[-t <test> ...]")
parser.disable_interspersed_args()
parser.add_option("-b", "--build_dir",
help="the location of the compiler output")
parser.add_option("-t", "--test", action="append", default=[],
help="which test to run, supports test:gtest_filter format "
"as well.")
parser.add_option("", "--baseline", action="store_true", default=False,
help="generate baseline data instead of validating")
parser.add_option("", "--gtest_filter",
help="additional arguments to --gtest_filter")
parser.add_option("", "--gtest_repeat",
help="argument for --gtest_repeat")
parser.add_option("-v", "--verbose", action="store_true", default=False,
help="verbose output - enable debug log messages")
parser.add_option("", "--tool", dest="valgrind_tool", default="memcheck",
help="specify a valgrind tool to run the tests under")
parser.add_option("", "--tool_flags", dest="valgrind_tool_flags", default="",
help="specify custom flags for the selected valgrind tool")
parser.add_option("", "--keep_logs", action="store_true", default=False,
help="store memory tool logs in the <tool>.logs directory "
"instead of /tmp.\nThis can be useful for tool "
"developers/maintainers.\nPlease note that the <tool>"
".logs directory will be clobbered on tool startup.")
options, args = parser.parse_args()
if options.verbose:
logging_utils.config_root(logging.DEBUG)
else:
logging_utils.config_root()
if not options.test:
parser.error("--test not specified")
if len(options.test) != 1 and options.gtest_filter:
parser.error("--gtest_filter and multiple tests don't make sense together")
for t in options.test:
tests = WebRTCTests(options, args, t)
ret = tests.Run()
if ret: return ret
return 0
if __name__ == "__main__":
# Overwrite the ChromeTests tests dictionary with our WebRTC tests.
# The cmdline option allows the user to pass any executable as parameter to
# the test script, which is useful when developing new tests that are not yet
# present in this script.
chrome_tests.ChromeTests._test_list = {
"cmdline": chrome_tests.ChromeTests.RunCmdLine,
"signal_processing": WebRTCTests.TestSignalProcessing,
"resampler": WebRTCTests.TestResampler,
"vad": WebRTCTests.TestVAD,
"cng": WebRTCTests.TestCNG,
"g711": WebRTCTests.TestG711,
"g722": WebRTCTests.TestG722,
"pcm16b": WebRTCTests.TestPCM16B,
"neteq": WebRTCTests.TestNetEQ,
"audio_conference_mixer": WebRTCTests.TestAudioConferenceMixer,
"media_file": WebRTCTests.TestMediaFile,
"rtp_rtcp": WebRTCTests.TestRTPRTCP,
"test_bwe": WebRTCTests.TestBWE,
"udp_transport": WebRTCTests.TestUDPTransport,
"webrtc_utility": WebRTCTests.TestWebRTCUtility,
"vp8": WebRTCTests.TestVP8,
"video_coding": WebRTCTests.TestVideoCoding,
"video_processing": WebRTCTests.TestVideoProcessing,
"system_wrappers": WebRTCTests.TestSystemWrappers,
"test_support": WebRTCTests.TestTestSupport,
}
ret = _main(sys.argv)
sys.exit(ret)

View File

@ -0,0 +1,102 @@
#!/bin/bash
# Copyright (c) 2011 The WebRTC project authors. All Rights Reserved.
#
# Use of this source code is governed by a BSD-style license
# that can be found in the LICENSE file in the root of the source
# tree. An additional intellectual property rights grant can be found
# in the file PATENTS. All contributing project authors may
# be found in the AUTHORS file in the root of the source tree.
# Set up some paths and re-direct the arguments to webrtc_tests.py
# This script is a copy of the chrome_tests.sh wrapper script with the following
# changes:
# - The locate_valgrind.sh of Chromium's Valgrind scripts dir is used to locate
# the Valgrind framework install.
# - webrtc_tests.py is invoked instead of chrome_tests.py.
# - Chromium's Valgrind scripts directory is added to the PYTHONPATH to make it
# possible to execute the Python scripts properly.
export THISDIR=`dirname $0`
ARGV_COPY="$@"
# We need to set CHROME_VALGRIND iff using Memcheck or TSan-Valgrind:
# tools/valgrind/chrome_tests.sh --tool memcheck
# or
# tools/valgrind/chrome_tests.sh --tool=memcheck
# (same for "--tool=tsan")
tool="memcheck" # Default to memcheck.
while (( "$#" ))
do
if [[ "$1" == "--tool" ]]
then
tool="$2"
shift
elif [[ "$1" =~ --tool=(.*) ]]
then
tool="${BASH_REMATCH[1]}"
fi
shift
done
NEEDS_VALGRIND=0
NEEDS_DRMEMORY=0
case "$tool" in
"memcheck")
NEEDS_VALGRIND=1
;;
"tsan" | "tsan_rv")
NEEDS_VALGRIND=1
;;
"drmemory" | "drmemory_light" | "drmemory_full")
NEEDS_DRMEMORY=1
;;
esac
# For WebRTC, we'll use the locate_valgrind.sh script in Chromium's Valgrind
# scripts dir to locate the Valgrind framework install
CHROME_VALGRIND_SCRIPTS=$THISDIR/../valgrind
if [ "$NEEDS_VALGRIND" == "1" ]
then
CHROME_VALGRIND=`sh $CHROME_VALGRIND_SCRIPTS/locate_valgrind.sh`
if [ "$CHROME_VALGRIND" = "" ]
then
# locate_valgrind.sh failed
exit 1
fi
echo "Using valgrind binaries from ${CHROME_VALGRIND}"
PATH="${CHROME_VALGRIND}/bin:$PATH"
# We need to set these variables to override default lib paths hard-coded into
# Valgrind binary.
export VALGRIND_LIB="$CHROME_VALGRIND/lib/valgrind"
export VALGRIND_LIB_INNER="$CHROME_VALGRIND/lib/valgrind"
fi
if [ "$NEEDS_DRMEMORY" == "1" ]
then
if [ -z "$DRMEMORY_COMMAND" ]
then
DRMEMORY_PATH="$THISDIR/../../third_party/drmemory"
DRMEMORY_SFX="$DRMEMORY_PATH/drmemory-windows-sfx.exe"
if [ ! -f "$DRMEMORY_SFX" ]
then
echo "Can't find Dr. Memory executables."
echo "See http://www.chromium.org/developers/how-tos/using-valgrind/dr-memory"
echo "for the instructions on how to get them."
exit 1
fi
chmod +x "$DRMEMORY_SFX" # Cygwin won't run it without +x.
"$DRMEMORY_SFX" -o"$DRMEMORY_PATH/unpacked" -y
export DRMEMORY_COMMAND="$DRMEMORY_PATH/unpacked/bin/drmemory.exe"
fi
fi
# Add Chrome's Valgrind scripts dir to the PYTHON_PATH since it contains
# the scripts that are needed for this script to run
PYTHONPATH=$THISDIR/../python/google:$CHROME_VALGRIND_SCRIPTS python \
"$THISDIR/webrtc_tests.py" $ARGV_COPY