The basics:

• You need access to a machine with perfmon2 installed
• You need the perfmon manual: http://perfmon2.sourceforge.net/pfmon_usersguide.html
• You need to know which counters to get
  • pfmon -l lists all available counters
  • pfmon -i str lists detailed info about counters that contain str in their name
  • Basic counters that I got working on a Intel i7 (lxcmsi1): (boh? the machine now is down)
  • Basic counters that I got working on a Xeron E530 (pcphsft50): UNHALTED_CORE_CYCLES,INSTRUCTIONS_RETIRED
• You need a way to start and stop perfmon so that you profile only your code
  • Get the module addpkg PerfTools/Callgrind && cd PerfTools/Callgrind && patch -p0 < ~gpetrucc/public/pfmon-trigger.patch
  • If you want to instrument a fragment of C++ code, put it between cms_global_start_perfmon() and cms_global_start_perfmon() (you need to #include "PerfTools/Callgrind/interface/PerfMonHook.h" and to <use name=PerfTools/Callgrind> in your BuildFile)
  • If you want to instrument a sequence of CMSSW modules, define the startPerfmon and stopPerfmon modules as described here, and use them to bracket your sequence

process.startPerfmon = cms.EDAnalyzer("PerfMonTrigger",
    eventsToSkip    = cms.untracked.uint32(5),
    eventsToProfile = cms.untracked.uint32(20),
    operation       = cms.untracked.string("start")
)
process.stopPerfmon = process.startPerfmon.clone(operation = cms.untracked.string("stop"))

• • export LD_PRELOAD=libPerfToolsCallgrind.so . It will spam you out with ld.so error messages, but it works. Cleaner solution is to make a short script that just does the export and runs cmsRun.
  • invoke pfmon as in this example

pfmon --follow-all --resolv  \
          --trigger-code-start='cms_global_start_perfmon()' --trigger-code-stop='cms_global_stop_perfmon()' \
          -e UNHALTED_CORE_CYCLES,INSTRUCTIONS_RETIRED \
           cmsRun cfg.py 2>&1 | tee pfmon-cfg.log

-- GiovanniPetrucciani - 10 Mar 2009