Test of CMSSW_3_10_0_pre7io

Test used: CPT, with MTR3 job, some simple job config for GSIdcap test (added by Christoph to Leo's page)

Site	SE Technology	MTR3 32bit	MTR3 64bit
T2_BE_IIHE	dCache
T2_CH_CSCS	dCache
T2_ES_CIEMAT	dCache
T2_FR_GRIF_LLR	dpm
T2_IT_Bari	Lustre+Storm
T2_DE_DESY	dCache(GSIdcap)
Overall

Summary:

• Still issues with DPM without Sartirana's hack
• 64bit release breaks:
  • glite, due to bad (?) compiled libssl
  • CRAB, due to shipped python libraries v 2.6.4-cms9 (2.6.4-cms6 works)
• dCache with GSIdcap still does not work properly

32 bit

Notes:

• Bari failed because the files are not physically available (thanks Giacinto!)
• GRIF had a similar issue, solved (thanks Andrea!)

DPM crashes without A.Sartirana's workaround

Actually, on DPM he release is not working WITHOUT THE WORKAROUND described here. In order to avoid the hack, a customized job has been sent to GRIF:

#!/bin/bash

LOG="cmssw"

eval `scram ru -sh`

env
export LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH  | sed "s=/opt/exp_soft/cms/mylib:==g" `
echo "--------------------------------"
env

cmsRun -j ${LOG}.xml pset.py

where /opt/exp_soft/cms/mylib is where the "hacked" libraries are. Without them, a new error appears:

07-Dec-2010 19:23:24 CET  Initiating request to open file rfio:/dpm/in2p3.fr/home/cms/trivcat/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000/D005BB56-CA2B-DF11-BA08-0030487C60AE.root
%MSG-e Root_Error:  file_open TUnixSystem::DispatchSignals()  07-Dec-2010 19:23:24 CET pre-events
segmentation violation
%MSG
Attaching to program: /proc/16136/exe, process 16136
[Thread debugging using libthread_db enabled]
[New Thread 0xf3953b90 (LWP 16138)]
[New Thread 0xf4354b90 (LWP 16137)]
0xffffe410 in __kernel_vsyscall ()
Thread 3 (Thread 0xf4354b90 (LWP 16137)):
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xf57fdc4e in do_sigwait () from /lib/libpthread.so.0
#2  0xf57fdcef in sigwait () from /lib/libpthread.so.0
#3  0xf45290c1 in globus_l_callback_thread_signal_poll (user_arg=0x0)
    at globus_callback_threads.c:2841
#4  0xf4540293 in thread_starter (temparg=0xa46a450)
    at globus_thread_pthreads.c:508
#5  0xf57f5832 in start_thread () from /lib/libpthread.so.0
#6  0x00704f6e in clone () from /lib/libc.so.6

...

dCache with GSIdcap

File access fails with authentification problems in GSIdcap. The problem is only appearing if openssl libraries are loaded from CMS software area. In cases where the cmsRun does not access Frontier (no realistic use case), openssl libraries are loaded from the OS and GSIdcap works.

Using Frontier the failure looks like this:

Error ( POLLIN POLLERR POLLHUP) (with data) on control line [119]
Failed to create a control line
Failed open file in the dCache.
%MSG-w StorageFactory::stagein():  PoolSource:source@sourceConstruction  16-Dec-2010 11:03:02 CET pre-events
Failed to stage in file 'gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000//0EDC7961-822C-DF11-A0BE-001617E30E2C.root' because:
---- DCacheStorageMaker::stagein() BEGIN
Cannot stage in file 'gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000//0EDC7961-822C-DF11-A0BE-001617E30E2C.root', error was: Server rejected "hello" (dc_errno=26)
---- DCacheStorageMaker::stagein() END

%MSG
16-Dec-2010 11:03:02 CET  Initiating request to open file gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000//0EDC7961-822C-DF11-A0BE-001617E30E2C.root
Error ( POLLIN POLLERR POLLHUP) (with data) on control line [119]
Failed to create a control line
Failed open file in the dCache.
%MSG-s CMSException:  AfterFile 16-Dec-2010 11:03:02 CET pre-events
cms::Exception caught in cmsRun
---- FileOpenError BEGIN
---- StorageFactory::open() BEGIN
Failed to open the file 'gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000/0EDC7961-822C-DF11-A0BE-001617E30E2C.root' because:
---- DCacheFile::open() BEGIN
dc_open(name='gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000/0EDC7961-822C-DF11-A0BE-001617E30E2C.root', flags=0x0, permissions=0666) => error 'Server rejected "hello"' (dc_errno=26)
---- DCacheFile::open() END
---- StorageFactory::open() END

RootInputFileSequence::initFile(): Input file gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000//0EDC7961-822C-DF11-A0BE-001617E30E2C.root was not found or could not be opened.

Error occurred while creating source PoolSource
---- FileOpenError END


%MSG

I did not succeed to get this working by putting symbolic links to the OS openssl libraries. (Strange).

64 bit

Forcing 64 SCRAM arch with: export SCRAM_ARCH=slc5_amd64_gcc434

libssl issue

[leo@t3ui02 lib]$ file /lib64/libssl.so.0.9.8e
/lib64/libssl.so.0.9.8e: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), stripped
[leo@t3ui02 lib]$ file /shome/leo/CMSSW_RELEASES/slc5_amd64_gcc434/external/openssl/0.9.8e-cms2/lib/libssl.so.6
/shome/leo/CMSSW_RELEASES/slc5_amd64_gcc434/external/openssl/0.9.8e-cms2/lib/libssl.so.6: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped

CRAB segfaults

After the substitution, issues with CRAB:

[leo@t3ui02 test]$ crab -create -submit
crab:  Version 2.7.5 running on Wed Dec  8 15:57:50 2010 CET (14:57:50 UTC)

crab. Working options:
        scheduler           glite
        job type            CMSSW
        server              OFF
        working directory   /shome/leo/IntegrationTests/CMSSW_3_10_0_pre7io_64bit/src/Tests/TestJobs/test/T2_CH_CSCS-MTR3-RelValProdTTbarJobRobotMC_3XY_V24_JobRobotv1-10000-CMSSW_3_10_0_pre7io_64bit-201012081029/

/swshare/CRAB/CRAB_2_7_5_patch1/python/crab: line 34:   603 Segmentation fault      python $CRABPYTHON/crab.py $*

Using pdm module:

cd /shome/leo/IntegrationTests/CMSSW_3_10_0_pre7io_64bit/src/Tests/TestJobs/test
gridinit
export SCRAM_ARCH=slc5_amd64_gcc434
export VO_CMS_SW_DIR=/shome/leo/CMSSW_RELEASES/
source /shome/leo/CMSSW_RELEASES/cmsset_default.sh
cmsenv
crabinit
source /swshare/CRAB/CRAB_2_7_5_patch1/python/crab
[leo@t3ui04 test]$ python -m pdb $CRABDIR/python/crab.py -create -submit
> /swshare/CRAB/CRAB_2_7_5_patch1/python/crab.py(2)<module>()
-> import sys, os, time, string
(Pdb) b /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/CmsSiteMapper.py:212
Breakpoint 1 at /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/CmsSiteMapper.py:212
(Pdb) c
crab:  Version 2.7.5 running on Fri Dec 10 12:33:23 2010 CET (11:33:23 UTC)

crab. Working options:
        scheduler           glite
        job type            CMSSW
        server              OFF
        working directory   /shome/leo/IntegrationTests/CMSSW_3_10_0_pre7io_64bit/src/Tests/TestJobs/test/T2_CH_CSCS-MTR3-RelValProdTTbarJobRobotMC_3XY_V24_JobRobotv1-10000-CMSSW_3_10_0_pre7io_64bit-201012081727/

crab:  WARNING: CMSSW_3_10_0_pre7io on slc5_amd64_gcc434 is not a supported release. Submission may fail.
crab:  Contacting Data Discovery Services ...
crab:  Accessing DBS at: http://cmsdbsprod.cern.ch/cms_dbs_prod_global/servlet/DBSServlet
crab:  Requested dataset: /RelValProdTTbar/JobRobot-MC_3XY_V24_JobRobot-v1/GEN-SIM-DIGI-RECO has 300000 events in 1 blocks.

> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/CmsSiteMapper.py(212)load()

...

> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(123)load_report()
-> def load_report(self, report):
(Pdb) l
118             params = urllib.urlencode(params)
119             fd = urllib2.urlopen(self.sitedb_url, params)
120             return fd
121
122
123  ->     def load_report(self, report):
124             """
125             Load the SiteDB report and return the DOM contents.
126
127             Great care is taken to make this resilient, including timeouts and
128             fallback to the contents of the cache.
(Pdb)
129
130             @param report: The name of the SiteDB report
131             @returns: The DOM object representing the contents of the report
132             """
133
134             # Start off by setting the alarm clock to timeout faulty operations.
135             def interrupt_op(*args):
136                 raise AlarmClock()
137             signal.signal(signal.SIGALRM, interrupt_op)
138             try:
139                 # Check the contents of the local cache; only return here if they
(Pdb)
140                 # are fresh and parse cleanly.
141                 try:
142                     signal.alarm(self.alarm_timeout)
143                     fresh, results = self.check_cache(report)
144                     if results:
145                         try:
146                             results = parse(results)
147                             if fresh:
148                                 return results
149                         except:
150                             results = None
(Pdb) n
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(135)load_report()
-> def interrupt_op(*args):
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(137)load_report()
-> signal.signal(signal.SIGALRM, interrupt_op)
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(138)load_report()
-> try:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(141)load_report()
-> try:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(142)load_report()
-> signal.alarm(self.alarm_timeout)
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(143)load_report()
-> fresh, results = self.check_cache(report)
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(144)load_report()
-> if results:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(145)load_report()
-> try:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(146)load_report()
-> results = parse(results)
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(147)load_report()
-> if fresh:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(156)load_report()
-> try:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(157)load_report()
-> signal.alarm(self.alarm_timeout)
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(158)load_report()
-> urlresults = self.load_siteDB(report)
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(159)load_report()
-> try:
(Pdb)
> /swshare/CRAB/CRAB_2_7_5_patch1/external/ProdCommon/SiteDB/SiteDBReport.py(160)load_report()
-> urlresults = parse(urlresults)
(Pdb) s
--Call--
> /shome/leo/CMSSW_RELEASES/slc5_amd64_gcc434/external/python/2.6.4-cms9/lib/python2.6/xml/dom/minidom.py(1914)parse()
-> def parse(file, parser=None, bufsize=None):
(Pdb) l
1909        toktype, rootNode = events.getEvent()
1910        events.expandNode(rootNode)
1911        events.clear()
1912        return rootNode
1913
1914 -> def parse(file, parser=None, bufsize=None):
1915        """Parse a file into a DOM by filename or file object."""
1916        if parser is None and not bufsize:
1917            from xml.dom import expatbuilder
1918            return expatbuilder.parse(file)
1919        else:
(Pdb)
1920            from xml.dom import pulldom
1921            return _do_pulldom_parse(pulldom.parse, (file,),
1922                {'parser': parser, 'bufsize': bufsize})
1923
1924    def parseString(string, parser=None):
1925        """Parse a file into a DOM from a string."""
1926        if parser is None:
1927            from xml.dom import expatbuilder
1928            return expatbuilder.parseString(string)
1929        else:
1930            from xml.dom import pulldom
(Pdb) l
1931            return _do_pulldom_parse(pulldom.parseString, (string,),
1932                                     {'parser': parser})
1933
1934    def getDOMImplementation(features=None):
1935        if features:
1936            if isinstance(features, StringTypes):
1937                features = domreg._parse_feature_string(features)
1938            for f, v in features:
1939                if not Document.implementation.hasFeature(f, v):
1940                    return None
1941        return Document.implementation
(Pdb) n
> /shome/leo/CMSSW_RELEASES/slc5_amd64_gcc434/external/python/2.6.4-cms9/lib/python2.6/xml/dom/minidom.py(1916)parse()
-> if parser is None and not bufsize:
(Pdb)
> /shome/leo/CMSSW_RELEASES/slc5_amd64_gcc434/external/python/2.6.4-cms9/lib/python2.6/xml/dom/minidom.py(1917)parse()
-> from xml.dom import expatbuilder
(Pdb)
> /shome/leo/CMSSW_RELEASES/slc5_amd64_gcc434/external/python/2.6.4-cms9/lib/python2.6/xml/dom/minidom.py(1918)parse()
-> return expatbuilder.parse(file)
(Pdb)
Segmentation fault

The issue seems located in the python libraries shipped with CMSSW_3_10_0_pre7io, which are version 2.6.4-cms9. Doing a test and substituting in LD_LIBRARY_PATH with 2.6.4-cms6 (shipped with CMSSW_3_8_6) crab works.

dCache with GSIdcap

This fails with segmentation fault regardless of having Frontier in the game or not. The intersting difference regarding GSIdcap is the fact that the 32bit libgsiTunne.so is not linked shared against openssl while the 64bit flavour is:

ldd /opt/d-cache/dcap/lib/libgsiTunnel.so 
ldd: warning: you do not have execution permission for `/opt/d-cache/dcap/lib/libgsiTunnel.so'
        linux-gate.so.1 =>  (0xffffe000)
        libdl.so.2 => /lib/libdl.so.2 (0xf7de9000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0xf7db7000)
        libresolv.so.2 => /lib/libresolv.so.2 (0xf7da2000)
        libc.so.6 => /lib/libc.so.6 (0xf7c49000)
        /lib/ld-linux.so.2 (0x00bac000)

 ldd /opt/d-cache/dcap/lib/libgsiTunnel.so 
ldd: warning: you do not have execution permission for `/opt/d-cache/dcap/lib/libgsiTunnel.so'
        linux-gate.so.1 =>  (0xffffe000)
        libdl.so.2 => /lib/libdl.so.2 (0xf7de9000)
        libcrypt.so.1 => /lib/libcrypt.so.1 (0xf7db7000)
        libresolv.so.2 => /lib/libresolv.so.2 (0xf7da2000)
        libc.so.6 => /lib/libc.so.6 (0xf7c49000)
        /lib/ld-linux.so.2 (0x00bac000)
sgmcms@t2-cms-vo2: [~/Testing/slc5_amd64_gcc434/CMSSW_3_10_0_pre7io/src] ldd /opt/d-cache/dcap/lib64/libgsiTunnel.so 
        libglobus_gssapi_gsi_gcc64pthr.so.0 => /opt/globus/lib/libglobus_gssapi_gsi_gcc64pthr.so.0 (0x00002b62584ad000)
        libglobus_gsi_callback_gcc64pthr.so.0 => /opt/globus/lib/libglobus_gsi_callback_gcc64pthr.so.0 (0x00002b62586c5000)
        libglobus_gsi_cert_utils_gcc64pthr.so.0 => /opt/globus/lib/libglobus_gsi_cert_utils_gcc64pthr.so.0 (0x00002b62588cf000)
        libglobus_gsi_proxy_core_gcc64pthr.so.0 => /opt/globus/lib/libglobus_gsi_proxy_core_gcc64pthr.so.0 (0x00002b6258ad4000)
        libglobus_gsi_credential_gcc64pthr.so.0 => /opt/globus/lib/libglobus_gsi_credential_gcc64pthr.so.0 (0x00002b6258ce2000)
        libglobus_openssl_gcc64pthr.so.0 => /opt/globus/lib/libglobus_openssl_gcc64pthr.so.0 (0x00002b6258ef1000)
        libglobus_gsi_sysconfig_gcc64pthr.so.0 => /opt/globus/lib/libglobus_gsi_sysconfig_gcc64pthr.so.0 (0x00002b62590f3000)
        libglobus_openssl_error_gcc64pthr.so.0 => /opt/globus/lib/libglobus_openssl_error_gcc64pthr.so.0 (0x00002b62592fe000)
        libglobus_oldgaa_gcc64pthr.so.0 => /opt/globus/lib/libglobus_oldgaa_gcc64pthr.so.0 (0x00002b6259502000)
        libglobus_proxy_ssl_gcc64pthr.so.0 => /opt/globus/lib/libglobus_proxy_ssl_gcc64pthr.so.0 (0x00002b625970a000)
        libglobus_common_gcc64pthr.so.0 => /opt/globus/lib/libglobus_common_gcc64pthr.so.0 (0x00002b625990f000)
        libz.so.1 => /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/external/slc5_amd64_gcc434/lib/libz.so.1 (0x00002b6259b4c000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b6259b8a000)
        libltdl_gcc64pthr.so.3 => /opt/globus/lib/libltdl_gcc64pthr.so.3 (0x00002b6259ee1000)
        libssl.so.6 => /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/external/slc5_amd64_gcc434/lib/libssl.so.6 (0x00002b625a0e9000)
        libcrypto.so.6 => /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/external/slc5_amd64_gcc434/lib/libcrypto.so.6 (0x00002b625a132000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b625a295000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b625a49a000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003242a00000)

cmsRun crashes actually with traceback rather similar to the DPM case.

%MSG-e Root_Error:  PoolSource:source@sourceConstruction  TUnixSystem::DispatchSignals() 16-Dec-2010 11:08:42 CET pre-events
segmentation violation
%MSG
Attaching to program: /proc/5979/exe, process 5979
[Thread debugging using libthread_db enabled]
[New Thread 0x4206f940 (LWP 5981)]
[New Thread 0x4166e940 (LWP 5980)]
0x0000003242e99fff in waitpid () from /lib64/libc.so.6
Thread 3 (Thread 0x4166e940 (LWP 5980)):
#0  0x0000003243a0e838 in do_sigwait () from /lib64/libpthread.so.0
#1  0x0000003243a0e8dd in sigwait () from /lib64/libpthread.so.0
#2  0x00002b0d699dede5 in globus_l_callback_thread_signal_poll ()
   from /opt/globus/lib/libglobus_common_gcc64pthr.so.0
#3  0x00002b0d699f2c58 in thread_starter ()
   from /opt/globus/lib/libglobus_common_gcc64pthr.so.0
#4  0x0000003243a0673d in start_thread () from /lib64/libpthread.so.0
#5  0x0000003242ed3f6d in clone () from /lib64/libc.so.6
#6  0x0000000000000000 in ?? ()

Thread 2 (Thread 0x4206f940 (LWP 5981)):
#0  0x0000003243a0aee9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1  0x00002b0d699f2696 in globus_cond_wait ()
   from /opt/globus/lib/libglobus_common_gcc64pthr.so.0
#2  0x00002b0d699dfddd in globus_l_callback_thread_poll ()
   from /opt/globus/lib/libglobus_common_gcc64pthr.so.0
#3  0x00002b0d699f2edf in globus_l_thread_pool_thread_start ()
   from /opt/globus/lib/libglobus_common_gcc64pthr.so.0
#4  0x00002b0d699f2c58 in thread_starter ()
   from /opt/globus/lib/libglobus_common_gcc64pthr.so.0
#5  0x0000003243a0673d in start_thread () from /lib64/libpthread.so.0
#6  0x0000003242ed3f6d in clone () from /lib64/libc.so.6
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x2b0d67067390 (LWP 5979)):
#0  0x0000003242e99fff in waitpid () from /lib64/libc.so.6
#1  0x0000003242e3c331 in do_system () from /lib64/libc.so.6
#2  0x0000003242e3c687 in system () from /lib64/libc.so.6
#3  0x00002b0d65b32a65 in TUnixSystem::StackTrace() ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/external/slc5_amd64_gcc434/lib/libCore.so
#4  0x00002b0d65b33605 in TUnixSystem::DispatchSignals(ESignals) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/external/slc5_amd64_gcc434/lib/libCore.so
#5  <signal handler called>
#6  SSL_CIPHER_description (cipher=0x735b442e1905f2e0, 
    buf=0x7fff41ad4b00 "H*\025C2", len=256) at ssl_ciph.c:1110
#7  0x00002b0d68579001 in globus_i_gsi_gss_handshake ()
   from /opt/globus/lib/libglobus_gssapi_gsi_gcc64pthr.so.0
#8  0x00002b0d68574492 in gss_init_sec_context ()
   from /opt/globus/lib/libglobus_gssapi_gsi_gcc64pthr.so.0
#9  0x00002b0d68357507 in eInit () from /opt/d-cache/dcap/lib64/libgsiTunnel.so
#10 0x00002b0d6834006e in cache_connect (srv=0xa3c8120) at dcap.c:751
#11 0x00002b0d68340568 in serverConnect (node=0xa3c85a0) at dcap.c:569
#12 initControlLine (node=0xa3c85a0) at dcap.c:306
#13 0x00002b0d683412f4 in cache_open (node=0xa3c85a0) at dcap.c:250
#14 0x00002b0d68345438 in dc_open (
    fname=0xa3c8438 "gsidcap://dcache-cms-gsidcap.desy.de:22128//pnfs/desy.de/cms/tier2/store/mc/JobRobot/RelValProdTTbar/GEN-SIM-DIGI-RECO/MC_3XY_V24_JobRobot-v1/0000//0EDC7961-822C-DF11-A0BE-001617E30E2C.root", flags=2048)
    at dcap_open.c:280
#15 0x00002b0d68345cc3 in dc_stage (
    path=0x735b442e1905f2e0 <Address 0x735b442e1905f2e0 out of bounds>, 
    atime=<value optimized out>, location=0xa40b4a0 "") at dcap_open.c:409
#16 0x00002b0d6832ba51 in DCacheStorageMaker::stagein(std::string const&, std::string const&) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/pluginUtilitiesDCacheAdaptorPlugin.so
#17 0x00002b0d67879f25 in StorageFactory::stagein(std::string const&) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/libUtilitiesStorageFactory.so
#18 0x00002b0d6819081c in edm::RootInputFileSequence::RootInputFileSequence(edm::ParameterSet const&, edm::PoolSource const&, edm::InputFileCatalog const&, edm::PrincipalCache&, bool) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#19 0x00002b0d6815ff42 in edm::PoolSource::PoolSource(edm::ParameterSet const&, edm::InputSourceDescription const&) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#20 0x00002b0d6815dc84 in edmplugin::PluginFactory<edm::InputSource* ()(edm::ParameterSet const&, edm::InputSourceDescription const&)>::PMaker<edm::PoolSource>::create(edm::ParameterSet const&, edm::InputSourceDescription const&) const ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/pluginIOPoolInput.so
#21 0x00002b0d643c8155 in edm::InputSourceFactory::makeInputSource(edm::ParameterSet const&, edm::InputSourceDescription const&) const ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#22 0x00002b0d6436a4a5 in edm::makeInput(edm::ParameterSet&, edm::EventProcessor::CommonParams const&, edm::ProductRegistry&, edm::PrincipalCache&, boost::shared_ptr<edm::ActivityRegistry>, boost::shared_ptr<edm::ProcessConfiguration>) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#23 0x00002b0d6436dd13 in edm::EventProcessor::init(boost::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#24 0x00002b0d643766f3 in edm::EventProcessor::EventProcessor(boost::shared_ptr<edm::ProcessDesc>&, edm::ServiceToken const&, edm::serviceregistry::ServiceLegacy) ()
   from /opt/vo/cms/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_10_0_pre7io/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#25 0x000000000040d833 in main ()
A debugging session is active.

        Inferior 1 [process 5979] will be detached.

-- LeonardoSala - 07-Dec-2010