New project: analyse and improve the performance of massive repack

[root@c2cernt3srv301 ~]# vmgrlisttape -V T55654
T55654   T55654 SL8600_C 5000GC   aul ams_1            175.48GiB 20121028 FULL

showqueues -x | grep T10KC6
DN T10KC6 T10C6010@tpsrv673 FREE 358 (host=c2(atlas|alice|cernt3|lhcb|public).*) None

put the drive down and dedicate (both ways) it to the tape

-bash-3.2$ tape-mount --vid T55654
2012/11/09-10:49:09 TapeAdmin::get1stDrive [WARN] - no drive specified, will use 'T10C641E' (which has been identified as the first drive)
2012/11/09-10:49:55 tape-mount [INFO] - tape T55654 mounted

-bash-3.2$ time (dd if=/dev/tape of=/dev/null ibs=256k; while [ $? -eq 0 ]; do dd if=/dev/tape of=/dev/null ibs=256k; done) &> T55654_dd_read_res.txt

to check the historical contents of a tape go to: /afs/cern.ch/project/castor/tape/tape-stats and grep for the tape-name!:)

Tape request statistics are available in castortapelog in /var/log/tape/castor-tape.log as RT044 (grep for it:) ).

TransferTime=24 DataVolumeMB=3071.498905 DataRateMBs=127.979121

Possible bottlenecks:

• Repack and recall request initilization:
  • Theoretical speed:
  • Speed observed:
• Tape read speed:
  • Theoretical speed:
    • Oracle T10000C drives: 240 MB/sec
    • Oracle T10000B drives: 120 MB/sec
    • IBM TS1140 drives:
      • 13 speeds, from 76 MB/sec to 251 MB/sec for 3592 JC/JK/JY cartridges initialized in Gen 4 format
      • 13 speeds, from 74 MB/sec to 203 MB/sec for 3592 JB/JX cartridges initialized in Gen 4 format
      • 13 speeds, from 41 MB/sec to 163 MB/sec for 3592 JB/JX cartridges initialized in Gen 3 format
      • 13 speeds, from 39 MB/sec to 151 MB/sec for 3592 JB/JX cartridges initialized in Gen 2 format
      • 13 speeds, from 36 MB/sec to 144 MB/sec for 3592 JA/JJ/JR/JW cartridges initialized in Gen 2 or Gen 3 format
      • 13 speeds, from 18 MB/sec to 72 MB/sec for 3592 JA/JJ/JR/JW cartridges initialized in Gen 1 format
    • IBM TS1130 drives:
      • 163, 134, 109, 83, 56, 43 MB/sec for 3592 JB cartridges initialized in Gen 3 format
      • 150, 127, 104, 78, 52, 40 MB/sec for 3592 JB cartridges initialized in Gen 2 format
      • 71, 59, 47, 36, 24, 19, 13 MB/sec for 3592 JA cartridges initialized in Gen 1 format
  • Speed observed:
• Input tape to stager network speed: I observed a clustering of recall transfers on few diskservers bringing a contention problem with a consequent degradation of the performance (the overall recall performance of the cernt3 cluster was 600-800 MB/s). After discussing this with Giuseppe Lo Presti, we changed the scheduling algorithm for a completely random one and the performance improved up to about 1GB/s.
  • Theoretical speed:
    • Around 120 MB/s for a 1Gb eth connections
  • Speed observed:
• Migration request initialization:
  • Theoretical speed:
  • Speed observed:
• Stager to output tape network speed:
  • Theoretical speed:
  • Speed observed:
• Tape write speed:
  • Theoretical speed:
  • Speed observed:

-- DanieleFrancescoKruse - 12-Jun-2012