ATLAS DDM Operations at Prague, 2007

October 15

There are last 70 GB free on the shared pool dpmpool4. Check ddmf5 files on our dpm:

mysql -p -u dpm
use cns_db;
select m.name, m.filesize, date_format(from_unixtime(m.ctime),'%Y-%m-%d') from Cns_file_metadata m where name like 'ddmf5%';
+------------------------------------------+----------+------------------------------------------------+
| name                                     | filesize | date_format(from_unixtime(m.ctime),'%Y-%m-%d') |
+------------------------------------------+----------+------------------------------------------------+
| ddmf5                                    |        0 | 2007-10-02                                     |
| ddmf5.001013.GRIDKA.GLOBAL.Step1.v000411 |        0 | 2007-10-09                                     |
+------------------------------------------+----------+------------------------------------------------+

Just two directories left. I will delete them using rfrm:

rfrm -r /dpm/farm.particle.cz/home/atlas/ddmf5

During last 2 weeks we received 1.3 TB of AOD using DQ2:

select sum(filesize)/1000000000 as size_GB from Cns_file_metadata m where owner_uid=258 and  date_format(from_unixtime(ctime),'%Y-%m')="2007-10" group by owner_uid;
+---------+
| size_GB |
+---------+
| 1320.68 |
+---------+

(transfers are done using 1 proxy which is mapped to uid=258) All these files are AODs:

select m.name, m.filesize, date_format(from_unixtime(m.ctime),'%Y-%m-%d') as created from Cns_file_metadata m where owner_uid=258 and  date_format(from_unixtime(m.ctime),'%Y-%m')="2007-10" and filesize>0 and name not like '%AOD%';
Empty set (0.57 sec)

October 12

No more space on ATLAS dedicated pools:

POOL atlaspool3 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1308,1311,1307,1309,1310 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.82T FREE 696.00k (  0.0%)
  goliasx98.farm.particle.cz /mnt/array3 CAPACITY 1.82T FREE 696.00k (  0.0%)

Now ATLAS is using non dedicated pool, which will be filled very soon, there are only 173 GB free:

POOL dpmpool4 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 0 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 642.61G FREE 173.31G ( 27.0%)
  golias100.farm.particle.cz /mnt/array3 CAPACITY 642.61G FREE 173.31G ( 27.0%)

I asked to remove write permission for ATLAS to dpmpool4, done by Tomas.

September 11

Usage of atlaspool3 increased, but still we have some place there

POOL atlaspool3 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1308,1311,1307,1309,1310 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.82T FREE 1.39T ( 76.3%)

August 27

atlaspool2 is full, atlaspool3 is now used:

POOL atlaspool2 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1308,1311,1307,1309,1310 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 3.64T FREE 803.45k (  0.0%)
  goliasx98.farm.particle.cz /mnt/array1 CAPACITY 1.82T FREE 384.43k (  0.0%)
  goliasx98.farm.particle.cz /mnt/array2 CAPACITY 1.82T FREE 419.02k (  0.0%)


POOL atlaspool3 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1308,1311,1307,1309,1310 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.82T FREE 1.64T ( 90.4%)
  goliasx98.farm.particle.cz /mnt/array3 CAPACITY 1.82T FREE 1.64T ( 90.4%)

August 18

Soon we will run out of disk space for ATLAS:

POOL atlaspool2 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1307 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 3.64T FREE 351.01G (  9.4%)
  goliasx98.farm.particle.cz /mnt/array1 CAPACITY 1.82T FREE 167.24G (  9.0%)
  goliasx98.farm.particle.cz /mnt/array2 CAPACITY 1.82T FREE 183.77G (  9.9%)

How many test files from ddmf4 we got? I do not see any. All new files are AOD's.

August 14

Published available space did not change:

lcg-infosites --vo atlas se |grep 'Avail\|golias'
Avail Space(Kb) Used Space(Kb)  Type    SEs
578090000       21953552172     n.a     golias100.farm.particle.cz
1055023648      2752216544      n.a     golias26.farm.particle.cz

although now we have only 323+337 of free space on goliasx98.farm.particle.cz /mnt/array1,2. RT ticket opened.

August 8

Cleanup DPM database from files stored on a DPM pool node goliasx34. This machine was used to connect HP entry level storage, but it was unstable and after several service interventions returned to a supplier (now we still use it, but disk boxes directly attached to goliasx98, not via iSCSI). 19 files were still registered in the DPM db. Cleaned using:
./dpns-cleanup --dpns_host_name golias100.farm.particle.cz --server goliasx34.farm.particle.cz


Status of DPM: We have 578 GB of free space for ATLAS:

lcg-infosites --vo atlas se |grep 'Avail\|golias'
Avail Space(Kb) Used Space(Kb)  Type    SEs
578090000       21953552172     n.a     golias100.farm.particle.cz
1055023648      2752216544      n.a     golias26.farm.particle.cz

Status of all pools:

dpm-qryconf
POOL atlaspool2 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1307 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 3.64T FREE 1.05T ( 28.8%)
  goliasx98.farm.particle.cz /mnt/array1 CAPACITY 1.82T FREE 530.08G ( 28.5%)
  goliasx98.farm.particle.cz /mnt/array2 CAPACITY 1.82T FREE 543.72G ( 29.2%)
POOL atlastmp DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1307 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 0 FREE 0 (  0.0%)
  goliasx98.farm.particle.cz /mnt/array3 CAPACITY 1.82T FREE 1.82T (100.0%) RDONLY
POOL augerpool1 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1078 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.82T FREE 1.75T ( 96.2%)
  goliasx98.farm.particle.cz /mnt/array5 CAPACITY 1.82T FREE 1.75T ( 96.2%)
POOL calicepool1 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 2691 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.36T FREE 1.36T (100.0%)
  goliasx98.farm.particle.cz /mnt/array4 CAPACITY 1.36T FREE 1.36T (100.0%)
POOL d0pool1 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 2689 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.82T FREE 1.82T (100.0%)
  goliasx98.farm.particle.cz /mnt/array6 CAPACITY 1.82T FREE 1.82T (100.0%)
POOL dpmpool1 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 1307 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 2.93T FREE 42.40G (  1.4%)
  golias100.farm.particle.cz /mnt/array1/ CAPACITY 1.97T FREE 2.26G (  0.1%)
  golias100.farm.particle.cz /mnt/array2/ CAPACITY 916.71G FREE 813.59M (  0.1%)
  golias100.farm.particle.cz /nbd_1 CAPACITY 64.75G FREE 39.34G ( 60.8%)
POOL dpmpool4 DEFSIZE 1024.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 0 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 642.61G FREE 578.09G ( 90.0%)
  golias100.farm.particle.cz /mnt/array3 CAPACITY 642.61G FREE 578.09G ( 90.0%)
POOL starpool1 DEFSIZE 50.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 2696 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 9.61G FREE 9.09G ( 94.6%)
  golias100.farm.particle.cz /mnt/star_test/ CAPACITY 9.61G FREE 9.09G ( 94.6%)
POOL alicepool1 DEFSIZE 200.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 7.0d DEFPINTIME 2.0h MAX_LIFETIME 1.0m MAXPINTIME 12.0h FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GIDS 2688 S_TYPE - MIG_POLICY none RET_POLICY R
                              CAPACITY 1.61T FREE 1.53T ( 94.9%)
  cl5.ujf.cas.cz /raidRB CAPACITY 1.61T FREE 1.53T ( 94.9%)

August 1

Cleanup of the DPM database from the files lost on the se2 server. I used program dpns-cleanup provided by Lana. I ran it as root on golias100 in a directory /root/chudoba/cleanup/output.
./dpns-cleanup --dpns_host_name golias100.farm.particle.cz --server se2.farm.particle.cz
Log is in the file RemovinEntriesFromTheNSDB.log (later renamed to RemovinEntriesFromTheNSDB.log_se2_cleanup). 24414 files were deleted from the db (not from the disk! - files were lost due to se2 crash):
grep deleting  RemovinEntriesFromTheNSDB.log_se2_cleanup |grep successf |wc -l

-- JiriChudoba - 08 Aug 2007