File transfers and how to deal with failed transfers

In DIRAC there are two types of data transfers:

• upload from the WN to the SE
• FTS jobs for files replication

when a transfer is not successfully completed, then the client create a failover request, that will trigger a new transfer. In this way it's sure that the transfer eventually will take place. But up to now (Jan. 2011) the failed transfer was not cleaned! so the incomplete file was left on the SE.

This gave origin to corrupted files on some SE, visible in the namespace. But NOT registered in the LFC, because DIRAC client proceeds to register a replica only after the transfer has been completed.

In these cases, the SE admins asked to LHCb to remove the files from the storage.

In Jan 2011 Ricardo has put a fix for this to remove the file on the SE. From the WN: in SRM2StorageElement lines 693 and following , it's in the head of SVN. (Jan 2011). And for FTS transfers: there we use the " -o overwrite if exists" option. This is implemented and tagged, ready to be included in the hot release under preparation by Joel.

-- ElisaLanciotti - 21-Jan-2011