TWiki> Main Web>TWikiUsers>KarolisEigelis>TestCream (2008-11-24, KarolisEigelis) EditAttachPDF

Test CREAM

Direct submission

UI

Ref: CREAM-GuidePDF

Cream client configuration

Log files will be created only if command executed with "-d" flag (debugging)

Environment variable GLITE_CREAM_CLIENT_CONFIG will be read first, if cream client config file is not specified with executed command.

Cream UI config example: /opt/glite/etc/glite-ce-cream-cli/glite_cream.conf.template 

[kar@ui cream]$ cat $GLITE_CREAM_CLIENT_CONFIG
[
STATUS_VERBOSITY_LEVEL = 2; 

SUBMIT_LOG_DIR="/home/kar/cream/logs";
DELEGATE_LOG_DIR="/home/kar/cream/logs";
STATUS_LOG_DIR="/home/kar/cream/logs";
SUSPEND_LOG_DIR="/home/kar/cream/logs";
RESUME_LOG_DIR="/home/kar/cream/logs";
CANCEL_LOG_DIR="/home/kar/cream/logs";
PURGE_LOG_DIR="/home/kar/cream/logs";
PROXYRENEW_LOG_DIR="/home/kar/cream/logs";
]

JDL_DEFAULT_ATTRIBUTES variables can be specified in ui cream config. 

[
JDL_DEFAULT_ATTRIBUTES = [ 
                            JobType="Normal"; 
                            Type="Job"; 
                         ];

STATUS_VERBOSITY_LEVEL = 2; 

SUBMIT_LOG_DIR="/home/kar/cream/logs";
DELEGATE_LOG_DIR="/home/kar/cream/logs";
STATUS_LOG_DIR="/home/kar/cream/logs";
SUSPEND_LOG_DIR="/home/kar/cream/logs";
RESUME_LOG_DIR="/home/kar/cream/logs";
CANCEL_LOG_DIR="/home/kar/cream/logs";
PURGE_LOG_DIR="/home/kar/cream/logs";
PROXYRENEW_LOG_DIR="/home/kar/cream/logs";
]

Ref: CREAM-JDLPDF

Looking for creamCE 

[kar@ui cream]$ lcg-infosites --vo balticgrid ce --is bdii.grid.vgtu.lt|grep cream
   4       4       0              0        0    ce1.grid.vgtu.lt:8443/cream-pbs-balticgrid

Job Types

  • Normal
  • Parametric
  • MPICH

Normal
Parametric
MPICH
Check what MPI implementations cluster supports 
[kar@ui cream]$ ldapsearch -LLL -x -h ce1.grid.vgtu.lt:2170 -b mds-vo-name=VGTU-CREAM,o=grid|egrep "MPI-START|MPI_SHARED_HOME|MPICH|OPENMPI"
GlueHostApplicationSoftwareRunTimeEnvironment: MPICH
GlueHostApplicationSoftwareRunTimeEnvironment: MPICH-1.2.7p1
GlueHostApplicationSoftwareRunTimeEnvironment: MPICH2
GlueHostApplicationSoftwareRunTimeEnvironment: MPICH2-1.0.4
GlueHostApplicationSoftwareRunTimeEnvironment: OPENMPI
GlueHostApplicationSoftwareRunTimeEnvironment: OPENMPI-1.2.7
GlueHostApplicationSoftwareRunTimeEnvironment: MPI-START
GlueHostApplicationSoftwareRunTimeEnvironment: MPI_SHARED_HOME

Check if submission to CREAM-CE is allowed: 

[kar@ui cream]$ glite-ce-allowed-submission ce1.grid.vgtu.lt
Job Submission to this CREAM CE is enabled

Prepare job input files: 

[kar@ui mpich]$ export DPNS_HOST=se.grid.vgtu.lt
[kar@ui mpich]$ dpns-mkdir -p /dpm/grid.vgtu.lt/home/balticgrid/kar
[kar@ui mpich]$ lcg-cp file://$PWD/mpi-hooks.sh srm://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-hooks.sh
[kar@ui mpich]$ lcg-cp file://$PWD/mpi-start-wrapper.sh srm://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start-wrapper.sh
[kar@ui mpich]$ lcg-cp file://$PWD/pingtest.c srm://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/pingtest.c
[kar@ui mpich]$ dpns-ls /dpm/grid.vgtu.lt/home/balticgrid/kar
mpi-hooks.sh
mpi-start-wrapper.sh
pingtest.c

Set variable for cream ui config: 

[kar@ui mpich]$ export GLITE_CREAM_CLIENT_CONFIG=/home/kar/cream/ui_cream.conf

Prepare JDL file: 

[kar@ui mpich]$ cat mpi-start.jdl 
[     
CpuNumber = 4;
Executable = "mpi-start-wrapper.sh";
Arguments = "pingtest MPICH";
StdOutput = "mpi-start.out";
StdError = "mpi-start.err";

InputSandbox = { 
                 "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/pingtest.c",  
                 "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start-wrapper.sh",     
                 "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-hooks.sh"                   
                };                       

OutputSandbox = { 
                   "mpi-start.err", 
                   "mpi-start.out" 
                }; 
    
OutputSandboxBaseDestURI = "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar";
]

Submit Job: 

[kar@ui mpich]$ glite-ce-job-submit -D 2 -r ce1.grid.vgtu.lt/cream-pbs-balticgrid mpi-start.jdl
https://ce1.grid.vgtu.lt:8443/CREAM605721465

Job's status: 

[kar@ui mpich]$ glite-ce-job-status https://ce1.grid.vgtu.lt:8443/CREAM605721465

******  JobID=[https://ce1.grid.vgtu.lt:8443/CREAM605721465]
        Current Status = [DONE-OK]
        Working Dir    = [[reserved]]
        ExitCode       = [0]
        Grid JobID     = [N/A]
        LRMS Abs JobID = [[reserved]]
        LRMS JobID     = [[reserved]]
        Deleg Proxy ID = [2]
        DelegProxyInfo = [Valid From      : 11/23/08 6:09 PM
                          Valid To       : 11/24/08 6:10 AM
                          Holder Subject : DC=org,DC=balticgrid,OU=vgtu.lt,CN=Karolis Eigelis
                          Holder CA      : DC=org,DC=balticgrid,CN=Baltic Grid Certification Authority
                          
                          VO              : balticgrid
                          AC Issuer       : CN=host/voms.balticgrid.org, OU=eenet.ee, DC=balticgrid, DC=org
                          Attribute       : /balticgrid/Role=NULL/Capability=NULL 
                          ]
        Worker Node    = [compute-0-1.local]
        Local User     = [bg17]
        CREAM ISB URI  = [gsiftp://ce1.grid.vgtu.lt/opt/glite/var/cream_sandbox/balticgrid/DC_org_DC_balticgrid_OU_vgtu.lt_CN_Karolis_Eigelis_balticgrid_Role_NULL_Capability_NULL/CREAM605721465/ISB]
        CREAM OSB URI  = [gsiftp://ce1.grid.vgtu.lt/opt/glite/var/cream_sandbox/balticgrid/DC_org_DC_balticgrid_OU_vgtu.lt_CN_Karolis_Eigelis_balticgrid_Role_NULL_Capability_NULL/CREAM605721465/OSB]
        JDL            = [[ Arguments = "pingtest MPICH"; OutputSandboxBaseDestURI = "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar"; QueueName = "balticgrid"; Executable = "mpi-start-wrapper.sh"; CpuNumber = 4; StdOutput = "mpi-start.out"; OutputSandbox = { "mpi-start.err","mpi-start.out" }; VirtualOrganisation = "balticgrid"; StdError = "mpi-start.err"; BatchSystem = "pbs"; InputSandbox = { "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/pingtest.c","gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start-wrapper.sh","gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-hooks.sh" } ]]
        Type           = [normal]

        Job status changes:
        -------------------
        Status         = [REGISTERED] - [Sun 23 Nov 2008 23:19:17] (1227475157)
        Status         = [PENDING] - [Sun 23 Nov 2008 23:19:17] (1227475157)
        Status         = [RUNNING] - [Sun 23 Nov 2008 23:19:23] (1227475163)
        Status         = [REALLY-RUNNING] - [Sun 23 Nov 2008 23:19:29] (1227475169)
        Status         = [DONE-OK] - [Sun 23 Nov 2008 23:19:50] (1227475190)

        Issued Commands:
        -------------------

        *** Command Name              = [JOB_REGISTER]
            Command Category          = [JOB_MANAGEMENT]
            Command Status            = [SUCCESSFULL]
            Creation Time             = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Scheduling Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Processing Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Execution Completed Time  = [Sun 23 Nov 2008 23:19:17] (1227475157)


        *** Command Name              = [JOB_START]
            Command Category          = [JOB_MANAGEMENT]
            Command Status            = [SUCCESSFULL]
            Creation Time             = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Scheduling Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Processing Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Execution Completed Time  = [Sun 23 Nov 2008 23:19:25] (1227475165)


        *** Command Name              = [PROXY_RENEW]
            Command Category          = [JOB_MANAGEMENT]
            Command Status            = [SUCCESSFULL]
            Creation Time             = [Sun 23 Nov 2008 23:19:29] (1227475169)
            Start Scheduling Time     = [Sun 23 Nov 2008 23:19:29] (1227475169)
            Start Processing Time     = [Sun 23 Nov 2008 23:19:29] (1227475169)
            Execution Completed Time  = [Sun 23 Nov 2008 23:19:36] (1227475176)

Get Job's output: 

[kar@ui mpich]$ lcg-ls srm://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar
/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-hooks.sh
/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start-wrapper.sh
/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start.err
/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start.out
/dpm/grid.vgtu.lt/home/balticgrid/kar/pingtest.c
[kar@ui mpich]$ lcg-cp srm://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start.err file://$PWD/mpi-start.err
[kar@ui mpich]$ lcg-cp srm://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start.out file://$PWD/mpi-start.out
[kar@ui mpich]$ cat mpi-start.out 
Compiling /home/bg17/home_cream_189927535/CREAM189927535/pingtest
mpicc -o /home/bg17/home_cream_189927535/CREAM189927535/pingtest /home/bg17/home_cream_189927535/CREAM189927535/pingtest.c
Successfully compiled /home/bg17/home_cream_189927535/CREAM189927535/pingtest
message size    transfertime            bandwidth
32 bytes                0.000029 sec            1.109570 MB/s
2048 bytes              0.000031 sec            66.884389 MB/s
131072 bytes            0.000858 sec            152.696941 MB/s
8388608 bytes           0.115781 sec            72.452083 MB/s
Executing post hook.

Nodes:
compute-0-1.local
compute-0-1.local
compute-0-0.local
compute-0-0.local

Finished the post hook.

glite-ce-allowed-submission

Check if submission to CREAM-CE is allowed: 
[kar@ui cream]$ glite-ce-allowed-submission ce1.grid.vgtu.lt
Job Submission to this CREAM CE is enabled

glite-ce-delegate-proxy 

[kar@ui cream]$ voms-proxy-init --voms balticgrid
Enter GRID pass phrase:
Your identity: /DC=org/DC=balticgrid/OU=vgtu.lt/CN=Karolis Eigelis
Creating temporary proxy .................................................................. Done
Contacting  voms.balticgrid.org:15000 [/DC=org/DC=balticgrid/OU=eenet.ee/CN=host/voms.balticgrid.org] "balticgrid" Done
Creating proxy ................................. Done
Your proxy is valid until Mon Nov 24 08:10:57 2008

[kar@ui cream]$ glite-ce-delegate-proxy -e ce1.grid.vgtu.lt:8443 2
2008-11-23 20:13:29,855 NOTICE - Proxy with delegation id [2] succesfully delegated to endpoint [https://ce1.grid.vgtu.lt:8443//ce-cream/services/gridsite-delegation]

glite-ce-proxy-renew 

[kar@ui cream]$ glite-ce-proxy-renew -e ce1.grid.vgtu.lt 2
2008-11-23 20:14:50,345 NOTICE - Proxy with delegation id [2] succesfully renewed to endpoint [https://ce1.grid.vgtu.lt:8443//ce-cream/services/gridsite-delegation]

glite-ce-job-submit 

[kar@ui cream]$ glite-ce-job-submit -D 2 -r ce1.grid.vgtu.lt/cream-pbs-balticgrid mpi-start.jdl
https://ce1.grid.vgtu.lt:8443/CREAM324798604

glite-ce-job-status 

[kar@ui mpich]$ glite-ce-job-status https://ce1.grid.vgtu.lt:8443/CREAM605721465

******  JobID=[https://ce1.grid.vgtu.lt:8443/CREAM605721465]
        Current Status = [DONE-OK]
        Working Dir    = [[reserved]]
        ExitCode       = [0]
        Grid JobID     = [N/A]
        LRMS Abs JobID = [[reserved]]
        LRMS JobID     = [[reserved]]
        Deleg Proxy ID = [2]
        DelegProxyInfo = [Valid From      : 11/23/08 6:09 PM
                          Valid To       : 11/24/08 6:10 AM
                          Holder Subject : DC=org,DC=balticgrid,OU=vgtu.lt,CN=Karolis Eigelis
                          Holder CA      : DC=org,DC=balticgrid,CN=Baltic Grid Certification Authority
                          
                          VO              : balticgrid
                          AC Issuer       : CN=host/voms.balticgrid.org, OU=eenet.ee, DC=balticgrid, DC=org
                          Attribute       : /balticgrid/Role=NULL/Capability=NULL 
                          ]
        Worker Node    = [compute-0-1.local]
        Local User     = [bg17]
        CREAM ISB URI  = [gsiftp://ce1.grid.vgtu.lt/opt/glite/var/cream_sandbox/balticgrid/DC_org_DC_balticgrid_OU_vgtu.lt_CN_Karolis_Eigelis_balticgrid_Role_NULL_Capability_NULL/CREAM605721465/ISB]
        CREAM OSB URI  = [gsiftp://ce1.grid.vgtu.lt/opt/glite/var/cream_sandbox/balticgrid/DC_org_DC_balticgrid_OU_vgtu.lt_CN_Karolis_Eigelis_balticgrid_Role_NULL_Capability_NULL/CREAM605721465/OSB]
        JDL            = [[ Arguments = "pingtest MPICH"; OutputSandboxBaseDestURI = "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar"; QueueName = "balticgrid"; Executable = "mpi-start-wrapper.sh"; CpuNumber = 4; StdOutput = "mpi-start.out"; OutputSandbox = { "mpi-start.err","mpi-start.out" }; VirtualOrganisation = "balticgrid"; StdError = "mpi-start.err"; BatchSystem = "pbs"; InputSandbox = { "gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/pingtest.c","gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-start-wrapper.sh","gsiftp://se.grid.vgtu.lt/dpm/grid.vgtu.lt/home/balticgrid/kar/mpi-hooks.sh" } ]]
        Type           = [normal]

        Job status changes:
        -------------------
        Status         = [REGISTERED] - [Sun 23 Nov 2008 23:19:17] (1227475157)
        Status         = [PENDING] - [Sun 23 Nov 2008 23:19:17] (1227475157)
        Status         = [RUNNING] - [Sun 23 Nov 2008 23:19:23] (1227475163)
        Status         = [REALLY-RUNNING] - [Sun 23 Nov 2008 23:19:29] (1227475169)
        Status         = [DONE-OK] - [Sun 23 Nov 2008 23:19:50] (1227475190)

        Issued Commands:
        -------------------

        *** Command Name              = [JOB_REGISTER]
            Command Category          = [JOB_MANAGEMENT]
            Command Status            = [SUCCESSFULL]
            Creation Time             = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Scheduling Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Processing Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Execution Completed Time  = [Sun 23 Nov 2008 23:19:17] (1227475157)


        *** Command Name              = [JOB_START]
            Command Category          = [JOB_MANAGEMENT]
            Command Status            = [SUCCESSFULL]
            Creation Time             = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Scheduling Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Start Processing Time     = [Sun 23 Nov 2008 23:19:17] (1227475157)
            Execution Completed Time  = [Sun 23 Nov 2008 23:19:25] (1227475165)


        *** Command Name              = [PROXY_RENEW]
            Command Category          = [JOB_MANAGEMENT]
            Command Status            = [SUCCESSFULL]
            Creation Time             = [Sun 23 Nov 2008 23:19:29] (1227475169)
            Start Scheduling Time     = [Sun 23 Nov 2008 23:19:29] (1227475169)
            Start Processing Time     = [Sun 23 Nov 2008 23:19:29] (1227475169)
            Execution Completed Time  = [Sun 23 Nov 2008 23:19:36] (1227475176)

glite-ce-job-suspend

Possible only on these job states:

  • Running
  • Idle

glite-ce-job-resume

Possible only on this job state:

  • Held

glite-ce-job-cancel

Possible only on these job states:

  • Pending
  • Idle
  • Running
  • Really-Running
  • Held

glite-ce-job-purge

Possible only on these job states:

  • Registered
  • Done-OK
  • Done-Failed
  • Aborted
  • Cancelled

glite-ce-job-list 

[kar@ui cream]$ glite-ce-job-list ce1.grid.vgtu.lt
https://ce1.grid.vgtu.lt:8443/CREAM064670125
https://ce1.grid.vgtu.lt:8443/CREAM070084752
https://ce1.grid.vgtu.lt:8443/CREAM122863713
https://ce1.grid.vgtu.lt:8443/CREAM147794363
https://ce1.grid.vgtu.lt:8443/CREAM168366289
https://ce1.grid.vgtu.lt:8443/CREAM183387815
https://ce1.grid.vgtu.lt:8443/CREAM194272022
https://ce1.grid.vgtu.lt:8443/CREAM255920954
https://ce1.grid.vgtu.lt:8443/CREAM260357786
https://ce1.grid.vgtu.lt:8443/CREAM282830659

creamCE notes

CE configured with YAIM 

/opt/glite/yaim/bin/yaim -c -s site-info-VGTU-CREAM.def -n MPI_CE -n creamCE -n TORQUE_server -n TORQUE_utils -n MON -n BDII_site

WN configured with YAIM 

/opt/glite/yaim/bin/yaim -c -s site-info-VGTU-CREAM.def -n MPI_WN -n glite-WN -n TORQUE_client -n TORQUE_utils

Torque server cosmetics to run MPI jobs on multi-cores; N = total number of CPUs 

qmgr -c 'set server resources_available.nodect = N'

MPI-START

site-info.def for default mpi-start: 

MPI_MPICH_ENABLE="yes"
MPI_MPICH2_ENABLE="yes"
MPI_OPENMPI_ENABLE="yes"
MPI_MPICH_PATH="/opt/mpich-1.2.7p1/"
MPI_MPICH_VERSION="1.2.7p1"
MPI_MPICH2_PATH="/opt/mpich2-1.0.4/"
MPI_MPICH2_VERSION="1.0.4"
MPI_OPENMPI_VERSION="1.2.7"
MPI_OPENMPI_PATH="/usr/lib/openmpi/1.2.7-gcc"
MPI_SHARED_HOME="yes"
MPI_SSH_HOST_BASED_AUTH="no"

More info about MPI-START here.

-- KarolisEigelis - 19 Nov 2008

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r8 - 2008-11-24 - KarolisEigelis
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright &© 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
or Ideas, requests, problems regarding TWiki? use Discourse or Send feedback