-- DanicaStojiljkovic - 01 Jul 2009

Automatic Savannah TAsk Submission (ASTAS)

This is a set of python scripts suitable for bulk submission of savannah tasks, bugs or patches from command line interface. It is based on python scripts developed by SA3 for submitting of savannah bugs/patches.

It consists of 3 scripts:

• savannah_my.py
• astas_conf_creator.py
• astas_prepare_xmls.py

Flow chart of task submission of glite deployment tasks with ASTAS:

You can download the tarball at putlink. After you unpack it make sure that

python

and

python/libs/lib

are in your

PYTHONPATH

savannah_my.py

This script is used for submitting, cloning, editing and updating savannah requests (tasks, bugs or patches). It is based on savannah CLI developed by SA3 for managing bugs and patches. It is slightly modified in order to allow managing tasks, and a few more features are added (XML template creation, and bulk submission of tasks)

Script to interface to savannah tracking system

Savannah CLI 

Usage:  savannah_my

    -h --help          Show this help
    -l --login         Allows the user to explicitly log in Savannah
    -V --verbose       Activate progress messages

  Mandatory parameter:
    -t --tracker       Savannah tracker [patch,bugs,task]
    -o --operate       Operations (see below for supported operations)

  Optional parameter (depending on operation):
    -i --item          Savannah item number
    -n --name          Savannah parameter name
    -v --value         New value/comment
    -g --group         Savannah group (for gLite this is jra1mdw)
    -f --file          File containing submit information in xml format
    -a --att_tracker   Tracker of the new attached item [patch, bugs, task]
    -d --dir           Path containing all xmls to be submitted with submit_all

          Supported operations are:

                 Operation                                       | required options
                 ------------------------------------------------------------------
                 attach             Attach item v (tracker       | [i,a,v]
                                    specified with a) to item i  |
                 comment            Add a comment                | [i,v]
                 set                Set value                    | [i,n,v]
                 list               List available values        | [i]
                 list_dependencies  List the dependencies        | [i]
                 get                Get a parameter              | [i,n]
                 get_available      Get available values         | [i,n]
                                    for selectbox                |
                 clone              clone (priority not cloned)  | [i,g]
                 submit             submit a new item            | [g,f]
                 submit_all         submit multiple new items    | [g,d]
                 xml_template       create template for XML file | [g,f]
                                    used with 'submit' option    |

For submitting new item, this script takes an XML file as an input. This file should contain all mandatory fields for submitting savannah request, and optional fields are of course allowed.

Here is a small example

<submit>
        <planned_starting_date_dayfd >1</planned_starting_date_dayfd>
        <planned_starting_date_monthfd >7</planned_starting_date_monthfd>
        <planned_starting_date_yearfd>2009</planned_starting_date_yearfd>
        <planned_close_date_dayfd>10</planned_close_date_dayfd>
        <planned_close_date_monthfd>7</planned_close_date_monthfd>
        <planned_close_date_yearfd>2009</planned_close_date_yearfd>
        <privacy >1</privacy>
        <assigned_to >3221</assigned_to>
        <summary>SUBJECT</summary>
        <details>DETAILS</details>
        .........
</submit>

The same script is modified so it can be used to create the template of the XML file. It can be created automatically and filled in with default values

        python savannah_my.py -o xml_template -g <project_name_in_savannah> -t [task | bug | patch] -f <output_file.xml>

You can interactively fill in this template if you use --interactive option:

        python savannah_my.py -o xml_template -g <project_name_in_savannah> -t [task | bug | patch] -f <output_file.xml>

You will be asked for the value of each field and possible options will be shown where available (where dropdown box is used in savannah page).

When you have your final XML file that is ready to be submitted, use the following command:

        python savannah_my.py -o submit -g <project_name_in_savannah> -t [task | bug | patch] -f <input_file.xml>

If you want to submit multiple files described in multiple XML file, use

        python savannah_my.py -o submit_all -g <project_name_in_savannah> -t [task | bug | patch] -d <dirName>

astas_conf_creator.py

This script will generate configuration file used to produce XML files for bulk submision to savannah.
It will:
   * query savannah database for patches included in an update (production or pps)
   * determine which services are affected by an update
   * query PPS Registry to determine which sites are involved
     in testing activity (pre-deployment, preview or early adopters???)
   * create config XML file where it will write those information in systematic way.
     This confgig file is to be used as input for astas_prepare_xmls.py script


    Options:
      -h --help         Print this help and exit
      -p --PPS          Set this flag if update is in pre-production area.
          Default value: unset (search for production updates)
      -r --release      glite release number (3.1.0 or 3.2.0).
          Default value: "3.1.0"
      -u --update       Update number
          Default value: "" - catches all updates
      -f --file         name of the output configuration file
          Default value: "conf.xml"
      -a --activity     Activity code
                            1 - Pre-deployment testing
                            2 - Release testing
                            3 - Pilot Service in Production
                            4 - Pilot Service in PPS
                            10 - Staged roll-out
          Default value: "1"

astas_prepare_xmls.py

This script will use configuration file produced with astas_conf_creator and a template file in order to produce
list of xml files, ready for submission with savannah_my script.


      -h --help         Print this help and exit

    Mandatory options:
      -d --dir          Directory where the output XML files will be stored
      -c --config       configuration file name
      -t --template     template file name
    Optional:
      -s --start        planned starting date
                        Default value: today
      -D --days         due time for task completition, (number of days)
                        Default value: 1
      -e --effort       estimated number of hours needed for task completition
                        Default value: 4

Template variables

While building template which will later be used with astas_prepare_xmls script, you can use template variables that will later on be populated by this script. There are several built in (python style) variables that you can use:

%(updateName)s - Complete update name, like glite 3.1.0 PPS Update 50

%(release)s - The release number, like 3.1.0

%(updateNumber)s - Update number

%(serviceName)s - name of the service that this task affects

%(metapackages)s - comma separated list of affected metapackages

%(squadName)s - Real name of the site that the task is assigned to

%(patches)s - comma separated list of patches that are affecting this service in this update

%(reconf)s - does the service need reconfiguration ("yes" or "no")

%(restart)s - does the service need restart ("yes" or "no")

%(startDate)s - date of task due start

%(endDate)s - date of scheduled task completion

%(hours)s - estimated effort needed for task in number of hours

When running script astas_prepare_xmls, some of these values are read from the "update" node of conf file (like release, update name,updateNumber). Others are read from "service" nodes (serviceName, patches, restart, reconf, squadName). startDate and endDate are generated by the script. It always match the start and end date of a task. By default it is today for startDate and tomorrow for endDate, but it can be overridden by astas_prepare_xmls options --start and --days, and --effort option will replace the default value "4" of variable %(hours) with the one you specify.

There is also a way to use custom variable names in templates. If anywhere in the template xml, you write %(my_custom_variable)s , that value will be replaced by the value specified as attribute in node "update" of you config file. For example:

<update name="3.2.0 PPS Update 06"
         release="3.2.0"
         updateNumber="06"
         patches="3074, 3079, 3097, 3106, 3112, 3119, 3159, 3172, 3206"
         my_custom_variable1="Hello"
         my_custom_variable2="World" 
>
..........

You can edit config file manually to include values of your variables, but you can also populate this field by using --keywords option of astas_conf_creator. The sintax for this option (in the case described above is)

--keywords my_custom_variable1="Hello":my_custom_variable2="World"

Note: Having keywords in config file that are not used in template is fine, but other way round will produce an error

• ASTAS-flowchart.pptx: Flow chart of task submission of glite deployment tasks with ASTAS