Learning Git  For ATLAS collaboration users Rate This Page 12345 Score: 0, My vote: 0, Total votes: 0 Responsible: WillButtinger

Recommended global settings

here

The instructions below will use acm, which is available on the bash shell environment only. So please ensure you are using bash for this tutorial.

Basic concepts

• upstream: this is the centrally maintained copy of the athena repository. If your code gets into here, it's visible for everyone. It lives on the gitlab servers.
• origin: this is your fork of the upstream repository. It lives on the gitlab servers. Think of it like a staging area for your developments. This copy allows gitlab to show on the web interface what changes you have made to the central code when you make requests to have your developments put into upstream.
• local: This is the copy of the repository on your actual machine you are developing code. You will push (a command-line procedure) developments to the origin repository, then merge request (a gitlab web interface procedure) those developments into upstream.

You might discover that creating the local repository can take quite a while if you always create it by cloning the origin repository - this will be true for large repositories like the ATLAS offline software repo (athena). Therefore in the workflow described later on this page we will also involve a fourth repo:

• cache: This is another local copy of the repository, which you will make by cloning a gitlab repository. You will create this once, then your many local repositories can be made by cloning from this repository, which is faster than cloning from gitlab every time.

FYI, gitlab calls a repository a "project".

In the examples below we also make use of a lot of cmake. Here it is important to standard that:

• TestArea: This is your build directory. It was where your code gets compiled. You will get an environment variable called $TestArea provided to you
• SourceArea: This is your source directory. It is where your source code lives. the acmSetup command will provide an environment variable $SourceArea that points to this location. If you do not use acm then you will not have this environment variable.

1. Creating your origin repository

described here

Over time your origin repository can become stale in some respects (e.g. branches that were alive at the time you forked, but have died since then). If you are a git ninja you can learn how to prune your origin repository by issuing commands from your local repository (or through the gitlab web interface). But an alternative is to, every now and again just delete your origin repository on gitlab (google how to delete a gitlab project). Note that you should only do this if you haven't got any pending merge requests or open issues.

2. Creating your cache repository

As discussed above, it's a good idea when working with large repositories to create a local cache repository, which we will clone our local repositories from. So here's how we create it from our origin repository:

lsetup git
cd $HOME
mkdir gitcache
cd gitcache
git atlas init-workdir https://:@gitlab.cern.ch:8443/$USER/athena.git

This can take some time, but only needs doing once. If your gitlab username is different to your local username, replace $USER with your gitlab username.

Note that you also can do a full git clone of the repository if you want, rather than the above 'sparse clone' ... i.e. you can alternatively do:

#For people who prefer a full clone of the repo for their 'cache'
lsetup git
cd $HOME
mkdir gitcache
cd gitcache
git clone https://:@gitlab.cern.ch:8443/$USER/athena.git

3. Creating your local repository

I advise that you create your local repository inside the source directory of one of your projects. See here for how I advise you set up a new project area, including setting up a particular release.

Now, to actually create your local repository you have to make a choice, are you going to work with sparse checkout or sparse build:

• Sparse checkout: Only the packages you want to compile will be checked out. Everything in your source directory will be compiled
• Sparse build: All the offline code is checked out. Only the packages you specify with a text file will be compiled.

I will show how to work with both in this section, but first we have to set up a release ...

Setting up a release

First you should set up a release. I will assume you are also starting a new 'project' so you need to create a source directory (the $SourceArea) and a build directory (the $TestArea). Here's how (remember, only available in bash at the moment):

mkdir MyProject
cd MyProject
mkdir source build run
cd build
acmSetup --sourcearea=../source AthAnalysis,21.2,latest

(Note, technically you dont even need the "--sourcearea=../source" part because that is the default value. sourcearea is/will be the location of your source code).

Whenever you come back on a new terminal to work on your project you just need to do:

cd MyProject/build
acmSetup

See equivalent without using acm ... it's a lot more typing! ... hide ...

mkdir MyProject
cd MyProject
mkdir source build run
cd build
asetup AthAnalysis,21.2,latest,here
mv CMakeLists.txt ../source/
cmake ../source
source */setup.sh

cd MyProject/build
asetup
source */setup.sh

Optional: Working with sparse build (slower than sparse checkout)

Clone the athena project like this:

acm clone_project athena $HOME/gitcache/athena

By default, acm will not compile anything from the athena directory. Suppose you wanted to compile the package PileupReweighting in athena, you can add it to the compilation like this:

acm add_pkg athena/PhysicsAnalysis/AnalysisCommon/PileupReweighting

and then compile it with:

acm compile

See equivalent without using acm (again, lots more typing!!) ... hide ...

cd $TestArea
cat - > package_filters.txt << EOF
- athena/.*
+ .*
EOF
cd ../source
git clone https://:@gitlab.cern.ch:8443/$USER/athena.git
cd athena
git remote add upstream https://:@gitlab.cern.ch:8443/atlas/athena.git
git fetch upstream
cd $TestArea

cmake -DATLAS_PACKAGE_FILTER_FILE=$TestArea/package_filters.txt $TestArea

+ athena/.*PileupReweighting
- athena/.*
+ .*

cmake $TestArea
cmake --build $TestArea

Working with sparse checkout

Using sparse checkout with acm is very simple. Just replace the clone_project command above with sparse_clone_project. You will see the clone runs much quicker and only a tiny part of the athena repository is checked out. Packages will appear in the local repository as you call the add_pkg command.

acm sparse_clone_project athena $HOME/gitcache/athena
acm add_pkg athena/PhysicsAnalysis/AnalysisCommon/PileupReweighting
acm compile

Note that acm can also add packages from an entire subdirectory, e.g.

acm add_pkg athena/PhysicsAnalysis/AnalysisCommon/.\*

4. Creating a development branch

By this stage you should have a testarea (build directory) and a sourcearea (source directory) in place, with the source area containing a local copy of the athena repository, which is under source/athena.

Before you develop for athena, you should set up a development branch, with the relevant starting point being the branch you set up with asetup. Do this:

cd $TestArea/../source/athena
git checkout -b MyDevelopmentFor-$AtlasBuildBranch upstream/$AtlasBuildBranch

At any point you can list your development branches with the git branch -vv option, e.g. for me this shows:

bash-3.2$ git branch -vv
* MyDevelopmentFor-21.2 4084edd [upstream/21.2] Update AnalysisTop/version.txt ....
  master                1c4aaf2 [origin/master] Merge branch 'cherry-pick-01c67090' into 'master'

It says I'm currently developing a branch that is 'tracking' the upstream/21.2 branch (i.e. the 21.2 of the upstream copy of the repository).

5. Developing code and committing to the local repository

Now you would go ahead and make changes to the code. For example, I could create a new file:

cd $TestArea/../source/athena
touch foo

Now you need to stage your changes for a commit. You stage files with the git add command. E.g.

git add foo

You can see the status of your staged and unstaged changes with git status. You then commit with (here's example of a commit message):

git commit -m "made a dummy file"

Now your change is pushed into the local repository. You can see that you have gotten ahead of the upstream repository with git branch -vv, e.g. which shows now for me:

* MyDevelopmentFor-21.2 2bc2929 [upstream/21.2: ahead 1] made a dummy file

So I'm ahead of the upstream repository. While you were developing your code, commits might have been merged into to the upstream repository. To check this, you need to fetch the information about the upstream repository, then you can check the branch status again:

git fetch upstream
git branch -vv

If it shows you are behind on commits, you can take a look at the commits that you are behind on with:

git log HEAD..@{u}

If you see you have gotten behind by any commits, you might like to rebase your development branch on top of that latest state of the upstream branch. To do this, you would just do (remember you must have done the fetch first):

git rebase @{u}

In the above command @{u} is interpreted by git to mean "replace this with whatever the upstream tracking branch is" (i.e. the branch name in square brackets in the git branch -vv command).

Note that the rebase will only work if you have 'dirty' files, i.e. files you have modified but not committed the changes for. You will be told you can try to "stash" the changes (with git stash) before rebasing, and then unstash the changed (with git stash pop) after, and hopefully your changes are not in conflict with the rebase changes. You can accomplish this all in one move with:

git rebase @{u} --autostash

6. Pushing to origin repository and starting a merge request

When you are ready to publish your developments, push them to your origin repository:

git push origin HEAD[:SomeBranchName]

This means "push my current branch in its current state to the origin repository". The optional part is if you want the name of the branch to be different on the origin repository (you can always rename the current local branch with git branch -m SomeBranchName if you like). Usually you don't bother renaming it. If you don't rename it, any subsequent push can then be achieved simply with:

git push origin HEAD

Then you go to the gitlab webpage of the origin repository (your fork of the athena project), and create a merge request for that branch. Remember to select the branch you want the merge request to go into in the upstream repository (it defaults to the master branch, which isn't always what you want).

7. Advanced Topic #1: Modifying a base project

Suppose I want to make a modification to a base project, e.g. I am working with the AthDerivation project but I want to compile (some parts of) it against a modified version of the AthDerivationExternals project. Here's how to do it.

You should set up your main development project as usual, but then you should add base project code as a subproject in your SourceArea. The source for AthDerivationExternals is in the atlas/atlasexternals repo on gitlab, so do:

mkdir source build
cd build
acmSetup --sourcedir=../source AthDerivation,21.2,latest
acm clone_project atlasexternals atlas/atlasexternals.git

Now you want to modify the code for that project however you wish. Then you need to compile that project. You do this in a new build directory, and then source the setup script, which will ensure that project is picked up by your environment:

cd $TestArea/..
mkdir buildext
cd buildext
cmake $SourceArea/atlasexternals/Projects/AthDerivationExternals/
make
source */setup.sh

Now all you have to do is ensure your main TestArea picks up this new project, which you do like this:

acm find_packages

Then you should be ready to compile your AthDerivation code and it will pick up your local version of the base project, e.g. the usual:

acm sparse_clone_project athena
acm add_pkg athena/PhysicsAnalysis/AnalysisCommon/ReweightUtils
acm compile

8. Advanced Topic #2: Adding a new external

You can add a new external with a package like https://github.com/lawrenceleejr/Ext_RestFrames ... check it out as usual:

acm clone_project https://github.com/lawrenceleejr/Ext_RestFrames.git

ideally the externals.cmake would instead have been called FindRestFrames.cmake and then in your top-level project file you would just need to have:

list( INSERT CMAKE_MODULE_PATH 0 ${CMAKE_SOURCE_DIR}/Ext_RestFrames )

then in your package where you want to use this external you do the usual:

find_package( RestFrames )

...

INCLUDE_DIRS ... ${RESTFRAMES_INCLUDE_DIRS}
LINK_LIBRARIES ... ${RESTFRAMES_LIBRARIES}

but you'll also need to explicitly declare the dependency, so ensure the external project is built first: add_dependency( MyPackageLib RestFrames )

99. Other common git tasks

Compare two different releases

There are various depths of information you can obtain about a difference between two releases. All these commands are called from inside your athena local repo directory.

They all make use of git log command, with the following essential syntax (ignoring all options):

git log <branch> <from>..<to>

where <branch> is the name of the branch you want to compare in, e.g. upstream/21.2, and <from>..<to> are the git tags (i.e. releases) you want to compare, e.g. release/21.2.3..release/21.2.4 to compare between the 21.2.3 and 21.2.4 releases. This also works with nightlies. You can see all the tags available with (optional pattern argument to filter):

git tag -l [<pattern>]

git log --pretty=oneline --name-only upstream/21.2 \
   release/21.2.3..release/21.2.4

git log upstream/21.2 release/21.2.3..release/21.2.4 \
    -- PhysicsAnalysis/AnalysisCommon/PileupReweighting

git log -p upstream/21.2 release/21.2.3..release/21.2.4 \
    -- PhysicsAnalysis/AnalysisCommon/PileupReweighting

git documentation on git log