Migration To Git

Recommended Reading

• Official Gerrit introduction: https://gerrit-review.googlesource.com/Documentation/intro-quick.html.
• Gerrit tutorial: http://www.vogella.com/tutorials/Gerrit/article.html

Instructions for Simantics Developers

Gerrit

1. Request a Gerrit account. If you already have a SVN account, it should work also in Gerrit.
2. Generate SSH Public/Private key. If you already have one available, skip this step
   • Windows
     • Install PuTTY (msi) or download only PuTTYgen (exe) (http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html)
     • Run PuTTYgen and generate a key
       • For Type of key to generate, select SSH-2 RSA
       • Press Generate button
       • Move your mouse over the window until the progress bar is finished
       • Write a Key passphrase to protect your private key in case your local machine is compromised.
       • Save the generated public and private keys in C:\Users\{username}\.ssh as follows:
         • Start cmd.exe and create the target directory using mkdir .ssh
         • Save private key using Conversions -> Export OpenSSH Key. The name id_rsa is preferred because it will work out of the box with all tools.
         • At the top of the window, there is a text field labeled Public key for pasting into OpenSSH authorized_keys. Copy the public key from there and save it into a text file at C:\Users\{username}\.ssh\id_rsa.pub
   • Linux
     • Generate public and private keys using ssh-keygen -t rsa
3. Go to https://gerrit.simantics.org/r/ and log in with your Gerrit username/password
4. Select your name in the top-right corner and press Settings
5. Go to section SSH Public Keys
6. Open the file containing the generated public key in a text editor and copy-paste its contents to the box in Gerrit and and press Add
7. Add and verify your contact information (name + e-mail). This will allow administrators to add you to the necessary groups to get commit access:
   • Go to section Contact Information and enter your Full Name there
   • Add a Preferred Email address using the button Register New Email
   • Wait for the authorization mail to arrive. It will look something like the following and contain a longish link that tends to end with == characters. Note that outlook tends to leave the trailing == characters out of the link it generates automatically. That's why it is recommended to copy the full URL from the mail into your browser to verify your e-mail address.

Subject:  [Gerrit Code Review] Email Verification

Welcome to Gerrit Code Review at www.simantics.org.

To add a verified email address to your user account, please
click on the following link while signed in as .....:

https://gerrit.simantics.org/r/#/../...............................................................................==

If you have received this mail in error, you do not need to take any
action to cancel the account.  The address will not be activated, and
you will not receive any further emails.

If clicking the link above does not work, copy and paste the URL in a
new browser window instead.

This is a send-only email address.  Replies to this message will not
be read or answered.

8. Finally, ask User:Tuukka Lehtonen to add you to the necessary groups and wait for this to happen.

Eclipse

The following instructions are written for Eclipse Neon. There are some differences if you are using Mars (4.5) or Oxygen (4.7).

1. Go to preferences (General / Network Connections / SSH2) and ensure that SSH2 home points to the directory where you saved the private key.
2. Go to preferences (Team / Git / Configuration) and ensure that the tab User Settings has the following two setting: user.name and user.email. The specified name and email must match the values you've put in Gerrit to be able to push changes.
   
   • Also, git has a setting called core.autocrlf which is used for controlling the way git enforces line feeds to be encoded in a certain way in commits. Read the documentation and also this to learn more. We recommend to set it to true in Windows in order to avoid committing CRLFs to repository.
   • If you have a git command line client installed you can use the command git config --list --show-origin to see your current global settings and find out exactly where your settings are coming from.
3. Open Git perspective
4. Press button Clone a Git Repository and add the clone to this view (third button in the toolbar of Git view)
5. Select Clone URI (Don't select Gerrit, it doesn't work very well at the moment)
6. Go back to Gerrit. Press Projects in the top-left menu and List in the submenu.
7. Find a row named simantics/platform and press (gitweb) link at the last column.
8. Copy the URI starting with ssh: (ssh://{username}@www.simantics.org:29418/simantics/platform.git) and paste it to URI field in Clone Git Repository dialog.
9. This should automatically fill the other fields.
10. Press Next. If your private key has a passphrase, Eclipse now asks it.
11. Select all branches to be cloned (default) and press Next.
12. Give a local directory where the repository is cloned to and press Finish.
13. After the cloning is finished, select the platform repository in the Git Repositories view and select Import Projects... from context menu.
14. Press Next, select all projects but org.simantics.root and press Finish.
15. Expand the tree view of platform, expand Branches under it and Local under it. Open context menu on master branch and select Configure Branch.... Select Rebase checkbox and press Ok.

The repository you cloned contains the target platform definition that needs to be set in order for the imported Simantics codebase to compile. You can find the required target platform definition file from org.simantics.sdk.build.targetdefinition/org.simantics.sdk.build.targetdefinition.target. Open it in the target definition editor and set it as the active target platform by pressing the link Set as Target Platform at the top right corner of the editor.

Congratulations, you've now set up your workspace for developing the Simantics codebase in Eclipse!

Updating your local repository and working space

1. Open context menu on platform in the Git perspective and select Pull.

Committing your changes back to shared repository

1. Select platform in the Git perspective and open Git Staging view.
2. Unstaged Changes shows all files you have modified.
3. Drag&drop those changes you want to commit to the box Staged Changes.
4. Write a commit message
5. Ensure that Add Change-Id button is toggled down and the commit message contains a row starting Change-Id:.
   • This is required by Gerrit to be able to bind commits to specific reviews. Gerrit will replace the default zero-valued Id with a generated one when the change is pushed for review.
6. Press Commit and Push...
7. This opens a dialog that asks the Gerrit Branch and a Topic. Topic can be added optionally to tie multiple changes together.
   • Example of a topic: https://gerrit.simantics.org/r/#/q/topic:svn
8. Go to Gerrit. In the top-left menu select My and select Changes.
9. Your change can be seen now in the section Outgoing reviews.
10. Click the change. This opens a review page.
11. Ask someone to review your change. This can be done also in Gerrit by pressing Add... button in the right of the Reviewers field.
12. If you need to make modifications to your changes based on reviews, you'll need to Amend the earlier changes to send a new patch set to Gerrit. In this case, ensure that the Amend (Edit Previous Commit) button is toggled down and you should be able to modify the earlier commit message and push the amended changes.
13. When somebody has reviewed and accepted the change with +2, the review page will contain a Submit button that can be used to merge the change to the master branch.

Commit message format

In Git, the first row of the commit message is used for shortlog and generating patch file names so keep it short, concise and as descriptive as possible. The commit box in the UI shows a vertical line marking the maximum length of a commit message line and the view will inform you if you're exceeding limits. Note that the commit message box will automatically break long lines of text into separate lines to help you write a cleanly formatted commit message.

To keep commits attached to Redmine issues, use refs #nnn, where nnn is the issue number. This must be specified before Change-Id:. The Simantics platform Gerrit is configured to show refs #nnn lines as hyperlinks to platform Redmine issues, i.e. to URL https://www.simantics.org/redmine/issues/nnn.

Often platform commits are also related to some issue or issues from other projects built on top of the platform. In order to retain some information about this association in the commit messages we recommend using separate rows in commit messages in form[PRIVATE-mmm] where mmm is the issue number in the other project.

Gerrit reviews require a row starting Change-Id: as the last row of the commit message (excluding Signed-off-by rows). This allows Gerrit to bind commits to specific reviews. Gerrit will replace the default zero-valued Id with a generated one when the change is pushed for review.

All in all, a typical commit message should look like this:

Short description of my change in under 65 characters
 
A longer description of my changes
that can span multiple lines.
 
refs #nnn
[PRIVATE-mmm] (optional, use this if the commit relates to another project)

Change-Id: I0000000000000000000000000000000000000000

Simantics Member Read-only Git Access

1. Register to member wiki at https://www.simantics.org/members/index.php
2. Log into Gerrit using the credentials found at https://www.simantics.org/members/index.php/Code_Repository_Access
3. Clone https://member@gerrit.simantics.org/r/p/simantics/platform.git using the HTTP password found at https://gerrit.simantics.org/r/#/settings/http-password

The instructions for using Eclipse with this git repository apply also for the generic member account. Just remember to replace the cloned SSH repository address with the HTTP address shown above.

Testing Reviewed Changes

To test reviewed code changes locally, first you must have an Eclipse IDE setup according to the instructions in #Instructions for Simantics Developers or #Simantics Member Read-only Git Access.

Now, let's say for example we wish to grab a patch set from a review, such as review #72 (https://gerrit.simantics.org/r/72).

1. Open Git perspective
2. Select root node of the cloned simantics-platform repository from the Git Repositories view
3. From the context menu of the repository, select Fetch from Gerrit...
4. In the Change field, type the change number 72 and press CTRL+SPACE to start content assist. Content assist will provide you with the selections 72 - 1, and 72 - 2, i.e. patch sets 1 and 2 of change 72. After selecting a patch set, the field will be filled with the value refs/changes/72/72/1 or refs/changes/72/72/2 depending on the patch set you selected.
5. The default settings are otherwise recommended, i.e. Create a Local Branch and Checkout new branch to switch to the new branch immediately.

There you have it, a local branch that has exactly the same repository state as in the code to be reviewed. Start testing!

Platform Git Branching Model

Branch naming conventions:

• features: feature/<PROJECT NAME>-<issuenumber>.
  • Use these for developing a single feature. To keep the version history of feature branches cleaner, merge commit pushing is limited to project owners only. This means that feature branches should be used for implementing a particular thing and then merged back to master or some other branch. They should not be used as locations for eternal development and never ending try-outs.
• releases: release/x.y.z, where x, y and z are major, minor and service version numbers respectively.
  • Only for release stabilization branches.
• private work branches: private/<PROJECT or USER NAME>-<possible descriptive suffix>.
  • Should be used for personal work, trials and possibly even user or project specific custom branches that need to be kept. Gerrit is not configured enforce any rules on how these branches are used.
• hotfixes: hotfix/x.y.z.<hotfix number>.

Basic access rights:

action \ head	master release/* hotfix/*	feature/*	private/*
Code Review -/+2	Simantics developers	Simantics developers	Simantics developers
Submit	Owners only	Simantics developers	Simantics developers
Push	Owners only	Simantics developers	Simantics developers
Push Merge Commits	Owners only	Owners Only	Simantics developers

The current plan is to loosely follow GitFlow rules but we've decided to go without a separate develop branch. This means that we treat master as a development version instead of a production version. The actual release/* branches and tags are used to produce production versions. The rationale behind this is that we do not have a continuously evolving production version. Instead we produce new releases and products pick them up as they see fit.

Related reading:

• Git Flow branching model by Vincent Driessen
• http://endoflineblog.com/gitflow-considered-harmful

Transferred Projects

The following table lists the Git repositories that have been created so far and where their content is from in the SVN repositories.

SVN Location	Gerrit project
All over https://www.simantics.org/svn/simantics/	https://gerrit.simantics.org/r/#/admin/projects/simantics/platform
https://www.simantics.org/svn/simantics/3d	https://gerrit.simantics.org/r/#/admin/projects/simantics/3d
https://www.simantics.org/svn/simantics/fmi	https://gerrit.simantics.org/r/#/admin/projects/simantics/fmil
https://www.simantics.org/svn/simantics/r	https://gerrit.simantics.org/r/#/admin/projects/simantics/r
Everything except OpenModelica bundles from https://www.simantics.org/svn/simantics/sysdyn	https://gerrit.simantics.org/r/#/admin/projects/simantics/sysdyn
Parts of https://www.simantics.org/svn/simantics/third-party/trunk	https://gerrit.simantics.org/r/#/admin/projects/simantics/third-party
https://www.simantics.org/svn/members/fmi	https://gerrit.simantics.org/r/#/admin/projects/members/fmi

Still Untransferred Project Locations

• https://www.simantics.org/svn/simantics/interoperability
• https://www.simantics.org/svn/simantics/modelica
• https://www.simantics.org/svn/simantics/sysdyn: OpenModelica parts need to be put in a separate repository due to the huge size of OM. Preferable this repository should support LFS
• https://www.simantics.org/svn/simantics/tutorials

Q & A

Will the complete SVN history be transferred to Git?

Sadly, no it will not, at least for everything that is considered part of the current SVN Simantics SDK (see revisions list on page https://www.simantics.org/jenkins/job/target-simantics-head-sdk/lastStableBuild/).

The reason for not attempting to salvage the full history into git is in the Simantics SVN repository's structure. In SVN a "project structure" is a single combination of the folders trunk, branches and tags. In git branches and tags do not exist like this as separate directories in the repository itself. The git-svn tool is perfectly capable of transforming a single project structure into git repository with all its history, branches and tags. At some point in the past we for decided to split the SVN repository contents into multiple project structures in the hope of modularizing the platform SDK. That hope never quite realized and what's worse, that structure has made platform release engineering much more laborous and complex than it would be with a single project structure. Now in this move to Git we decided to simplify things and just put the platform SDK projects back in a single monolithic structure of bundles, features, tests and release engineering projects. All in all, we just didn't have enough git expertise to be able to transfer all the history scattered in these multiple project structures into a single git repository and moving the current SVN project structures each into its own git repository was not an enticing option either.

So how are we doing it right now? We have a separate migration git repository that contains a python script that simply exports all listed projects from SVN into the simantics-platform git repository structure. Our simantics-platform git repository contains a branch called 'svn'. The SVN exporting python script was originally used to initially populate the git repository, back when we were first testing the migration. Another python script is then used to bring in changes from SVN by exporting on top of the existing simantics-platform git repository. These changes are then committed into the svn branch and merged into master from there. We are not using git svn fetch because we're just not proficient enough with git-svn to be able to use it separately for each and every project that has moved location in git already. The git master branch has already been developed onwards, but merging changes from the svn branch to it is still very easy.

Note that the SVN repository will remain in place to keep all old versions of the platform that are still needed for maintaining existing products. We will not be moving these old branches/versions into git.

Note that in general if you have a single SVN project structure to transfer to git it is recommended to use git svn to transfer the complete history to Git.

How do I convert my SVN repository into a Gerrit/git project & repository?

The following is an example of converting a single SVN project location (trunk/,branches/,tags/) into a Gerrit-hosted git repository. The instructions assume you are working on a Linux installation which has the git command line tools installed.

SVN project structure location to be converted: https://www.simantics.org/svn/simantics/3d/

Firstly, create a new project in Gerrit ([1]). Create the project without an empty initial commit because the initial commits will come from the SVN repository anyway. In this example the project name simantics/3d is used.

Next clone the newly created git repository:

git clone ssh://<your-username>@www.simantics.org:29418/simantics/3d.git

Step into the cloned repository directory:

cd 3d

Initialize git svn with the correct SVN repository location:

git svn init https://www.simantics.org/svn/simantics/3d --stdlayout --prefix=svn/

Start fetching commits from the SVN repository into the empty git repository:

git svn fetch | tee fetch.log

Next, due to the way in which git svn converts branches and tags, we want to do a bit of cleanup to convert the branches into local branches before pushing them to the Gerrit's repository. Download this shell script and execute it in the repository's root directory:

sh Git-svn-convert-branches.sh

Also, download this shell script and run it to convert all the tag branches created by git svn into proper git tags:

sh Git-svn-convert-tags.sh

Now, check git log --graph --oneline --all --decorate=full to see whether the conversion turned out to your liking. Also, check how large the repository became. If it turned out larger than you would expect it is probably caused by smaller or larger binary artifacts that have been committed into the SVN repository. If it turns out you want to leave certain projects outside the git svn conversion, you can use the --ignore-paths argument with git svn init. For example you could use --ignore-paths 'org.simantics.openmodelica|org.simantics.om' to leave out all projects starting with org.simantics.openmodelica or org.simantics.om.

If you are satisfied, push all the commits and tags to remote origin:

$ git push --all
$ git push --tags

The branch/tag conversion scripts are heavily inspired by the script available at http://timepit.eu/~frank/blog/2008/08/converting_git-svn_tags/.

Why are there .keep files in Git repository folders?

• https://git.wiki.kernel.org/index.php/GitFaq#Can_I_add_empty_directories.3F
• http://stackoverflow.com/questions/115983/how-can-i-add-an-empty-directory-to-a-git-repository

Our recommendation is to add an empty file named .keep to the empty folder to ensure it is kept in git.

How can I make gerrit check X?

Gerrit integrates a prolog based submit rule system that seems to be the ultimate measure for configuring how reviews work. The prolog rules can be used, among other things, to achieve the following:

• Example 8: Make change submittable only if Code-Review+2 is given by a non author
• Example 13: 1+1=2 Code-Review
• Example 15: Only allow Author to submit change

See the cookbook at: https://gerrit-review.googlesource.com/Documentation/prolog-cookbook.html

Schedule

We have set 16th December 2016 as the date when the Simantics SVN repository project locations trunks will be made read-only.

Only those locations that have been indicated as transferred to git in #Transferred Projects have been marked read-only.

SVN to Git Transition TODO

• Test and migrate all products trunk versions on top of the simantics-platform git master branch.
• Fix the SDK org.simantics.root build to produce source bundles. It should work already but it just doesn't at the moment. Perhaps something needs to defined in org.simantics.sdk.repository/pom.xml.
• Set a date for when SVN trunks will be changed to read-only (16.12.2016)
  • E-mail simantics-developers with instructions or links to instructions about the change.
• Double check all instructions:
  • Basic Eclipse IDE installation with Simantics graph compiler
  • Git instructions and platform development workflow
  • Gerrit instructions and practices. Only project owners shall have the right for direct push to git repositories. Others must always go through gerrit review. This is enforced via Gerrit access rights.

• Organize brief training and feedback session the current developers (26.10.2016)

The next step is to start thinking about moving your project/product over to Gerrit!