Development Practices

Contributing to the Simantics codebase

The development of the Simantics platform is coordinated by the Simantics Technical Board. The Technical Board consists of three to six members that are selected among the THTH Simantics Division member organisations by the Simantics General Meeting. The duties of the Simantics Technical Board are defined in the THTH Simantics Division Rules and Regulations.

The Simantics Technical Board shall:

1. prepare a proposal for the Management Board for the exact content of the new releases of Simantics
2. prepare a proposal for the Management Board for the content of the training sessions
3. update simantics.org website both public and member areas

All contributions made to Simantics modules go through the Simantics Technical Board (STB) via Gerrit code reviews. The Simantics Technical Board meetings are arranged 6-8 weeks apart. These meetings are documented in the Simantics ticket management system (Redmine).

Our Scrum

Simantics platform development progresses in 2 week sprints. The technical board convenes for sprint meetings at the end of every sprint for review, retrospective and planning of the next sprint.

The planning is done in the Simantics Platform Redmine project and its backlog.

Reporting Bugs and Feature Requests

Bugs and feature requests shall be reported in the platform backlog. New issues are processed by the STB in sprint meetings. As a minimum effort, the bug is prioritized in the backlog and put on hold if not taken up for implementation immediately. There is a stored query in Redmine for listing all new issues created within the last two weeks.

Version control

First things first, read through Subversion best practices.

Branching

In Simantics, only new major releases may contain breaking changes. Therefore in Simantics we strive to use a specialization of the Branch-When-Needed system, i.e. Branch-Every-Major-Change. This allows us to keep Jenkins continuous builds working with less disruption, and enables easier pre-release testing, migration and peer review.

The goal is to follow these rules:

• Users commit their day-to-day work on trunk.
• Rule #1: trunk must compile and pass regression tests at all times. Committers who violate this rule are publically humiliated.
• Rule #2: a single commit (changeset) should not too large that it discourages peer-review.
• Rule #3: if rules #1 and #2 come into conflict (i.e. it's impossible to make a series of small commits without disrupting the trunk), then the user should create a branch and commit a series of smaller changesets there. This allows peer-review without disrupting the stability of trunk. When the work is finished, the changes can either be integrated back to trunk or thrown away, depending on the outcome of the development.
• Rule #4: public API changes are major changes. If API is to be changed, follow rule #3 in development and integrate in a controlled fashion once the migration path is clear. Non-incremental ontology changes also count as major changes since they potentially prevent the use of existing Simantics databases.
• Rule #5: avoid making any unnecessary whitespace changes, such as reindenting lines or even complete source files as this diminishes the usefulness of source control annotation/blame features.

We strive to keep the trunk version stable. However, since most development happens there, breakage is sometimes unavoidable and often accidental. The word stable here implies mainly API stability but also functional stability. Functional stability must be enforced through both automatic and manual testing. If there are no test cases, functional stability becomes really hard to enforce. Therefore automated testing is key (Jenkins).

Violating these rules results in reverting the breaking commits from SVN unless the problems can be resolved otherwise through developer communication.

Integrating Branches

We obviously do not want development to continue eternally in branches so that changes are never merged between branches or integrated back to trunk. Branches are usually private or shared between a group of developers that work on a particular task. The process for integration to trunk is as follows:

• The Simantics Technical Board (STB) and its weekly meetings will be the place for the developers to advertise and present their changes to the board.
• The following things need to be documented for the changes in some way:
  • The what and why for the changes
  • Migration instructions from previous version (trunk):
    • A direct patch to fix problems
    • or plain-text instructions
• When integration is done, send an email to Simantics developers list about the change, linking to the redmine issue(s) in question.

Versioning

In Eclipse/OSGi conventions, components (bundles, plug-ins) are versioned <major>.<minor>.<service>[.<qualifier>].

Our approach to component versioning is as follows:

• All components are in one of two states: incubation or production
• Incubation components: <major> = 0, API breakage is allowed between <minor> versions
• Production components: <major> ≥ 1, API breakage is allowed between <major> versions

Releasing

Before every major (1.0 vs. 2.0) or minor (1.1 vs. 1.2) release, trunk shall be branched into a version branch (e.g. simantics-x.y). The branch is then tested and fixed until it is deemed fit for release by the STB. Fixes are generally merged from other branches into the release branch. Upon release, the branch is tagged as simantics-x.y.1.

A service release (1.1.1 vs. 1.1.2) is a revision of a previous major or minor release that provides important enhancements or bugfixes but is otherwise completely compatible with the previous releases. The service changes are generally merged into the simantics-x.y branch from other branches. Upon release, the branch is again tagged as simantics-x.y.z, where z is incremented from the previous release.

When releases are made, the involved sprints and relevant changes shall be documented in the project's Redmine Wiki.

Our release roadmap is here.

Creating a new plugin

Before a release, the plugin is in incubation phase and there are no quality or other requirements. The repository https://www.simulationsite.net/svn/simantics-incubator may be used to share or co-develop the plugin with other developers, but the plugin must not be included in the features defined in https://www.simulationsite.net/svn/simantics repository.

If the new plugin is a result of refactoring some already existing plugin into two pieces or otherwise is closely related to an other plugin under development, the new plugin can be committed to the same branch of https://www.simulationsite.net/svn/simantics where the other plugin is developed in. Note that whenever this kind of piecing happens, it has the potential to affect the consistency of other components or products. In such cases the refactor should be carried out in branches, not in trunk.

Naming a plugin

If the plugin is intended to be part of the official Simantics release, its name should begin with org.simantics followed by a word that describes the plugin. If the functionality of the same domain is divided into multiple plugins, the name may have more parts. The following suffixes are used:

• .feature for feature projects.
• .product A plugin defining a product.
• <plugin>.ui Provides UI-functionality for <plugin>. Separating base functionality and UI (that depends on SWT) is usually recommendable so that headless applications do not have unnecessary dependencies.
  • Example: org.simantics.simulation, org.simantics.simulation.ui

NOTE! If the the UI is light (e.g. wizards, dialogs, renderers, few external dependencies) the UI code can reside in <plugin>, a separate <plugin>.ui is not required. In such a case, the UI dependencies (to plugins such as org.eclipse.swt) are optional.

Naming in feature projects

Features have two distinct names: project name and feature ID.

1. The project name is defined in the .project file. It shall contain the .feature suffix since the project name affects how Eclipse checks the project out into workspaces from version control systems.
2. The feature ID is defined in feature.xml and it shall not contain the .feature suffix.
   • Rationale: current Buckminster resource map (RMAP) definitions rely on this convention for feature project resolution. Having a strict convention keeps RMAPs more uniform and clean.
3. Feature ID's should not overlap plug-in ID's if at all possible.
   • Rationale: Buckminster commands will not be able to perform actions on features if a plug-in with the same ID exists.
   • Suggestion 1: if you can't think of a unique ID for your feature otherwise, append -feature to it.
   • Suggestion 2: use the feature ID as a base for the IDs of all plug-ins included in the feature. For example, if feature ID is org.example, then plug-in IDs could be org.example.core, org.example.ui, org.example.ontology, etc.

This naming convention exists mainly because it works well with Buckminster.

Modifying a feature

When a plug-in is added to or removed from a feature, the following steps must be performed:

1. Increment the version number of the feature, primarily the service version number part.
2. Increment service version of all other features dependent on this feature
3. Fix .product files that refer to the re-versioned features

This is necessary most of all to keep Eclipse IDE happy and working properly with checked out plug-ins and the P2-based target platforms.

If these steps are not performed, Eclipse will keep using the target platform version of the modified feature not realizing the changes you've made to it. This results often in products failing to work properly or not starting at all.

Publising a standalone plugin

1. Ensure that the plugin satisfies the following quality requirements:
   • The plugin contains no compilation errors (against which dependencies???)
   • All public classes and interfaces are documented (javadoc)
   • Each public package contains a package-info.java containing package-related comments and annotations (see [1] chapter 7.4)
   • examples/
   • unittests/
   • Version information
   • (optional) Build Script (targets: clean, build, clean-build)
     • [name]_[version]-src.zip
     • [name]_[version].jar
     • [name]_[version].zip
       • doc/
         • manual.pdf
         • changelog.txt
       • examples/
       • unittests/
       • javadoc/ (optional)
       • [name]_[version]-src.zip
       • [name]_[version].jar
     • [name]_[version]-project.zip (Contains everything in workspace)
     • See Example: Non-OSGi
2. Create a homepage for the plugin in wiki (for example org.simantics.databoard) that contains:
   • Development documentation that is not written in javadoc (basic concepts, components etc.) [We should discuss more about what and how to document] and links to user documentation if the plugin contributes new user interfaces.
   • Change log
   • Plugin roadmap
   • Who are responsible for the plugin
   • (optional) Download to artifacts created by Build script
3. Commit the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/trunk
4. Create a tag of the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/tags/<version>
5. Inform other developers about the new plugin through the mailing list [simantics-developers@simantics.org]. See also Mailing Lists.

Including a new plug-in to existing features

Inclusion of a plug-in in a feature must always be carefully considered. Consider the following before including anything:

• Cohesion: Does the plug-in logically belong in the considered feature.xml?
• Dependencies: What does the new plug-in depend on? Which dependencies are optional? Do the dependencies of the feature's other plug-ins intersect naturally with those of the new plug-in?
• Permission: do not add anything without permission from the feature maintainer or send a patch and hope it will be merged.

It is highly recommended to have plug-ins included in a single feature only. Instead of including plug-ins in multiple features, prefer creating highly logical and cohesive features that can acyclically include each other. In general completely new functionality should be put in its own feature. If you are unable to decide where a plug-in should be included, ask yourself is the plug-in properly cohesive, are its dependencies minimal?

Note that it is not technically mandatory that each plug-in only resides in one feature - it is simply good practice and will most likely help in training people to use the platform and also make maintainance easier. Having multiple features including the same plug-in will not break builds, but will make the product structure more difficult to understand.

Modifying a feature may break:

1. Project sets
   • FIX: add a line in the related PSF file(s) for the new plug-in in all project sets that include the modified feature
2. Jenkins jobs that depend on the feature
   • See #Building Products for instructions on how to define headless product builds

Releasing a new version of a component

Here, a component stands for a plug-in, a set of plug-ins or a feature.

1. Increment version number(s), incl. MANIFEST.MF, plugin.xml, feature.xml, build.xml, homepage.
   • Bug fixes increment service number, API changes minor number and large releases major number.
2. Run test cases or test suite
3. Check java docs are compiled without errors. Read Javadocs
4. Update homepage
5. (optional) run build.xml and upload generated artifacts to homepage
6. Create SVN tag: trunk/ → tags/[version]
7. Inform other developers Simantics developers list.

Making refactorizations that affect other plugins

1. Plan refactorization beforehand with the maintainers of all affected plugins
2. Create a new branch for all affected plugins
3. Refactor, different possibilities
   • One person refactors all plugins
   • Every maintainer refactors his/her own plugins
     • Create rename scripts for other maintainers to help them
4. Check that everything works
5. If changes were large inform other developers in the mailing list
6. Commit plugins to trunk

Developing and maintaining a released plugin

Generally, people need to be able to commit their changes continuously to the SVN location dedicated for a component. Often committing is necessary for transferring their changes to another machine or just to keep their code safe. We can't allow developers committing API changes or broken temporary code into SVN locations that other projects are using - those need to be stable to some degree. Hence, the active development and API breaking must occur in SVN locations that are documented to be unstable and modifiable only by component maintainer(s). These locations shall be released to the general public according to the roadmap of the component. For every release, a tag shall be created. For every release external dependencies need to be documented: component name, version to use (tag). Releases are versioned. Version numbers are composed of three (3) segments: 3 integers and a string respectively named major.minor.service. Read [2] and [3].

1. the major segment indicates breakage in the API
2. the minor segment indicates "externally visible" changes
3. the service segment indicates bug fixes and the change of development stream (the semantics attached to development stream is new to this proposal, see below)

By default, the following SVN layout should be used for components:

component/

trunk/

• The main development branch, unstable by nature. This version should only be used by active developers of this component.

branches/

• Branches should be created:
  • when preparing for a release, for stabilizing the component while letting active development run along in trunk.
  • for working on bug-fix releases.
  • when working on larger changes, trying out things or working with other developers. These kinds of branches should either be integrated back to where they were branched from or removed as obsolete.

tags/

• Released versions of the component. Other developers and project-sets should be using these for development if the component isn't already deployed into the target platform or it is necessary to have the code in your workspace for some other reason.

Developers may initially feel that creating tickets for every measly issue is too time consuming but it is necessary to make any kind of change tracking possible for releases. Obviously during the initial development of a component the code changes a lot and all the time. This is exactly why we have an incubator for starting projects that does not impose these strict rules but is more like a playground where the developer can freely "hack around" disregarding all these policies. A good developer will get started on design, testing and documentation also right from the start. When the developer feels his component is ready for consumption of the general public and the component passes peer review, it shall be ceremonially transferred from incubation into the official project. Each official component should have documentation, unit tests and possibly examples or really good reasons for not having any.

Modifying the target platform

DEPRECATED: these instructions are mainly deprecated with the current Git-based system

The target platform is a composition of features and plug-ins that contains all the bundles that you as a developer can build your code against when starting from a blank table. In Simantics, it is a mixture of the standard Eclipse RCP platform, other general Eclipse components and some Simantics components.

It makes sense to deploy into the target platform only if your plug-ins:

• are general, i.e. reusable by many other components
• have little dependencies
• are stable, both by API and functionally

Deploying plug-ins into the target platform will most likely break:

1. Project sets
   • FIX: remove lines matching the plug-ins that were moved into the target platform

Deploying plug-ins for the target platform

The Simantics target platform is a P2 repository located at

• http://www.simantics.org/download/_version_/sdk (w/o source)
• http://www.simantics.org/download/_version_/sdk-source (w/ source)

where _version_ is one of head, 1.11, 1.10, etc. head is trunk.

The repository is built using the buckminster build definition projects at svn:releng/sdk/branches. Ultimately, these builds use the feature project at svn:workbench/trunk/org.simantics.sdk.site.feature/ to define which features and plug-ins are part of the platform.

To include new features or plug-ins into the platform trunk development version:

1. Make your code/data available as plug-ins and/or features in a public repository, e.g. the Simantics repository.
2. Ask the target platform maintainer to include your components in the platform resource maps (see Buckminster#Resource_Maps).
3. Include them in org.simantics.sdk.site feature include them into the platform builds.

Jenkins build target-simantics-head-sdk takes care of building and publishing the trunk platform. Other branches have similar builds visible here.

Building Products

The best way to currently build products is to use the Eclipse tool Buckminster for the job.

Pre-requisites

1. You have a product configuration (.product file) for your product.
2. Your product configuration is based on features, not plug-ins
3. Buckminster locally installed or a jenkins installation with buckminster available

Example

To create your own (headless) buckminster product build follow these steps:

1. Export svn:tutorials/trunk/com.acme.movie.product.site.feature as a template for making your own headless buckminster product build.
2. To customize for your purposes, the following contents need to be configured:
   1. feature.xml
      • Change ID, version, name, provider and everything else to match your purposes.
      • Include all the features your .product file depends on, nothing else.
   2. Resources.rmap
      • Define mappings for all resources (features and plug-ins) needed by your product. See Buckminster definitive guide for help on defining resource maps.
   3. .credentials.properties
      • Insert any credentials needed to access resources defined in your resource map.
   4. site.cquery
      • Modify cq:rootRequest name to match your site feature ID.
   5. jenkins.buckminster.script
      • Replace com.acme.movie.product.site with what ever is your site feature's ID.
      • Replace p2.profileName with a new profile name for your product.
      • Replace p2.rootId to match the uid attribute defined in your Product Configuration file (.product).
      • Set build.label to what ever you want your built product .zip files to be named like.

Using Continuous Integration (Jenkins)

1. Install jenkins (see simantics.org jenkins)
2. Create new free-style software project job with appropriate name
3. Configure the project:
   1. In the Source code management section, use your version control system of choice to checkout the ...product.site.feature you've created for your build in the previous section. For the com.acme.movie example the proper settings would be:
      • Select Subversion Modules
      • Repository URL: https://www.simantics.org/svn/simantics/tutorials/trunk/com.acme.movie.product.site.feature
      • Local Module directory (optional): com.acme.movie.product.site
      • Check-out strategy: Always check out a fresh copy
   2. In the Build section:
      1. Add new Run Buckminster build step
      2. Select a Buckminster installation to use (latest)
      3. Copy the contents of jenkins.buckminster.script into the commands text box
   3. Last, you'll want to archive the vital artifacts produced by your build. In the Post-build Actions section:
      1. Check Archive the artifacts
      2. Set value of Files to archive to buckminster.output/com.acme.movie.product.site_*-eclipse.feature/*.zip (obviously changing the value according to the name of your product site feature).

That should do it. Further configuration of the build is up to you, i.e. scheduling, parsing warnings, and so on.

External links

• Buckminster Homepage
• Eclipse Buckminster, The Definitive Guide