Difference between revisions of "Development Practices"

From Developer Documents
Jump to navigation Jump to search
m
 
(97 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<center>''If it isn't in the wiki, it doesn't exist.''</center>
+
== Contributing to the Simantics codebase ==
  
== Using version control ==
+
The development of the Simantics platform is coordinated by the ''Simantics Technical Board''. The Technical Board consists of three to six members that are selected among the THTH Simantics Division member organisations by the Simantics General Meeting. The duties of the Simantics Technical Board are defined in the [https://www.simantics.org/about/thth-simantics THTH Simantics Division Rules and Regulations].
 +
 
 +
The Simantics Technical Board shall:
 +
# prepare a proposal for the Management Board for the exact content of the new releases of Simantics
 +
# prepare a proposal for the Management Board for the content of the training sessions
 +
# update simantics.org website both public and member areas
 +
 
 +
'''All contributions made to Simantics modules go through the Simantics Technical Board (STB) via Gerrit code reviews.''' The Simantics Technical Board meetings are arranged 6-8 weeks apart. These meetings are documented in the Simantics ticket management system ([https://www.simantics.org/redmine Redmine]).
 +
 
 +
== Our Scrum ==
 +
 
 +
Simantics platform development progresses in 2 week sprints. The technical board convenes for sprint meetings at the end of every sprint for review, retrospective and planning of the next sprint.
 +
 
 +
The planning is done in the [https://www.simantics.org/redmine/projects/simantics-platform Simantics Platform Redmine project] and its [https://www.simantics.org/redmine/rb/master_backlog/simantics-platform?jump=rb_master_backlogs# backlog].
 +
 
 +
=== Reporting Bugs and Feature Requests ===
 +
 
 +
Bugs and feature requests shall be reported in the platform backlog. New issues are processed by the STB in sprint meetings. As a minimum effort, the bug is prioritized in the backlog and put on hold if not taken up for implementation immediately. There is a [https://www.simantics.org/redmine/projects/simantics/issues?query_id=92 stored query] in Redmine for listing all new issues created within the last two weeks.
 +
 
 +
== Version control ==
  
 
First things first, read through [http://svn.apache.org/repos/asf/subversion/trunk/doc/user/svn-best-practices.html Subversion best practices].
 
First things first, read through [http://svn.apache.org/repos/asf/subversion/trunk/doc/user/svn-best-practices.html Subversion best practices].
Line 7: Line 26:
 
=== Branching ===
 
=== Branching ===
  
In Simantics, only new major releases may contain breaking changes. Therefore in Simantics we strive to use a specialization of the ''Branch-When-Needed'' system, i.e. '''Branch-Every-Major-Change'''. This allows us to keep continuous builds (hudson) working with less disruption, and enables easier pre-release testing, migration and peer review.
+
In Simantics, only new major releases may contain breaking changes. Therefore in Simantics we strive to use a specialization of the ''Branch-When-Needed'' system, i.e. '''Branch-Every-Major-Change'''. This allows us to keep [http://www.simantics.org/jenkins/ Jenkins continuous builds] working with less disruption, and enables easier pre-release testing, migration and peer review.
 +
 
 +
The goal is to follow these rules:
 +
* Users commit their day-to-day work on trunk.
 +
* '''Rule #1''': trunk must compile and pass regression tests at all times. Committers who violate this rule are publically humiliated.
 +
* '''Rule #2''': a single commit (changeset) should not too large that it discourages peer-review.
 +
* '''Rule #3''': if rules #1 and #2 come into conflict (i.e. it's impossible to make a series of small commits without disrupting the trunk), then the user should create a branch and commit a series of smaller changesets there. This allows peer-review without disrupting the stability of trunk. When the work is finished, the changes can either be integrated back to trunk or thrown away, depending on the outcome of the development.
 +
* '''Rule #4''': public API changes are major changes. If API is to be changed, follow rule #3 in development and integrate in a controlled fashion once the migration path is clear. Non-incremental ontology changes also count as major changes since they potentially prevent the use of existing Simantics databases.
 +
* '''Rule #5''': avoid making any unnecessary whitespace changes, such as reindenting lines or even complete source files as this diminishes the usefulness of source control annotation/blame features.
  
The rules are:
+
We strive to keep the trunk version ''stable''. However, since most development happens there, breakage is sometimes unavoidable and often accidental. The word ''stable'' here implies mainly API stability but also functional stability. Functional stability must be enforced through both automatic and manual testing. If there are no test cases, functional stability becomes really hard to enforce. Therefore automated testing is key ([http://www.simantics.org/jenkins/view/Testing/ Jenkins]).
* Users commit their day-to-day work on /trunk.
 
* '''Rule #1''': /trunk must compile and pass regression tests at all times. Committers who violate this rule are publically humiliated.
 
* '''Rule #2''': a single commit (changeset) must not be so large so as to discourage peer-review.
 
* '''Rule #3''': if rules #1 and #2 come into conflict (i.e. it's impossible to make a series of small commits without disrupting the trunk), then the user should create a branch and commit a series of smaller changesets there. This allows peer-review without disrupting the stability of /trunk.
 
  
;Pros: /trunk is guaranteed to be stable at all times. The hassle of branching/merging is somewhat rare.
+
Violating these rules results in reverting the breaking commits from SVN unless the problems can be resolved otherwise through developer communication.
  
The word '''stable''' here implies mainly API stability but also functional stability. Functional stability must be enforced through both automatic and manual testing. If there are no test cases, functional stability becomes really hard to enforce. Therefore testing is key.
+
=== Integrating Branches ===
 +
We obviously do not want development to continue eternally in branches so that changes are never merged between branches or integrated back to trunk. Branches are usually private or shared between a group of developers that work on a particular task. The process for integration to trunk is as follows:
 +
* The Simantics Technical Board ('''STB''') and its weekly meetings will be the place for the developers to advertise and present their changes to the board.
 +
* The following things need to be documented for the changes in some way:
 +
** The '''what''' and '''why''' for the changes
 +
** Migration instructions from previous version (trunk):
 +
*** A direct patch to fix problems
 +
*** or plain-text instructions
 +
* When integration is done, send an email to [mailto:simantics-developers@lists.simantics.org Simantics developers list] about the change, linking to the redmine issue(s) in question.
  
 
=== Versioning ===
 
=== Versioning ===
Line 27: Line 58:
 
* Incubation components: <major> = 0, API breakage is allowed between <minor> versions
 
* Incubation components: <major> = 0, API breakage is allowed between <minor> versions
 
* Production components: <major> &ge; 1, API breakage is allowed between <major> versions
 
* Production components: <major> &ge; 1, API breakage is allowed between <major> versions
 +
 +
=== Releasing ===
 +
 +
Before every major (1.0 vs. 2.0) or minor (1.1 vs. 1.2) release, trunk shall be branched into a version branch (e.g. simantics-x.y). The branch is then tested and fixed until it is deemed fit for release by the ''STB''. Fixes are generally merged from other branches into the release branch. Upon release, the branch is tagged as simantics-x.y.1.
 +
 +
A service release (1.1.1 vs. 1.1.2) is a revision of a previous major or minor release that provides important enhancements or bugfixes but is otherwise completely compatible with the previous releases. The service changes are generally merged into the simantics-x.y branch from other branches. Upon release, the branch is again tagged as simantics-x.y.z, where z is incremented from the previous release.
 +
 +
When releases are made, the involved sprints and relevant changes shall be documented in the project's [https://www.simantics.org/redmine/projects/simantics-platform/wiki/ChangeLog Redmine Wiki].
 +
 +
Our release roadmap is [[Roadmap|here]].
  
 
== Creating a new plugin ==
 
== Creating a new plugin ==
Line 33: Line 74:
 
the plugin must not be included in the features defined in https://www.simulationsite.net/svn/simantics repository.
 
the plugin must not be included in the features defined in https://www.simulationsite.net/svn/simantics repository.
  
If the new plugin is a result of refactoring some already existing plugin into two pieces or otherwise is closely related to an other plugin under development, the new plugin can be committed to the same branch of https://www.simulationsite.net/svn/simantics where the other plugin is developed in. However, unreleased plugins should not be committed to trunk.
+
If the new plugin is a result of refactoring some already existing plugin into two pieces or otherwise is closely related to an other plugin under development, the new plugin can be committed to the same branch of https://www.simulationsite.net/svn/simantics where the other plugin is developed in. Note that whenever this kind of piecing happens, it has the potential to affect the consistency of other components or products. In such cases the refactor should be carried out in branches, not in trunk.
  
 
=== Naming a plugin ===
 
=== Naming a plugin ===
  
 
If the plugin is intended to be part of the official Simantics release, its name should begin with ''org.simantics'' followed by a word that describes the plugin. If the functionality of the same domain is divided into multiple plugins, the name may have more parts. The following suffixes are used:
 
If the plugin is intended to be part of the official Simantics release, its name should begin with ''org.simantics'' followed by a word that describes the plugin. If the functionality of the same domain is divided into multiple plugins, the name may have more parts. The following suffixes are used:
* ''.feature'' A plugin (?) defining a feature
+
* ''.feature'' for feature projects.
 
* ''.product'' A plugin defining a product.
 
* ''.product'' A plugin defining a product.
 
* ''<plugin>.ui'' Provides UI-functionality for ''<plugin>''. Separating base functionality and UI (that depends on SWT) is usually recommendable so that headless applications do not have unnecessary dependencies.
 
* ''<plugin>.ui'' Provides UI-functionality for ''<plugin>''. Separating base functionality and UI (that depends on SWT) is usually recommendable so that headless applications do not have unnecessary dependencies.
Line 44: Line 85:
  
 
'''NOTE!''' If the the UI is light (e.g. wizards, dialogs, renderers, few external dependencies) the UI code can reside in ''<plugin>'', a separate ''<plugin>.ui'' is not required. In such a case, the UI dependencies (to plugins such as ''org.eclipse.swt'') are ''optional''.
 
'''NOTE!''' If the the UI is light (e.g. wizards, dialogs, renderers, few external dependencies) the UI code can reside in ''<plugin>'', a separate ''<plugin>.ui'' is not required. In such a case, the UI dependencies (to plugins such as ''org.eclipse.swt'') are ''optional''.
 +
 +
==== Naming in feature projects ====
 +
 +
Features have two distinct names: '''project name''' and '''feature ID'''.
 +
 +
# The '''project name''' is defined in the ''.project'' file. It shall contain the ''.feature'' suffix since the project name affects how Eclipse checks the project out into workspaces from version control systems.
 +
# The '''feature ID''' is defined in feature.xml and it shall not contain the ''.feature'' suffix.
 +
#* Rationale: current [http://www.eclipse.org/buckminster/ Buckminster] [http://wiki.eclipse.org/Buckminster_Resource_Map resource map] (RMAP) definitions rely on this convention for feature project resolution. Having a strict convention keeps RMAPs more uniform and clean.
 +
# Feature ID's should not overlap plug-in ID's if at all possible.
 +
#* Rationale: Buckminster commands will not be able to perform actions on features if a plug-in with the same ID exists.
 +
#* Suggestion 1: if you can't think of a unique ID for your feature otherwise, append ''-feature'' to it.
 +
#* Suggestion 2: use the feature ID as a base for the IDs of all plug-ins included in the feature. For example, if feature ID is org.example, then plug-in IDs could be org.example.core, org.example.ui, org.example.ontology, etc.
 +
 +
This naming convention exists mainly because it works well with Buckminster.
 +
 +
== Modifying a feature ==
 +
 +
When a plug-in is added to or removed from a feature, the following steps must be performed:
 +
# Increment the version number of the feature, primarily the service version number part.
 +
# Increment service version of all other features dependent on this feature
 +
# Fix .product files that refer to the re-versioned features
 +
 +
This is necessary most of all to keep Eclipse IDE happy and working properly with checked out plug-ins and the P2-based target platforms.
 +
 +
If these steps are not performed, Eclipse will keep using the target platform version of the modified feature not realizing the changes you've made to it. This results often in products failing to work properly or not starting at all.
  
 
== Publising a standalone plugin ==
 
== Publising a standalone plugin ==
Line 67: Line 133:
 
#*** [name]_[version].jar
 
#*** [name]_[version].jar
 
#** [name]_[version]-project.zip (Contains everything in workspace)
 
#** [name]_[version]-project.zip (Contains everything in workspace)
#** See Example: [[svn:tagi/org.simantics.datatype/trunk/build.xml|Non-OSGi]], [[svn:tagi/org.simantics.data/trunk/build2.xml|OSGi]]  
+
#** See Example: [[svn:foundation/databoard/trunk/org.simantics.databoard/build2.xml|Non-OSGi]]
# Create a homepage for the plugin in wiki (for example [[org.simantics.datatype]]) that contains:
+
# Create a homepage for the plugin in wiki (for example [[org.simantics.databoard]]) that contains:
 
#* Development documentation that is not written in javadoc (basic concepts, components etc.) [We should discuss more about what and how to document] and links to user documentation if the plugin contributes new user interfaces.
 
#* Development documentation that is not written in javadoc (basic concepts, components etc.) [We should discuss more about what and how to document] and links to user documentation if the plugin contributes new user interfaces.
 
#* Change log
 
#* Change log
Line 76: Line 142:
 
# Commit the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/trunk
 
# Commit the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/trunk
 
# Create a tag of the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/tags/<version>
 
# Create a tag of the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/tags/<version>
# Inform other developers about the new plugin in [simantics-developers@simantics.org]. See also [[Mailing Lists]].
+
# Inform other developers about the new plugin through the mailing list [simantics-developers@simantics.org]. See also [https://www.simantics.org/wiki/index.php/Mailing_Lists Mailing Lists].
  
 
=== Including a new plug-in to existing features ===
 
=== Including a new plug-in to existing features ===
Line 85: Line 151:
 
* '''Permission''': do not add anything without permission from the feature maintainer or send a patch and hope it will be merged.
 
* '''Permission''': do not add anything without permission from the feature maintainer or send a patch and hope it will be merged.
  
Note that it is not mandatory that each plug-in only resides in one feature. Having multiple features including the same plug-in will not break a PDE build.
+
It is highly recommended to have plug-ins included in a single feature only. Instead of including plug-ins in multiple features, prefer creating highly logical and cohesive features that can acyclically include each other. In general completely new functionality should be put in its own feature. If you are unable to decide where a plug-in should be included, ask yourself is the plug-in properly cohesive, are its dependencies minimal?
 +
 
 +
Note that it is not technically mandatory that each plug-in only resides in one feature - it is simply good practice and will most likely help in training people to use the platform and also make maintainance easier. Having multiple features including the same plug-in will not break builds, but will make the product structure more difficult to understand.
  
Modifying a feature will most likely break:
+
Modifying a feature may break:
# [http://www.simantics.org/hudson Hudson PDE builds] that depend on the feature
 
#* TODO: link to instructions for general hudson and pde build configuration
 
 
# [[svn:project-set/|Project sets]]
 
# [[svn:project-set/|Project sets]]
#* FIX: add a line for the new plug-in in all project sets that include the modified feature
+
#* FIX: add a line in the related PSF file(s) for the new plug-in in all project sets that include the modified feature
 +
# [http://www.simantics.org/jenkins Jenkins jobs] that depend on the feature
 +
#* See [[#Building Products]] for instructions on how to define headless product builds
 +
 
 +
== Releasing a new version of a component ==
  
== Releasing a new version of a plugin ==
+
Here, a component stands for ''a plug-in'', ''a set of plug-ins'' or ''a feature''.
  
# Increment version number, incl. plugin.xml, build.xml, homepage. Bug fixes increment minor number, API changes major number.
+
# Increment version number(s), incl. MANIFEST.MF, plugin.xml, feature.xml, build.xml, homepage.
 +
#* Bug fixes increment service number, API changes minor number and large releases major number.
 
# Run test cases or test suite
 
# Run test cases or test suite
 
# Check java docs are compiled without errors. Read Javadocs
 
# Check java docs are compiled without errors. Read Javadocs
Line 101: Line 172:
 
# (optional) run build.xml and upload generated artifacts to homepage
 
# (optional) run build.xml and upload generated artifacts to homepage
 
# Create SVN tag: trunk/ &rarr; tags/[version]
 
# Create SVN tag: trunk/ &rarr; tags/[version]
# Inform other developers [simantics-developers@simantics.org] ([[Mailing Lists]]).
+
# Inform other developers [mailto:simantics-developers@lists.simantics.org Simantics developers list].
  
 
== Making refactorizations that affect other plugins ==
 
== Making refactorizations that affect other plugins ==
Line 131: Line 202:
 
:::** when preparing for a release, for stabilizing the component while letting active development run along in ''trunk''.
 
:::** when preparing for a release, for stabilizing the component while letting active development run along in ''trunk''.
 
:::** for working on bug-fix releases.
 
:::** for working on bug-fix releases.
:::** when working on a larger changes, trying out things or working with other developers. These kinds of branches should either be integrated back to where they were branched from or removed as obsolete.
+
:::** when working on larger changes, trying out things or working with other developers. These kinds of branches should either be integrated back to where they were branched from or removed as obsolete.
 
:: tags/
 
:: tags/
 
:::* Released versions of the component. Other developers and project-sets should be using these for development if the component isn't already deployed into the target platform or it is necessary to have the code in your workspace for some other reason.
 
:::* Released versions of the component. Other developers and project-sets should be using these for development if the component isn't already deployed into the target platform or it is necessary to have the code in your workspace for some other reason.
  
Developers may initially feel that creating tickets for every measly issue is too time consuming but it is necessary to make any kind of change tracking possible for releases. Obviously during the initial development of a component the code changes a lot and all the time. This is exactly why we have an ''incubator'' for starting projects that does not impose these strict rules but is more like a playground where the developer can freely "hack around" disregarding all these policies. When the developer feels his component is ready for consumption of the general public and the component passes peer review, it shall be ceremonially transferred from incubation into the official project. Each component should have documentation, unit tests and possibly examples or '''really good''' reasons for not having any.
+
Developers may initially feel that creating tickets for every measly issue is too time consuming but it is necessary to make any kind of change tracking possible for releases. Obviously during the initial development of a component the code changes a lot and all the time. This is exactly why we have an ''incubator'' for starting projects that does not impose these strict rules but is more like a playground where the developer can freely "hack around" disregarding all these policies. A good developer will get started on design, testing and documentation also right from the start. When the developer feels his component is ready for consumption of the general public and the component passes peer review, it shall be ceremonially transferred from incubation into the official project. Each official component should have documentation, unit tests and possibly examples or '''really good''' reasons for not having any.
  
 
== Modifying the target platform ==
 
== Modifying the target platform ==
  
The target platform is a composition of features and plug-ins that contains all the bundles that you as a developer can build your code against when starting from a blank table. It is a mixture of the standard Eclipse RCP platform, other general Eclipse components and also some Simantics components.
+
DEPRECATED: these instructions are mainly deprecated with the current Git-based system
 +
 
 +
The target platform is a composition of features and plug-ins that contains all the bundles that you as a developer can build your code against when starting from a blank table. In Simantics, it is a mixture of the standard Eclipse RCP platform, other general Eclipse components and some Simantics components.
  
 
It makes sense to deploy into the target platform only if your plug-ins:
 
It makes sense to deploy into the target platform only if your plug-ins:
Line 148: Line 221:
 
Deploying plug-ins into the target platform will most likely break:
 
Deploying plug-ins into the target platform will most likely break:
  
# [http://www.simantics.org/hudson Hudson PDE builds] that depend on the feature
 
#* Remove the deployed plug-ins from the plugins/ directory of the build structure to take the deployed plug-ins from the target platform
 
#* The target platform(s) used by the builds must be updated
 
#* TODO: link to instructions for general hudson and PDE build configuration
 
 
# [[svn:project-set/|Project sets]]
 
# [[svn:project-set/|Project sets]]
 
#* FIX: remove lines matching the plug-ins that were moved into the target platform
 
#* FIX: remove lines matching the plug-ins that were moved into the target platform
Line 157: Line 226:
 
=== Deploying plug-ins for the target platform ===
 
=== Deploying plug-ins for the target platform ===
  
;Standard Procedure:
+
The Simantics target platform is a P2 repository located at
# Read the instructions below and deploy your plug-ins and features as P2 repositories
+
* http://www.simantics.org/download/_version_/sdk (w/o source)
# Test your deployed components with the standard Simantics Workbench application (search for simantics-workbench.product in your development workspace)
+
* http://www.simantics.org/download/_version_/sdk-source (w/ source)
# After testing your deployed versions, make your built P2 repository available on the internet and notify the [[Target Platform#Maintainer|target platform maintainer]] to grab your changes and update the official target platform.
+
where ''_version_'' is one of '''head''', '''1.11''', '''1.10''', etc. '''head''' is trunk.
 +
 
 +
The repository is built using the buckminster build definition projects at [[svn:releng/sdk/branches]]. Ultimately, these builds use the feature project at [[svn:workbench/trunk/org.simantics.sdk.site.feature/]] to define which features and plug-ins are part of the platform.
 +
 
 +
To include new features or plug-ins into the platform trunk development version:
 +
# Make your code/data available as plug-ins and/or features in a public repository, e.g. the [[svn:/|Simantics repository]].
 +
# Ask the [[Target_Platform#Maintenance|target platform maintainer]] to include your components in the [http://www.simantics.org/download/head/rmap platform resource maps] (see [[Buckminster#Resource_Maps]]).
 +
# Include them in [[svn:workbench/trunk/org.simantics.sdk.site.feature/|org.simantics.sdk.site]] feature include them into the platform builds.
 +
 
 +
Jenkins build [http://www.simantics.org/jenkins/job/target-simantics-head-sdk/ target-simantics-head-sdk] takes care of building and publishing the trunk platform. Other branches have similar builds visible [http://www.simantics.org/jenkins/view/Platform/ here].
  
There are two main ways of deploying for the target platform:
+
== Building Products ==
* deploying plug-ins and fragments directly
 
* deploying plug-ins and fragments through features
 
  
These steps can be performed using the Eclipse ''export wizard'' through the ''Plug-in Development/Deployable features'' and ''Plug-in Development/Deployable plug-ins and fragments'' wizards
+
The best way to currently build products is to use the Eclipse tool [http://www.eclipse.org/buckminster/ Buckminster] for the job.
  
When considering whether your component should have a feature or not, consider these points:
 
* Eclipse products/applications are always made up of either a set of features or a set of plug-ins. In order for someone to use your component in an product, either your features or your plug-ins need to be included somewhere in the product. If a product is using features and you don't provide one, the product either has to create a new feature and include it or include your plug-ins in his own features.
 
* Features can be used as a way to give a collective version all plug-ins of a component. The feature will identify the versions of its included plug-ins but these need not be identical to the feature version. Not all plug-ins have to change versions while a feature version changes each time a new release of the component is made.
 
** '''IMPORTANT''': The feature deployment build will produce a ''P2 repository'' containing ''content.jar'', ''artifacts.jar'' and two directories, ''features/'' and ''plugins/''. The features-directory will contain the feature as JAR files which is correct when used as a P2 repository. But if you want to install your deployed features into the current target platforms, the feature jars must be unpacked with the JAR file names as the directory names' '''Don't unpack directly into the features directory''', the feature JARs do not contain a common parent folder. For example, if you've built your feature with an ID '''my.feature''' and version '''1.0.0''', the build will produce a file '''my.feature_1.0.0.jar'''. The contents of this jar must be unpacked into '''features/my.feature_1.0.0/'''. If you copied the JAR files into the features directory and unpack them there, '''do not leave the JARs in the features directory'''. This will also break the features.
 
  
 +
;Pre-requisites:
 +
# You have a product configuration (.product file) for your product.
 +
# Your product configuration is based on ''features'', not plug-ins
 +
# Buckminster locally installed or a jenkins installation with buckminster available
  
  
== Buckminster Install Instructions ==
+
; Example
* Install P2 Director ( http://www.eclipse.org/buckminster/downloads.html )
+
To create your own (headless) buckminster product build follow these steps:
** Install Eclipse-Helios
+
# Export [[svn:tutorials/trunk/com.acme.movie.product.site.feature]] as a template for making your own headless buckminster product build.
** Go to eclipse helios directory.
+
# To customize for your purposes, the following contents need to be configured:
** Create director.bat
+
## feature.xml
<pre>@eclipsec.exe -nosplash -application org.eclipse.equinox.p2.director %*</pre>
+
##* Change ID, version, name, provider and everything else to match your purposes.
 +
##* Include all the features your .product file depends on, nothing else.
 +
## Resources.rmap
 +
##* Define mappings for all resources (features and plug-ins) needed by your product. See [[#External links|Buckminster definitive guide]] for help on defining resource maps.
 +
## .credentials.properties
 +
##* Insert any credentials needed to access resources defined in your resource map.
 +
## site.cquery
 +
##* Modify cq:rootRequest name to match your site feature ID.
 +
## jenkins.buckminster.script
 +
##* Replace '''com.acme.movie.product.site''' with what ever is your site feature's ID.
 +
##* Replace p2.profileName with a new profile name for your product.
 +
##* Replace p2.rootId to match the '''uid''' attribute defined in your Product Configuration file (.product).
 +
##* Set build.label to what ever you want your built product .zip files to be named like.
  
* Install Buckminster
+
=== Using Continuous Integration (Jenkins) ===
<pre>
 
  director
 
-r http://www.simantics.org/update/
 
-d <buckminster install folder>
 
-p Buckminster
 
-i org.eclipse.buckminster.cmdline.product
 
</pre>
 
* Install Addons
 
**Go to buckminster install folder.
 
  buckminster listsite
 
  buckminster listsite http://www.simantics.org/update/buckminster/
 
  buckminster install http://www.simantics.org/update/ org.eclipse.buckminster.subversive.headless.feature
 
  buckminster install http://www.simantics.org/update/ org.eclipse.buckminster.pde.headless.feature
 
  
 +
# Install [http://jenkins-ci.org/ jenkins] (see [http://www.simantics.org/jenkins simantics.org jenkins])
 +
# Create new ''free-style software project'' job with appropriate name
 +
# Configure the project:
 +
## In the ''Source code management'' section, use your version control system of choice to checkout the '''...product.site.feature''' you've created for your build in the previous section. For the com.acme.movie example the proper settings would be:
 +
##* Select ''Subversion Modules''
 +
##* Repository URL: https://www.simantics.org/svn/simantics/tutorials/trunk/com.acme.movie.product.site.feature
 +
##* Local Module directory (optional): com.acme.movie.product.site
 +
##* Check-out strategy: Always check out a fresh copy
 +
## In the ''Build'' section:
 +
### Add new '''Run Buckminster''' build step
 +
### Select a Buckminster installation to use (latest)
 +
### Copy the contents of '''jenkins.buckminster.script''' into the ''commands'' text box
 +
## Last, you'll want to archive the vital artifacts produced by your build. In the ''Post-build Actions'' section:
 +
### Check ''Archive the artifacts''
 +
### Set value of ''Files to archive'' to '''buckminster.output/com.acme.movie.product.site_*-eclipse.feature/*.zip''' (obviously changing the value according to the name of your product site feature).
  
  
 +
That should do it. Further configuration of the build is up to you, i.e. scheduling, parsing warnings, and so on.
  
Original repositories, Use these incase of problems
+
== External links ==
* http://download.eclipse.org/releases/helios
 
* http://download.eclipse.org/tools/buckminster/updates-3.6
 
* http://download.eclipse.org/tools/buckminster/headless-3.6/
 
* http://download.cloudsmith.com/buckminster/external-3.6
 
  
Or execute this:
+
* [http://www.eclipse.org/buckminster/ Buckminster Homepage]
clipsec.exe -nosplash -application org.eclipse.equinox.p2.director -d ~/buckminster -p Buckminster -r http://download.eclipse.org/tools/buckminster/headless-3.6/,http://download.cloudsmith.com/buckminster/external-3.6,http://download.eclipse.org/releases/helios -i org.eclipse.buckminster.pde.headless.feature.feature.group,org.eclipse.buckminster.subversive.headless.feature.feature.group,org.eclipse.buckminster.core.headless.feature.feature.group,org.eclipse.buckminster.cmdline.product
+
* [http://www.eclipse.org/downloads/download.php?file=/tools/buckminster/doc/BuckyBook.pdf Eclipse Buckminster, The Definitive Guide]
  
./director -d ~/buckminster -p Buckminster -r http://download.eclipse.org/tools/buckminster/headless-3.6/,http://download.cloudsmith.com/buckminster/external-3.6,http://download.eclipse.org/releases/helios -i org.eclipse.buckminster.pde.headless.feature.feature.group,org.eclipse.buckminster.subversive.headless.feature.feature.group,org.eclipse.buckminster.core.headless.feature.feature.group,org.eclipse.buckminster.cmdline.product
+
[[Category: Miscellaneous Documents]]

Latest revision as of 11:48, 28 March 2021

Contributing to the Simantics codebase

The development of the Simantics platform is coordinated by the Simantics Technical Board. The Technical Board consists of three to six members that are selected among the THTH Simantics Division member organisations by the Simantics General Meeting. The duties of the Simantics Technical Board are defined in the THTH Simantics Division Rules and Regulations.

The Simantics Technical Board shall:

  1. prepare a proposal for the Management Board for the exact content of the new releases of Simantics
  2. prepare a proposal for the Management Board for the content of the training sessions
  3. update simantics.org website both public and member areas

All contributions made to Simantics modules go through the Simantics Technical Board (STB) via Gerrit code reviews. The Simantics Technical Board meetings are arranged 6-8 weeks apart. These meetings are documented in the Simantics ticket management system (Redmine).

Our Scrum

Simantics platform development progresses in 2 week sprints. The technical board convenes for sprint meetings at the end of every sprint for review, retrospective and planning of the next sprint.

The planning is done in the Simantics Platform Redmine project and its backlog.

Reporting Bugs and Feature Requests

Bugs and feature requests shall be reported in the platform backlog. New issues are processed by the STB in sprint meetings. As a minimum effort, the bug is prioritized in the backlog and put on hold if not taken up for implementation immediately. There is a stored query in Redmine for listing all new issues created within the last two weeks.

Version control

First things first, read through Subversion best practices.

Branching

In Simantics, only new major releases may contain breaking changes. Therefore in Simantics we strive to use a specialization of the Branch-When-Needed system, i.e. Branch-Every-Major-Change. This allows us to keep Jenkins continuous builds working with less disruption, and enables easier pre-release testing, migration and peer review.

The goal is to follow these rules:

  • Users commit their day-to-day work on trunk.
  • Rule #1: trunk must compile and pass regression tests at all times. Committers who violate this rule are publically humiliated.
  • Rule #2: a single commit (changeset) should not too large that it discourages peer-review.
  • Rule #3: if rules #1 and #2 come into conflict (i.e. it's impossible to make a series of small commits without disrupting the trunk), then the user should create a branch and commit a series of smaller changesets there. This allows peer-review without disrupting the stability of trunk. When the work is finished, the changes can either be integrated back to trunk or thrown away, depending on the outcome of the development.
  • Rule #4: public API changes are major changes. If API is to be changed, follow rule #3 in development and integrate in a controlled fashion once the migration path is clear. Non-incremental ontology changes also count as major changes since they potentially prevent the use of existing Simantics databases.
  • Rule #5: avoid making any unnecessary whitespace changes, such as reindenting lines or even complete source files as this diminishes the usefulness of source control annotation/blame features.

We strive to keep the trunk version stable. However, since most development happens there, breakage is sometimes unavoidable and often accidental. The word stable here implies mainly API stability but also functional stability. Functional stability must be enforced through both automatic and manual testing. If there are no test cases, functional stability becomes really hard to enforce. Therefore automated testing is key (Jenkins).

Violating these rules results in reverting the breaking commits from SVN unless the problems can be resolved otherwise through developer communication.

Integrating Branches

We obviously do not want development to continue eternally in branches so that changes are never merged between branches or integrated back to trunk. Branches are usually private or shared between a group of developers that work on a particular task. The process for integration to trunk is as follows:

  • The Simantics Technical Board (STB) and its weekly meetings will be the place for the developers to advertise and present their changes to the board.
  • The following things need to be documented for the changes in some way:
    • The what and why for the changes
    • Migration instructions from previous version (trunk):
      • A direct patch to fix problems
      • or plain-text instructions
  • When integration is done, send an email to Simantics developers list about the change, linking to the redmine issue(s) in question.

Versioning

In Eclipse/OSGi conventions, components (bundles, plug-ins) are versioned <major>.<minor>.<service>[.<qualifier>].

Our approach to component versioning is as follows:

  • All components are in one of two states: incubation or production
  • Incubation components: <major> = 0, API breakage is allowed between <minor> versions
  • Production components: <major> ≥ 1, API breakage is allowed between <major> versions

Releasing

Before every major (1.0 vs. 2.0) or minor (1.1 vs. 1.2) release, trunk shall be branched into a version branch (e.g. simantics-x.y). The branch is then tested and fixed until it is deemed fit for release by the STB. Fixes are generally merged from other branches into the release branch. Upon release, the branch is tagged as simantics-x.y.1.

A service release (1.1.1 vs. 1.1.2) is a revision of a previous major or minor release that provides important enhancements or bugfixes but is otherwise completely compatible with the previous releases. The service changes are generally merged into the simantics-x.y branch from other branches. Upon release, the branch is again tagged as simantics-x.y.z, where z is incremented from the previous release.

When releases are made, the involved sprints and relevant changes shall be documented in the project's Redmine Wiki.

Our release roadmap is here.

Creating a new plugin

Before a release, the plugin is in incubation phase and there are no quality or other requirements. The repository https://www.simulationsite.net/svn/simantics-incubator may be used to share or co-develop the plugin with other developers, but the plugin must not be included in the features defined in https://www.simulationsite.net/svn/simantics repository.

If the new plugin is a result of refactoring some already existing plugin into two pieces or otherwise is closely related to an other plugin under development, the new plugin can be committed to the same branch of https://www.simulationsite.net/svn/simantics where the other plugin is developed in. Note that whenever this kind of piecing happens, it has the potential to affect the consistency of other components or products. In such cases the refactor should be carried out in branches, not in trunk.

Naming a plugin

If the plugin is intended to be part of the official Simantics release, its name should begin with org.simantics followed by a word that describes the plugin. If the functionality of the same domain is divided into multiple plugins, the name may have more parts. The following suffixes are used:

  • .feature for feature projects.
  • .product A plugin defining a product.
  • <plugin>.ui Provides UI-functionality for <plugin>. Separating base functionality and UI (that depends on SWT) is usually recommendable so that headless applications do not have unnecessary dependencies.
    • Example: org.simantics.simulation, org.simantics.simulation.ui

NOTE! If the the UI is light (e.g. wizards, dialogs, renderers, few external dependencies) the UI code can reside in <plugin>, a separate <plugin>.ui is not required. In such a case, the UI dependencies (to plugins such as org.eclipse.swt) are optional.

Naming in feature projects

Features have two distinct names: project name and feature ID.

  1. The project name is defined in the .project file. It shall contain the .feature suffix since the project name affects how Eclipse checks the project out into workspaces from version control systems.
  2. The feature ID is defined in feature.xml and it shall not contain the .feature suffix.
    • Rationale: current Buckminster resource map (RMAP) definitions rely on this convention for feature project resolution. Having a strict convention keeps RMAPs more uniform and clean.
  3. Feature ID's should not overlap plug-in ID's if at all possible.
    • Rationale: Buckminster commands will not be able to perform actions on features if a plug-in with the same ID exists.
    • Suggestion 1: if you can't think of a unique ID for your feature otherwise, append -feature to it.
    • Suggestion 2: use the feature ID as a base for the IDs of all plug-ins included in the feature. For example, if feature ID is org.example, then plug-in IDs could be org.example.core, org.example.ui, org.example.ontology, etc.

This naming convention exists mainly because it works well with Buckminster.

Modifying a feature

When a plug-in is added to or removed from a feature, the following steps must be performed:

  1. Increment the version number of the feature, primarily the service version number part.
  2. Increment service version of all other features dependent on this feature
  3. Fix .product files that refer to the re-versioned features

This is necessary most of all to keep Eclipse IDE happy and working properly with checked out plug-ins and the P2-based target platforms.

If these steps are not performed, Eclipse will keep using the target platform version of the modified feature not realizing the changes you've made to it. This results often in products failing to work properly or not starting at all.

Publising a standalone plugin

  1. Ensure that the plugin satisfies the following quality requirements:
    • The plugin contains no compilation errors (against which dependencies???)
    • All public classes and interfaces are documented (javadoc)
    • Each public package contains a package-info.java containing package-related comments and annotations (see [1] chapter 7.4)
    • examples/
    • unittests/
    • Version information
    • (optional) Build Script (targets: clean, build, clean-build)
      • [name]_[version]-src.zip
      • [name]_[version].jar
      • [name]_[version].zip
        • doc/
          • manual.pdf
          • changelog.txt
        • examples/
        • unittests/
        • javadoc/ (optional)
        • [name]_[version]-src.zip
        • [name]_[version].jar
      • [name]_[version]-project.zip (Contains everything in workspace)
      • See Example: Non-OSGi
  2. Create a homepage for the plugin in wiki (for example org.simantics.databoard) that contains:
    • Development documentation that is not written in javadoc (basic concepts, components etc.) [We should discuss more about what and how to document] and links to user documentation if the plugin contributes new user interfaces.
    • Change log
    • Plugin roadmap
    • Who are responsible for the plugin
    • (optional) Download to artifacts created by Build script
  3. Commit the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/trunk
  4. Create a tag of the plugin to https://www.simulationsite.net/svn/simantics/<moduleName>/tags/<version>
  5. Inform other developers about the new plugin through the mailing list [simantics-developers@simantics.org]. See also Mailing Lists.

Including a new plug-in to existing features

Inclusion of a plug-in in a feature must always be carefully considered. Consider the following before including anything:

  • Cohesion: Does the plug-in logically belong in the considered feature.xml?
  • Dependencies: What does the new plug-in depend on? Which dependencies are optional? Do the dependencies of the feature's other plug-ins intersect naturally with those of the new plug-in?
  • Permission: do not add anything without permission from the feature maintainer or send a patch and hope it will be merged.

It is highly recommended to have plug-ins included in a single feature only. Instead of including plug-ins in multiple features, prefer creating highly logical and cohesive features that can acyclically include each other. In general completely new functionality should be put in its own feature. If you are unable to decide where a plug-in should be included, ask yourself is the plug-in properly cohesive, are its dependencies minimal?

Note that it is not technically mandatory that each plug-in only resides in one feature - it is simply good practice and will most likely help in training people to use the platform and also make maintainance easier. Having multiple features including the same plug-in will not break builds, but will make the product structure more difficult to understand.

Modifying a feature may break:

  1. Project sets
    • FIX: add a line in the related PSF file(s) for the new plug-in in all project sets that include the modified feature
  2. Jenkins jobs that depend on the feature

Releasing a new version of a component

Here, a component stands for a plug-in, a set of plug-ins or a feature.

  1. Increment version number(s), incl. MANIFEST.MF, plugin.xml, feature.xml, build.xml, homepage.
    • Bug fixes increment service number, API changes minor number and large releases major number.
  2. Run test cases or test suite
  3. Check java docs are compiled without errors. Read Javadocs
  4. Update homepage
  5. (optional) run build.xml and upload generated artifacts to homepage
  6. Create SVN tag: trunk/ → tags/[version]
  7. Inform other developers Simantics developers list.

Making refactorizations that affect other plugins

  1. Plan refactorization beforehand with the maintainers of all affected plugins
  2. Create a new branch for all affected plugins
  3. Refactor, different possibilities
    • One person refactors all plugins
    • Every maintainer refactors his/her own plugins
      • Create rename scripts for other maintainers to help them
  4. Check that everything works
  5. If changes were large inform other developers in the mailing list
  6. Commit plugins to trunk

Developing and maintaining a released plugin

Generally, people need to be able to commit their changes continuously to the SVN location dedicated for a component. Often committing is necessary for transferring their changes to another machine or just to keep their code safe. We can't allow developers committing API changes or broken temporary code into SVN locations that other projects are using - those need to be stable to some degree. Hence, the active development and API breaking must occur in SVN locations that are documented to be unstable and modifiable only by component maintainer(s). These locations shall be released to the general public according to the roadmap of the component. For every release, a tag shall be created. For every release external dependencies need to be documented: component name, version to use (tag). Releases are versioned. Version numbers are composed of three (3) segments: 3 integers and a string respectively named major.minor.service. Read [2] and [3].

  1. the major segment indicates breakage in the API
  2. the minor segment indicates "externally visible" changes
  3. the service segment indicates bug fixes and the change of development stream (the semantics attached to development stream is new to this proposal, see below)

By default, the following SVN layout should be used for components:

component/
trunk/
  • The main development branch, unstable by nature. This version should only be used by active developers of this component.
branches/
  • Branches should be created:
    • when preparing for a release, for stabilizing the component while letting active development run along in trunk.
    • for working on bug-fix releases.
    • when working on larger changes, trying out things or working with other developers. These kinds of branches should either be integrated back to where they were branched from or removed as obsolete.
tags/
  • Released versions of the component. Other developers and project-sets should be using these for development if the component isn't already deployed into the target platform or it is necessary to have the code in your workspace for some other reason.

Developers may initially feel that creating tickets for every measly issue is too time consuming but it is necessary to make any kind of change tracking possible for releases. Obviously during the initial development of a component the code changes a lot and all the time. This is exactly why we have an incubator for starting projects that does not impose these strict rules but is more like a playground where the developer can freely "hack around" disregarding all these policies. A good developer will get started on design, testing and documentation also right from the start. When the developer feels his component is ready for consumption of the general public and the component passes peer review, it shall be ceremonially transferred from incubation into the official project. Each official component should have documentation, unit tests and possibly examples or really good reasons for not having any.

Modifying the target platform

DEPRECATED: these instructions are mainly deprecated with the current Git-based system

The target platform is a composition of features and plug-ins that contains all the bundles that you as a developer can build your code against when starting from a blank table. In Simantics, it is a mixture of the standard Eclipse RCP platform, other general Eclipse components and some Simantics components.

It makes sense to deploy into the target platform only if your plug-ins:

  • are general, i.e. reusable by many other components
  • have little dependencies
  • are stable, both by API and functionally

Deploying plug-ins into the target platform will most likely break:

  1. Project sets
    • FIX: remove lines matching the plug-ins that were moved into the target platform

Deploying plug-ins for the target platform

The Simantics target platform is a P2 repository located at

where _version_ is one of head, 1.11, 1.10, etc. head is trunk.

The repository is built using the buckminster build definition projects at svn:releng/sdk/branches. Ultimately, these builds use the feature project at svn:workbench/trunk/org.simantics.sdk.site.feature/ to define which features and plug-ins are part of the platform.

To include new features or plug-ins into the platform trunk development version:

  1. Make your code/data available as plug-ins and/or features in a public repository, e.g. the Simantics repository.
  2. Ask the target platform maintainer to include your components in the platform resource maps (see Buckminster#Resource_Maps).
  3. Include them in org.simantics.sdk.site feature include them into the platform builds.

Jenkins build target-simantics-head-sdk takes care of building and publishing the trunk platform. Other branches have similar builds visible here.

Building Products

The best way to currently build products is to use the Eclipse tool Buckminster for the job.


Pre-requisites
  1. You have a product configuration (.product file) for your product.
  2. Your product configuration is based on features, not plug-ins
  3. Buckminster locally installed or a jenkins installation with buckminster available


Example

To create your own (headless) buckminster product build follow these steps:

  1. Export svn:tutorials/trunk/com.acme.movie.product.site.feature as a template for making your own headless buckminster product build.
  2. To customize for your purposes, the following contents need to be configured:
    1. feature.xml
      • Change ID, version, name, provider and everything else to match your purposes.
      • Include all the features your .product file depends on, nothing else.
    2. Resources.rmap
      • Define mappings for all resources (features and plug-ins) needed by your product. See Buckminster definitive guide for help on defining resource maps.
    3. .credentials.properties
      • Insert any credentials needed to access resources defined in your resource map.
    4. site.cquery
      • Modify cq:rootRequest name to match your site feature ID.
    5. jenkins.buckminster.script
      • Replace com.acme.movie.product.site with what ever is your site feature's ID.
      • Replace p2.profileName with a new profile name for your product.
      • Replace p2.rootId to match the uid attribute defined in your Product Configuration file (.product).
      • Set build.label to what ever you want your built product .zip files to be named like.

Using Continuous Integration (Jenkins)

  1. Install jenkins (see simantics.org jenkins)
  2. Create new free-style software project job with appropriate name
  3. Configure the project:
    1. In the Source code management section, use your version control system of choice to checkout the ...product.site.feature you've created for your build in the previous section. For the com.acme.movie example the proper settings would be:
    2. In the Build section:
      1. Add new Run Buckminster build step
      2. Select a Buckminster installation to use (latest)
      3. Copy the contents of jenkins.buckminster.script into the commands text box
    3. Last, you'll want to archive the vital artifacts produced by your build. In the Post-build Actions section:
      1. Check Archive the artifacts
      2. Set value of Files to archive to buckminster.output/com.acme.movie.product.site_*-eclipse.feature/*.zip (obviously changing the value according to the name of your product site feature).


That should do it. Further configuration of the build is up to you, i.e. scheduling, parsing warnings, and so on.

External links