Org.simantics.scenegraph

org.simantics.scenegraph (SVN) is a scene graph library for 2D rendering with a focus on supporting remote rendering using a client-server architecture.

Overview

org.simantics.scenegraph provides a generic serializable scene graph datastructure ( wikipedia:Scene graph ) for displaying 2D and 3D graphics.

The idea of scene graph is that it is an independent datastructure that can be rendered as such. It can only access local variables, thus it provides a strict interface between the visualization and program logic. Mainly for this reason scene graph is an ideal datastructure for rendering graphics remotely.

Architecture

org.simantics.scenegraph is designed to be used for rendering graphics locally on desktop application, but also remotely from server to client. For this reason, it has some features that a normal scene graph implementation doesn't have.

In org.simantics.scenegraph, the scene graph datastructure is a tree consisting of ParentNodes and Nodes.

• Node is a leaf in the tree, thus it cannot have children, but it should be used to render visible objects.
• ParentNode is used to group a set of nodes (children) and to give some common parameters for the children.

To enable scene graph datastructure synchronization from a server to a client, a mechanism for detecting changes in the datastructure is required. For this purpose, all changes to Node variables must be made through a public method (usually setter). These methods that change local variables are marked with annotations, thus the solution is very similar to what Hibernate uses with entities.

Node methods can have four different annotations, three for using scene graph remotely, and one for synchronizing node variables automatically from the graph:

• SyncField(..) When a method annotated with this is called, the fields listed in the annotation are updated to the client side (if running in remote mode)
• ServerSide Methods with ServerSide annotation are intended to be executed on server-side, thus it can access resources outside the scenegraph datastructure (using interfaces)
• ClientSide Methods with this annotation are executed on client-side, thus it should only access resources inside the scenegraph datastructure
• PropertySetter This annotation is used to map graph properties to node automatically. This annotation can only be used in setter methods, that take one argument. Notice that the argument type must match with the object that is read from the graph (i.e., for double values you must use Double type instead of double, float, or Float).

Without these annotations, the Node will not work properly when using scenegraph remotely.

These annotations (except PropertySetter) are handled with cglib MethodInterceptor which is a proxyclass for the actual class implementation, and contains an interceptor for performing actions before and after the actual method is called. For this reason, the Node classes should not be instantiated directly, but through ParentNode.addNode method. Cglib support is implemented in org.simantics.scenegraph.remote plugin, and thus org.simantics.scenegraph plugin does not depend on cglib.

ParentNode and Node classes implement only functionalities for creating and modifying the scene graph datastructure but do not contain any methods for rendering.

2D

Java 2D API compatible implementation of the scene graph is in org.simantics.scenegraph.g2d package:

• G2DNode is a baseclass for all leaf nodes that perform 2D rendering
• G2DParentNode is a baseclass for the 2D compatible parent nodes
• IG2DNode is a common interface for G2DNode and G2DParentNode which should be used when traversing through the data structure.
• G2DSceneGraph G2DParentNode implementation containing event listeners and some extra features. This should always be the root component of a G2D scene graph datastructure.

Example G2DNode:

public class BackgroundNode extends G2DNode {
    private static final long serialVersionUID = -1L;	
    protected Rectangle2D bounds = null;
	
    @SyncField("bounds")
    public void setBounds(Rectangle2D bounds) {
        this.bounds = bounds;
    }
	
    @Override
    public void render(Graphics2D g) {
        g.fill(bounds);
    }
}

Because org.simantics.scenegraph is an independent scene graph datastructure, it can be rendered as such. Rendering can be performed by calling G2DSceneGraph.render(Graphics2D g2d) method. No changes are allowed to the scene graph while the structure is rendered (and rendering methods should not make any changes to the structure either, excluding transient node fields that are not synchronized between server and client).

3D

JOGL compatible implementation of the scenegraph can be found under package org.simantics.scenegraph.g3d.jogl in a separate plug-in org.simantics.scenegraph.g3d.

• JOGLNode
• JOGLParentNode
• JOGLPaintable
• JOGLSceneGraph

See org.simantics.scenegraph.g3d.eclipse.JOGLLocalView for an example how to use org.simantics.scenegraph with OpenGL.

Serialization

For serializing the scene graph datastructure there is class called org.simantics.scenegraph.remote.Serializer. The Serializer simply listens for any changes in the scene graph, and translates those changes into events which are then serialized and sent to ObjectOutputStream. To enable Serializer to listen for changes in the scene graph, the scene graph root node must be initialized with cglib. For this purpose there is a static method in Serializer:

 G2DSceneGraph root = Serializer.newRemoteSceneGraphNode(G2DSceneGraph.class);
 Serializer serializer = new Serializer(root, input, output);
 serializer.start();

After this, the root node can be used normally. Any child added to it will be created by cglib automatically, hence Serializer works automatically with these whole scene graph data structure.

On the client side, Serializer can be directly attached to object streams, and it will create G2DSceneGraph object for you:

 Serializer serializer = new Serializer(input, output);
 serializer.start();
 G2DSceneGraph root = (G2DSceneGraph)serializer.getRoot();

Because the system works asynchronously, you should actually attach UpdateListener to serializer in real case.

Notice that you have to attach the serializer to the scene graph before any other operation. Serializer is an instance of PropertyChangeListener which will be set to listen the changes in the scene graph datastructure. If there are changes made before the serializer is attached, these changes will not be serialized.

Serializer uses ObjectInput and ObjectOutput to serialize and unserialize the scene graph, thus it can be used to write the scenegraph to a file and from a file, or to synchronize the scenegraph in a client-server system.

Using scenegraph

• Create scenegraph

G2DSceneGraph sg = new G2DSceneGraph();
sg.setRootPane(instanceOfJComponent); // In case you want to use swing components, you must set rootPane.
// Register event handlers
addKeyListener(sg);
addMouseWheelListener(sg);
addMouseListener(sg);
addMouseMotionListener(sg);
addFocusListener(sg);

• Create node

ShapeNode shape = sg.addNode(ShapeNode.class);
shape.setShape(new Rectangle2D.Double(10, 10, 100, 100));
shape.setColor(Color.RED);

• Render

sg.render(g2d);

• Move shape

shape.setTransform(AffineTransform.getTranslateInstance(5, 5));

• Updating node from stateless object

ShapeNode shape = sg.getOrCreateNode("bounds", ShapeNode.class);
shape.setShape(new Rectangle2D.Double(10, 10, 100, 100));
shape.setColor(Color.RED);

Scenegraph in Simantics

• Each canvas participant that wants to draw something should implement methods annotated with @SGInit and @SGCleanup

public class Participant extends AbstractCanvasParticipant {
   protected ShapeNode node = null;

   @SGInit
   public void initSG(G2DParentNode parent) {
       node = parent.addNode(ShapeNode.class);
       node.setZIndex(PAINT_PRIORITY);
       node.setShape(new Rectangle2D.Double(1, 1, 20, 20));
   }

   @SGCleanup
   public void cleanupSG() {
       node.remove();
       node = null;
   }
}

• Optionally, you can specify a designation for @SGInit that specifies how you want your nodes to be transformed, i.e. in control space or in canvas space. The default value is canvas space. Use the scene graph viewer to understand how this affects your scene graph.

@SGInit(designation = SGDesignation.CONTROL)

• Each diagram element can implement interface SceneGraph to give their own contribution to the scene graph.

public class ElementSGNode implements SceneGraph {
   public static final Key SG_NODE = new SceneGraphNodeKey(Node.class, "SG_NODE");

   @Override
   public void init(final IElement e, G2DParentNode parent) {
       ShapeNode node = (ShapeNode) e.getHint(SG_NODE);
       if (node == null) {
           node = parent.addNode(ShapeNode.class);
           e.setHint(ElementHints.KEY_SG_NODE, node);
       }
       e.setShape(new Rectangle2D.Double(1, 1, 20, 20));
   }

   @Override
   public void cleanup(IElement e) {
       Node node = e.removeHint(SG_NODE);
       if (node != null)
           node.remove();
   }
}

org.simantics.scenegraph features

org.simantics.scenegraph package contains lots of readymade nodes and features. For a list of g2d nodes, see org.simantics.scenegraph.g2d.nodes.

Swing support

It is possible to integrate Swing components to 2D scenegraph by using org.simantics.scenegraph.g2d.nodes.ComponentNode class. See for example MonitorNode or Trend2DNode for an example.

Eclipse view

org.simantics.scenegraph.eclipse package contains local and remote view of 2D and 3D scene graphs. You can try out these demos by executing the project as Eclipse Application and choosing the views "SceneGraph Remote View", "SceneGraph Local View", "JOGL SceneGraph Local View", "JOGL SceneGraph Remote View".

2D-3D integration

It is possible to create a G2DParentNode that works as a wrapper between 2D and 3D scenegraphs. This is not implemented yet.

Node Bounds

In order to perform any kind of rendering optimization in the future some kind of spatial subdivision methods need to be employed. For example Trolltech QT GraphicsView [1] implements BSP-based partitioning to accelerate interaction and rendering in its 2D systems. Bounds information is also useful for picking facilitation.

Currently scene graph nodes do not have any kind of bounds information, but the ideas to implement bounds checking will be written under this section.

Implementation

• Each node has a method getBoundsInLocal() which returns a Rectangle2D instance describing the 2D boundaries of the node in the element's local coordinate system.
• Node base implementation classes have a method getBounds() that returns the node boundaries with local transforms applied.
• Nodes that have children (i.e., a parent node that does not draw anything) will return union of its childrens bounds.
• G2DSceneGraph root node will cache bounds information.
• Bounds will need caching in order to be efficiently implemented in parent nodes. One option is to implement a mechanism for propagating transformation changes up in the node tree to keep parent nodes aware of transformation changes in their children. Possily implement a listener in parent nodes that can be registered into any transformable node.

Questions

How to implement Sysdyn -scenario

In this scenario, diagram consists of nodes and arcs between them. Nodes are text boxes that scale according to the text inside them. Arcs go from the center of one node to another so that it clipped to the edges of the nodes. An arrow is drawn in the head of the arc (at the point where the arc is clipped). The following editing operations are supported

1. Changing the name of a node
2. Moving a node. The arcs should be updated during the operation.
3. Changing the curvature of a arc. This is done by dragging some point on the arc to other point.

See [2]. The operations should be reponsive so an implementation that does not need server for updates during the operations is preferred.

Specific questions:

• Can nodes refer to each other? For example, can arc be implemented as

public class ArcNode extends G2DNode {
    protected IConnectable tail = null;
    protected IConnectable head = null;
    protected double angle = 0.0;

    @Override
    public void render(Graphics2D g) {
        calculate an arc from tail to head with given angle
        clip the arc to the clip bounds of the tail and the head
        (or check if connectables have been changed and update the arc if necessary)
    }
}

where

public interface IConnectable {
    double getX();
    double getY();
    Rectangle2D getClipBounds();
}

and

public class TextNode extends G2DNode implements IConnectable { ... }

...answer

  In theory, nodes could refer to each other. However, after serialization the references wouldn't work, thus the references must be id based:

  public class ArcNode extends G2DNode {
    protected String tailId = null;
    protected String headId = null;
    protected double angle = 0.0;

    @SyncField("tailId")
    public void setTailId(String id) {
        this.tailId = id;
    }
    @SyncField("headId")
    public void setHeadId(String id) {
        this.headId = id;
    }
    @SyncField("angle")
    public void setAngle(double angle) {
        this.angle = angle;
    }
    @Override
    public void render(Graphics2D g) {
        calculate an arc from tail to head with given angle
        clip the arc to the clip bounds of the tail and the head
        (or check if connectables have been changed and update the arc if necessary)
    }
}

• How derived fields in the nodes are updated when server updates the client-side node?

public class TextNode extends G2DNode implements IConnectable {
    // The properties of the node
    protected double centerX = 0.0;
    protected double centerY = 0.0;
    protected String label = "";

    // Auxiliary values computed in updateDerivedFields
    protected Rectangle2D clipBounds = null;	   
    protected float textX;
    protected float textY;

    private void updateDerivedFields() {
        ...
    }
}

...answer:

The methods in IConnectable should be actually implemented by every G2DNode (Hence, we might combine these interfaces). Notice that you cannot use serialization annotations with private methods.
public class TextNode extends G2DNode implements IConnectable {
    // The properties of the node
    protected double centerX = 0.0;
    protected double centerY = 0.0;
    protected String label = "";

    // Auxiliary values computed in updateDerivedFields
    protected Rectangle2D clipBounds = null;	   
    protected float textX;
    protected float textY;

    @SyncField({"clipBounds, textX, textY"})
    protected void updateDerivedFields() {
        // Update fields, but create clone of clipBounds to enable comparison between old and new value
    }
} 

• How the operation 3 is implemented in client-side? For example, when an arc node notifies that it is being dragged, it should prevent other nodes from getting any mouse events. It should also have some place to store the state of the current operation so that this kind of transient structures do not have to be stored to the nodes. In the applet, this is implemented as:

   class Mover implements ActionHandler {
       public void handleDrag(double x, double y) {
           angle = Arcs.angleOfArc(tail.getX(), tail.getY(), x, y, head.getX(), head.getY());
       }
       public void handleRelease(double x, double y) {}        
   }

   @Override
   public ActionHandler handlePress(double x, double y) {
       double dx = x-cx;
       double dy = y-cy;
       double dist = dx*dx + dy*dy;
       if(dist < (r+SELECTION_TOLERANCE)*(r+SELECTION_TOLERANCE) &&
           dist > (r-SELECTION_TOLERANCE)*(r-SELECTION_TOLERANCE)) {
           double angle = Arcs.normalizeAngle(Math.atan2(-dy, dx));
           if(Arcs.areClockwiseOrdered(angle0, angle, angle1))
               return new Mover();
       }
       return null;
   }

...answer

public class ArcNode extends G2DNode {
   protected double angle;
   protected boolean drag = false;

   public void handleEvent(AWTEvent event) {
       if(!(event instanceof MouseEvent)) return;
       MouseEvent me = (MouseEvent)event;
       if(me.getID() == MouseEvent.MOUSE_PRESSED) {
           double dx = me.getPoint().getX()-cx;
           double dy = me.getPoint().getY()-cy;
           double dist = dx*dx + dy*dy;
           if(dist < (r+SELECTION_TOLERANCE)*(r+SELECTION_TOLERANCE) &&
               dist > (r-SELECTION_TOLERANCE)*(r-SELECTION_TOLERANCE)) {
               double angle = Arcs.normalizeAngle(Math.atan2(-dy, dx));
               if(Arcs.areClockwiseOrdered(angle0, angle, angle1))
                   drag = true;
           }
           me.consume();
       } else if(me.getID() == MouseEvent.MOUSE_DRAGGED && drag == true) {
           G2DNode tail = findNode(tailId); // NOTE: findMode not implemented yet
           G2DNode head = findNode(headId);
           angle = Arcs.angleOfArc(tail.getX(), tail.getY(), me.getPoint().getX(), me.getPoint().getY(), head.getX(), head.getY());
           me.consume();
       } else if(me.getID() == MouseEvent.MOUSE_RELEASED) {
           drag = false;
       }
   }
}

Development Ideas

Node ⇔ ID mapping

(ticket:871)

Just as in DOM implementations the client has the possibility to search for document nodes by their ID, linking between scene graph nodes could be achieved through the use of scene graph -locally unique ID's. Just as in DOM, the id attribute is not mandatory. Lookup will only be possible if an ID is provided.

A lookup facility could again be made available in the scene graph root node. Considering performance, it may become a bottleneck if all ID references would need to first traverse to the scene graph root node to use the ID lookup facility. One way to alleviate this problem is to have a transient root node cache in each node. Obviously this will make the scene graph more space-consuming.

Implementation suggestion #1:

• Add interface ILookupService:
  • NOTE: the lookup must be bijective (ID <=> INode)
  • void map(String id, INode node);
  • void unmap(String id);
  • void unmap(INode node);
  • INode lookupNode(String id);
  • <T extends INode> T lookupNode(String id, Class<T> clazz);
  • String lookupId(INode node);
• Add ParentNode<?> INode.getRootNode() for allowing optimization of implementation for getting the scene graph root node.
• Add ILookupService NodeUtil.getLookupService(INode node):
  • ParentNode<?> root = node.getRootNode();
  • return root != null && (root instanceof ILookupService) ? (ILookupService) root : null;
• Add INode NodeUtil.lookup(INode node, String id):
  • ILookupService lookup = getLookupService(node);
  • return lookup != null ? lookup.lookupNode(id) : null;
• Make G2DSceneGraph implement ILookupService
• Add get/setLookupId(String) to INode, annotate setLookupId with @SyncField("lookupId")
• Implement getLookupId():
  • return NodeUtil.getLookupService(this).getId(this);
• Implement setLookupId(String):
  • NodeUtil.getLookupService(this).map(lookupId, this);
• When nodes are removed from the scene graph, possible lookup mapping must also be removed
  • Where exactly should this be performed ??
• Optimization:
  • transient ParentNode<?> cache for the root node in the basic node implementation for quicker access to the root node

TODO

• decide threading model of ILookupService
• specify exactly how removal works

Shared Context

A scene graph in itself forms a kind of context. It would sometimes be useful to be able to store context local data accessible to any node within that graph, e.g. for having different kinds of caches in the graph.

One way to implement this would be to add a data map to the scene graph root node (SceneGraphNode) which can be made available to all nodes by traversing to root.

Evaluation

Not useful as a separate mechanism

The same effect can be achieved by just using the basic scene graph mechanisms by adding non-renderable data nodes to the graph, which can be found by either browsing the scene graph from to and from the root node or through the mechanism described at #Node_.E2.87.94_ID_mapping.

Template Nodes

It is very common in rendering of graphical scenes that a single type of graphical entity is rendered multiple times with different parametrisations, i.e. instantiated.

For the scene graph this would mean that it needs to be able to store the subgraph of an instantiated element and allow it to be referenced from the element instance graph locations. Also a mechanism for separating static data and instance data would be needed.

Evaluation

Node ⇔ ID map makes this possible

The evaluation of #Shared Context applies for this also.

Basic scene graph modification operations

• Add child node
• Get or create node (Either get child by id, or add new child and return it)
• Delete subtree (Remove node and its children)
• Delete node (Delete node from tree tree and move its children to its parent)
• Remove all children
• Append parent node (Append new node between specific child and parent)

Links

• org.simantics.g2d
• org.simantics.diagram
• cglib
• SVG Salamander

Current Development

• Open Tickets
• Closed Tickets

See also

• TP:Animated Symbols

Contact

juha-pekka.laine@semantum.fi