Multitouch Surface support for XNA-based applications is provided via the Microsoft.Surface.Core.Manipulations namespace in an event-driven model. I was interested in investigating general multitouch processing within a game-loop model, i.e. not involving events, and thought it would be instructuve to develop my own class to process manipulations in this way.

In a similar manner to the Affine2DManipulationProcessor, I wanted to support arbitrary combinations of translation, rotation, and scale. Translation is trivial, since it is simply a matter of tracking the changes in each manipulation point between calls and taking the average.

Rotation and scale is a little more tricky, since each of these values are relative to a centre-point, or manipulation origin. Rotation is calculated from the average angle between each point and the origin, and scale is calculated from the average distance. Unlike a mouse cursor, manipulation points come and go, so the key is only to track changes to those points which persist between calls, and then update the positions, origin, distance and angles for the next call.

In order to generalise manipulation points from multiple sources such as mouse, gamepad, Surface Contacts and .NET 4.0 Touch, I created my own Manipulation structure to abstract properties such as ID and position. My input processing can then build a collection of these objects from any relevant source(s) and pass them to my manipulation processor as part of the Update call as follows:

myManipulationProcessor.ProcessManipulators(myManipulators);

The cumulative transforms for translation, rotation and scale (according to the supported manipulation types specified) are then immediately available as properties on the manipulation processor.

In order to test the solution, I wrote a small harness to visualise the manipulation points, and their effect on a reference cross-hair. While I also added support for a mouse (right-click to add/remove points, left-drag to move points), multi-touch hardware is required to test scenarios when multiple points are added, moved, or removed simultaneusly. A screenshot is shown in Figure 1.

Figure 1. Multitouch manipulation with averaged manipulation origin.

One of the complexities of working with multiple manipulation points is deciding how to determine the manipulation origin. One option is to simply take the average position of each point, as shown in Figure 1. Another option is to introduce a speicific "pivot point" as the origin and use this to calculate scale and rotation. This pivot point can either be fixed, or tracking the object being manipulated. The latter two options are shown in Figures 2 and 3.

Figure 2. Multitouch manipulation with fixed pivot point.

Figure 3. Multitouch manipulation with pivot point centered on manipulation object.

The solution is demonstrated in the following video. Each approach for determining the manipulation origin is illustrated using translation, rotation, and scaling manipulations, initially in isolation and then in combination.

There's still some tweaking to do on my radial controls discussed in Part 1 and Part 2 of a previous post, but I wanted to integrate them into the project. I've included a couple of screenshots below. Note that the size of the control has been enlarged for clarity and that the menu structure is purely for test-purposes.

Figure 1. Radial control for boolean settings.

Figure 2. Radial control for numeric settings.

I decided to render all of the "focus arcs" around the edge of the control, highlighting the relevant arc as appropriate, so that the pointer for numeric values didn't "float" in space when it wasn't adjacent to the currently focussed segment.

The control allows me to define a hierarchical menu structure and update properties using reflection, so it's trivial to add another entry. I can also define min/max ranges, format strings, rates of change when using the keyboard etc. For example, the menu item for "zoom" shown in Figure 2 looks like this:

new MenuItem
{
    FormatString = "Zoom\n{0}",
    PropertyInfo = typeof(Camera).GetProperty("Zoom"),
    Min = 0.0f,
    Max = 45.0f,
    Rate = 1.0f,
    TickFormatString = "0.0",
    ValueFormatString = "0.00",
}

In Part 1 I showed some screenshots of a radial dial for controlling settings. I've made the following visual changes:

• Removed the central buttons and to allow a Surface tag to control the dial position and orientation
• Adjusted the size and fonts to use a tag 39mm in diameter (a poker chip)
• Added a label indicating current category above the dial which can also be used to drag the dial with the mouse
• Added a back button to the last (clockwise) segment to navigate back to the previous category
• Added a ring to the centre of the dial to indicate dial focus when more than one dial is present. The arc segments around the outside indicate item segment focus for a given dial
• Added highlighting to show selected or pressed item states

Figure 1. Radial control for boolean settings

The planetary bodies wouldn't be complete without their rings. Whilst Saturn has the most spectacular system, Jupiter, Uranus and Neptune also possess systems of their own. I chose to start with Saturn, and Figure 1 shows an initial screenshot.

Figure 1. Initial rendering of saturn ring system.

I had to consider the following points, and will discuss each in turn:

1. Mesh
2. Texture & Lighting
3. Size & Orientation
4. Shadows

Mesh

I implemented a simple algorithmic disc mesh. For the planetary bodies I can use a single mesh, scaled appropriately. However for each instance of a ring system I generate a seperate mesh since both inner and outer radii can change. In the future I may look at using a single mesh and using vertex displacement in the vertex shader to size the ring appropriately.

Texture & Lighting

I chose to start by lighting the ring equally from each side, though in reality the rings look different from above and below. Using a diffuse, per-pixel shader gives a term such as the following, where the absolute value gives the same lighting from above and below.

float diffuse = saturate(abs(dot(direction, normal)));

In order to be able to see through the gaps in the ring system, I blended the rings using the following render states:

SourceBlend = Blend.SourceAlpha;
DestinationBlend = Blend.InverseSourceAlpha;

There are numerous textures available for Saturn's rings. I took a thin slice for a texture map, and added an alpha channel corresponding to the ring transmittence.

Size & Orientation

I added a further xml file for the physical data on a ring system, and used it to size the ring mesh and apply the appropriate texture.

In order to actually see any light on the ring system, I also had to add the correct obliquity (axial tilt) to the planet, available from the NASA JPL Horizons system. This parameter also needed to be applied to the positions and orbits of the planet's moons.

Shadows

The final piece for my first iteration on the ring system was to add shadows. Shadows are cast from the planet onto the rings, and also from the rings onto the planet. For the former, when rendering the ring I implemented a ray-sphere intersection algorithm to add shadow whenever a ray fired toward the sun intersected the planet. For the latter, when rendering the planet, I implemented a simple ray-plane algorithm to add shadow whenever a ray fired from the planet intersected the ring.

Up to this point, I've had numerous settings which I can control using a keyboard-driven menu. One of the key things I've been looking at recently is to "touch-enable" these settings, so that a keyboard is not necessary. This will enable interaction via touchscreen, Microsoft Surface etc.

There are several different types of settings, including:

• Boolean values
• Floating-point values
• Integer values

The control had to support the following:

• Different data types
• Multidirectionality
• Multitouch manipulations, at least for translation and rotation
• Multiple simultaneous instances of the control
• Keyboard support with "tab stops"

Boolean Values

For boolean values, I've used a "radial" control which renders an arbitrary number of segments which can be selected/deselected. An early version is shown below in Figure 1.

Figure 1. Radial control for boolean settings

Floating-Point Values

When floating-point values are selected, I add a radial scale and a dial which can be rotated to set the appropriate value. An early version is shown below in Figure 2.

Figure 2. Radial control for float settings

Navigation

In order to navigate a "tree-view" of settings, the current "category" is displayed, along with a "Back" button to navigate back to the parent category. Child categories are shown as segments on the menu, along with settings for the current category. The currently-selected setting is indicated by a "focus" arc, and "pressed" buttons can be highlighted using different colors and opacities.

The Vector3 struct uses 32-bit floating-point values. On a solar-system scale, this does not have sufficient precision to accurately position objects such as the far outer planets.

A single-precision (32-bit) floating-point value stores 7 decimal digits of precision. Neptune, for example, has an approximate distance from the sun of 4.5x10⁹km. Expressed with 7 significant figures, this has an error of approximately 500km (one half of the value of the last significant place). For Pluto (although no longer classified by the IAU as a planet), at a distance of approximately 5.9x10⁹km, 500km is approximately one quarter of its diameter! Clearly single-precision floating point numbers are not sufficient to store these positions to the required level of accuracy.

One option is to use double-precision (64-bit) values, which store 15 decimal digits of precision. Most modern GPUs remain incapable of performing double-precision arithmetic, however this level of precision is only required to calculate the position of the object relative to the camera. This can be done on the CPU, after which point single-precision calculations are just fine, as shown below in Figures 1-3.

Figure 1. Uranus and moon orbits

Figure 2. Neptune and moon orbits

Figure 3. Pluto and moon orbits

An example struct for dealing with double-precision vectors is as follows:

public struct Vector3Double
{
    double _x, _y, _z;

    public double X { get { return _x; } set { _x = value; } }
    public double Y { get { return _y; } set { _y = value; } }
    public double Z { get { return _z; } set { _z = value; } }

    public Vector3Double(double x, double y, double z)
    {
        _x = x;
        _y = y;
        _z = z;
    }
}

Using operator overloading it is possible to work with Vector3Double the same way as Vector3, for example:

public static Vector3Double operator -(Vector3Double vector1, Vector3Double vector2)
{
    return new Vector3Double(
        vector1._x - vector2._x,
        vector1._y - vector2._y,
        vector1._z - vector2._z);
}

Conversions, for example to and from Vector3 can also be easily supported, for example:

public static implicit operator Vector3(Vector3Double vector)
{
    return new Vector3(
        (float)vector._x,
        (float)vector._y,
        (float)vector._z);
}

Note that while double-precision solves the problem to the levels of accuracy described here, a better long-term option will be to use fixed-precision types such as 64-bit integers.

The International Astronomical Union (IAU) defines an "official" set of boundaries for the 88 constellations. I've now added these.

What I initially saw as a bug turned out to result from the fact that Serpens is actually divided into two boundaries (all other constellations have a single boundary).

Figures 1-3. Constellation Boundaries

I've started looking into adding some deep sky objects such as star clusters and galaxies, and have been thinking about the following:

• Loading the data.
• Sprite markers for the position, size and type of deep sky object.
• Rendering a text label at the appropriate position and scale.

Loading Data

Once again there is plenty of data available in the public domain. I found some basic lists of objects to get me started, converted the data to XML and used the automatic content serialization in XNA Game Studio 3.1 to compress the XML to an XNB file, as per my Background Stars.

Sprite Markers

Adding sprites in the appropriate positions was relatively straightforward. In order to specify the size of the object, I created a custom VertexElement structure, VertexPositionColorSize, which let me set a VertexBuffer with all the data the shader needed to position, color and scale the sprite.

Rendering Text

The labels were more interesting in that there were several options. I had the following requirements:

• I wanted the labels to render efficiently, as there may be many instances visible at once.
• I was considering scaling the labels as a size/proximity dimension.
• I didn't need to use the depth buffer since these labels would always be at the "back".

With these requirements in mind, the following approaches were apparent:

• Use SpriteBatch.DrawString() with SpriteFonts.
• Using line primitives to render vector-based fonts.

The SpriteBatch approach is shown below in Figure 1. While not ideal for scaling labels to large sizes (the fonts are drawn as scaled bitmaps), the approach proved surprisingly performant.

Figure 1. Deep Sky Objects from Messier Catalog with SpriteBatch text

It is possible to extract vector definitions from the "Modern", "Roman", and "Script" Windows Fonts. I took the approach of pre-generating the line primitives for a given label at load time and passing them to the shader with a given scale and offset per frame. This proved slightly more performant than the SpriteBatch approach, and provided a good option for scaling the text to large sizes without occluding any objects in the background. Examples are shown in Figures 2 - 4.

Figure 2. Globular and Open Clusters with scaled PointSprites and Vector Fonts

Figure 3. Globular Clusters with Right Ascension, Declination Grid

Figure 4. Globular and Open Clusters with Background Stars and Earth

Note that the Deep Sky Objects are shown at an exaggerated scale for clarity, and without anti-aliasing.

One of the things I needed to start thinking about sooner or later was supporting High Dynamic Range (HDR) render targets, initially for the following reasons:

1. I wanted to use a realistic bloom effect for the sun, without affecting other rendered planetary bodies.
2. Atmospheric scattering algorithms lead to a wide range of luminance values.
3. Background stars span a wide range of luminance values.

The use of HDR in combination with Tone Mapping to normalise luminance values into a standard render target would allow me to deal with each of these issues. This post will focus on the just the first issue of rendering the sun.

Non-HDR Approach

There are a number of approaches to rendering bright objects. One example is the lens flare sample on the XNA Creators Club site, which uses occlusion queries on the GPU to render glow and flare sprites when the sun is visible. Figure 1 shows the use of this technique.

Figure 1. Non-HDR Sun effect using Occlusion Queries with Glow and Flare Sprites

HDR Approach

Another approach to rendering bright objects is based on image post-processing. The bloom sample on the XNA Creators Club is a good model for implementing bloom post-processing effects, along with additional control over color saturation. I combined this approach with an HDR render target (I used SurfaceFormat.HalfVector4), and added some HDR-textured spheres to represent bright lights. I rendered my scene as per normal and, using a post-processing bright-pass filter, extracted pixels for which the color range fell outside of the normal (0-1) range. I then used several passes of a Gaussian blur filter to create "star" and "bloom" effects and combined this with the original image, as shown in Figure 2.

Figure 2. HDR Post-Processing Test. Note that the "lights" are not being used in any lighting calculation for the central "planet" (a simple ambient value is applied to its texture), since this test was simply to demonstrate post-processing effects.

I then applied this post-processing approach as an alternative method to rendering the sun, as shown in Figure 3 below.

Figure 3. Sun effect HDR Post-Processing effects

Combinations of both approaches can also be used, as per Figure 4 below.

Figure 4. Sun effect using both HDR Post-Processing and Occlusion Query Sprites

Using HDR render targets is an expensive process, even for many current GPUs. However, it has a number of advantages over the Occlusion Query approach, such as:

• Rendering one or many HDR-textured items in a post-processing pixel shader has the same GPU cost, unlike running multiple occlusion queries.
• Since post-processing effects are pixel-based, this approach leads to more realistic results when HDR-textured items are partially occluded.

In Part 1 I showed some screenshots of a planet rendered using multiple effects. I'll discuss each of these in turn, and begin with a composite image to show how each effect contributes to the overall image.

Figure 1. Composite image showing per-pixel, single-direction lighting (top left), addition of bump-mapping and specular (Phong) reflection (top right), addition of atmospheric scattering (bottom left), and addition of clouds and cloud shadows (bottom right).

The first thing I needed to do was create a model. I could have used a predefined sphere, but chose instead to generate the mesh algorithmically so that I could easily control the number of vertices.

Texture

Once I had a model, the first thing I needed to do was to apply a texture. NASA has an extensive image library, and the Visible Earth site has a collection of land maps for Earth. These maps are Equidistant Cylindrical projections, so my texture coordinates were simply:

x = λ
y = θ

where

λ = longitude,
θ = latitude

Lighting

The Shader Series on the XNA Creators Club site is a great introduction to lighting. My initial lighting model was a simple per-pixel shader, with a single directional light source from the sun. I subsequently added specular reflection using a Phong shading algorithm.

Relief

In order to show surface features without significantly increasing the number of vertices in the planet model, bump (normal) mapping can be used. There are numerous sources of normal maps on the Internet, available in various formats (I'm using DDS), and a good sample of how to implement normal mapping in a shader can be found on the XNA Creators Club site.

Atmospheres

There are many discussions on the subject of atmospheric scattering, many of which reference the work by Nishita et al. The article "Accurate Atmospheric Scattering", GPU Gems 2, Chapter 16 by Sean O'Neil served as a good starting point and is available here.

Clouds

The Visible Earth site also has a collection of cloud maps. This texture is then rendered on another sphere model above the surface of the planet.

Shadows

It was an important effect to cast shadows from the clouds onto the surface, particularly toward the terminator where the shadows are longer and not directly below the clouds themselves. My first approach was to implement a sphere-ray intersection algorithm in a pixel shader to dertermine the surface position of a shadow cast from my cloud sphere, and subtract the result from the existing surface texture.