NUIverse for Windows

By
Dave
Project
Published
2 Apr 2013 15:05
Last Modified
2 Apr 2013 22:31

As previously discussed, my manipulation processor now supports WM_TOUCH messages, which means that I can do native multitouch on both Windows 7 and Windows 8. I have therefore updated NUIverse for a Windows-release, as shown below in Figure 1.

NUIverse for Windows

Figure 1. NUIverse for Windows (running windowed).

There are some key differences to the PixelSense-release, as follows:

  • No support for tagged objects, since it does not use the Surface 2.0 runtime, nor require PixelSense hardware (though it will run on the latter outside of the Surface Shell).
  • Since horizontal form-factor multitouch hardware is generally less-common than vertical form-factors, I have added a single-orientation configuration setting. This is true by default, since even if mounted horizontally, many touchscreens will not deliver the multitouch performance required for simultaneous multi-user interaction.
  • Since the Surface Shell added chrome to close the application and by default the application runs full-screen, either drag in a menu control (see note below) and use the exit menu, or press ESC if a keyboard is present.

One of the current issues when running a full-screen desktop app on Windows 8 is that the operating system captures initial touches used for edge-swipes. If touch is maintained after an initial edge-swipe, further edge-swipes are not captured and therefore will add NUIverse controls to the screen. An alternative is to touch the screen and simultaneously edge-swipe, or to use two fingers when edge-swiping.

Several key configuration settings (in NUIverse.exe.config) are worth mentioning. Note that there is no graphical interface for these settings, and that the configuration file needs to be edited by hand (I would recommend saving a copy first):

  • PixelWidth and PixelHeight control the resolution used for both windowed and full-screen mode.
  • FullScreen controls whether the application runs full-screen (true) or windowed (false).
  • For the configuration settings specified in mm to work correctly, set PixelsPerMm to the appropriate value, taking account physical screen size and either PixelWidth or PixelHeight (square pixels are assumed).

To install NUIverse for Windows, proceed as folows:

  1. Install the XNA 4.0 runtime.
  2. Download and extract NUIverse for Windows (2.75Mb) to a suitable location.
  3. Low-resolution textures for several planets and moons are included, but extras can be created or downloaded from http://nuiverse.com.

Windows Multitouch and XNA

By
Dave
Project
Published
9 Feb 2013 23:56
Last Modified
10 Apr 2013 00:08

I wanted to support multitouch in an XNA application on Windows, without using the Microsoft Surface 2.0 SDK and runtime. Unlike Windows Phone however, touch input on Windows is not natively supported by the XNA framework. I have therefore followed the recommended approach and added a new input source to my manipulation processor for WM_TOUCH messages. It therefore works on both Windows 7 and Windows 8.

I hook the Windows message loop to a managed function pointer using GetFunctionPointerForDelegate and Get/SetWindowLongPtr. I then register the window for multitouch using RegisterTouchWindow, and process WM_TOUCHDOWN, WM_TOUCHMOVE and WM_TOUCHUP messages.

While no longer requiring the Microsoft Surface 2.0 runtime, one option to inject WM_TOUCH messages on non-touch hardware is to use the Microsoft Surface 2.0 SDK Input Simulator, as shown below in Figure 1. Note that this method for simulating input will only work on Windows 7, however.

WM_TOUCH manipulation

Figure 1. WM_TOUCH manipulation, using the Surface 2.0 SDK Input Simulator.

Curiously, WM_TOUCH messages are only provided when not using a mouse. If the mouse is used, WM_TOUCHUP messages are fired for all current touches, so mixed-mode interactions when the mouse is used simultaneously are not possible.

Extensibility Model Part 2

By
Dave
Project
Published
20 Jan 2013 10:20
Last Modified
20 Jan 2013 21:31

I previously mentioned that I had implemented an extensibility model, and thought it useful to discuss an example of adding a simple model to earth orbit, as shown below in Figure 1 (further images of which are in the gallery).

Model

Figure 1. Model added to earth orbit. Colonial Raptor model (based on new TV series) by Coxxon.

The "extra" is defined as a folder containing the following items:

  • A model in XNB format. XNA has built-in content importers for .x and .fbx (2009.1) formats.
  • An optional pair of textures for both diffuse and emissive textures. These are standard image files.
  • An XML file defining the "extra", in this case as shown below in Listing 1.
<?xml version="1.0" encoding="utf-8"?>
<system name="solar">
  <planet name="earth">
     <!-- satellite
       name = object name
       box = box for pick tests, normalized relative xyz size (currently not used)
       size = maximum length, km
       description = label text
       specintensity = specular intensity, default 0.50 in app.config
       specpower = specular power, default 10 in app.config
       scale = scale factor to unit length, default 1
       model = path to model .xnb file
       texture = path to texture file
       emissive = path to emmissive texture file
       rotation = rotation = sidereal rotation period, days
    -->
    <satellite name="raptor" box="1,1,1" size="0.0086" description="raptor" specintensity="0.1" specpower="10" scale="1" model="colrap1cox.xnb" texture="texture/colrap1cox.jpg" emissive="emissive/colrap1cox.jpg" rotation="1000" >
      <!-- orbit
         a = semi-major axis, km
         e = eccentricity
         w = argument of perifocus, degrees (aka longitude of perihelion, argument of perigee)
         i = inclination to xy plane, degrees
         node = longitude of ascending node, degrees
         M = mean anomaly, degrees (J2000.0)
         P = period, days
         plane = orbital plane (Ecliptic, Equatorial, Laplace), default Ecliptic 
      -->
      <orbit a="6871" e="0" i="0" node="0" w="0" M="0" P="1000" plane="Equatorial" />
    </satellite>
  </planet>
</system>

Listing 1. XML configuration file for satellite model "extra"

This configuration file specifies that the model should be added to the planetoid "earth" in the "solar" system, both of which are defined in system.xml configuration file.

In order to scale the model correctly, a scale factor is applied to normalize the model to unit length. This can either be applied in the XML scale attribute, or specified in the XNA content processor scale attribute, in which case the XML attribute can be set to 1. A size attribute then defines the maximum length of the model in km. The Colonial Raptor shown in Figure 1 was defined with a size of 8.6m.

The textures are defined in sub-folders "texture" and "emissive". If an emissive texture is not available, an all-black image (e.g. JPEG file) can be used.

The rotation period defines how long it takes for the model to rotate while orbiting the planetoid. If this is the same as the P orbital element, then the same face of the model is presented to the planetoid throughout the orbit. The remaining standard orbital elements specify that the model is in a circular equatorial orbit at an altitude of 500km (the earth has a radius of 6,371km).

The extra is included automatically when added to the /data/extras folder.

NUIverse Video Part 2

By
Dave
Project
Published
19 Jul 2012 09:59
Last Modified
13 Jan 2013 17:17

I had the opportunity to demo NUIverse at the Microsoft Worldwide Partner Conference last week, and I thought I'd share the video which shows some updates since the previous recording.

Video 1. NUIverse on Samsung SUR40 with Microsoft PixelSense.

Key things demonstrated in the video include:

  • Multi-touch to control complex camera motion
  • Multi-direction UI consistent with a horizontal display form-factor and multiple concurrent users
  • Level-of-Detail rendering for planetary bodies and backgrounds
  • Independant control of time and position
  • Control selection using just-in-time-chrome
  • Satellite model rendering

Planet Selection

By
Dave
Project
Published
7 Jul 2012 18:10
Last Modified
10 Jul 2012 03:25

While it is possible to select a planetary body using a touch-and-hold gesture on its label or the body itself, finding the planet can be tricky. At the very least the orbital lines and labels need to be visible. A bigger problem, however is that the "inner" and "outer" planet orbits have markedly different scales. Hence when in orbit around an outer planet it will not generally be possible to resolve an inner planet for touch-and-hold.

I therefore added a further control for selecting a planet, as shown below in Figure 1. Since moons are relatively close to a planet in comparion to other planets, touch-and-hold can be used to move to a moon in orbit around the currently selected planet. The control can be added from the tag selection bar, as shown below in Figure 1. Touching a planet switches the camera to orbit mode around the body.

Planet Selector

Figure 1. Planet Selector Control. Earth is the current planetary system.

The planet names and images are built at run time to ensure that they reflect any extensions to the base data.

Scale

By
Dave
Project
Published
28 May 2012 11:22
Last Modified
13 Jan 2013 17:47

One of the difficulties when judging distances in the solar system, other than that they are very large, is that the solar system is mostly empty space. Orbit lines only give a hint of motion and indicative scale.

I've added some planar grids to the ecliptic to try to help visualise distances at logarithmic scales starting at 1AU, with each grid fading out as the next fades in. Grids can be circular, square, or hexagonal. Circular grids are useful to visualise orbit eccentricity, perihelion and aphelion. Square grids are useful to disambiguate from orbit lines.

Scale grid

Figure 1. Circular scale grid.1

Scale grid

Figure 2. Square scale grid.1

Scale grid

Figure 1. Hexagonal scale grid.1

I've yet to find a compelling use for hexagonal grids, however they appear to be useful for Klingon tactical displays.

Additional screenshots are in the gallery.

1 Background filter generated from image credit Nick Risinger, skysurvey.org

Constellations Part 2

By
Dave
Project
Published
20 May 2012 23:45
Last Modified
22 May 2012 19:37

In part 1 I discussed adding support for drawing constellations, boundaries and asterisms, along with their names.

I've now added support for constellation art. Rather than loading individual images for each constellation shown in the camera view, I opted instead to use the same rendering approach as for background filters, which gives on-demand loading at the appropriate level-of-detail. A screenshot is shown below in Figure 1.

Constellation Art

Figure 1. Constellation art. The field-of-view is 70°.

The Space Telescope Science Institiute worked with the U.S. Naval Observatory's Library to scan and painstakingly re-touch a set of beautiful engravings from the 17th century Firmamentum Sobiescianum sive Uranographia star atlas by Johannes Hevelius1. In order to use these images, I first needed to align them with the background stars, then re-project them to a suitable format. I opted for an equirectangular projection. Whilst there is almost certainly a choice of tools available for the job, I thought it would be fun to write my own. The result of applying it to Figure 2 is shown in Figure 3 below.


Ursa Major

Figure 2. Engraving of Ursa Major, "Firmamentum Sobiescianum sive Uranographia", Johannes Hevelius, 16901, positioned appropriately over an spherical, equatorial grid of right-ascension and declination.


Ursa Major

Figure 3. Equirectangular, planar re-projection of Figure 2, resulting in vertical and horizontal lines of right-ascension and declination respectively.

The app will ship with low-res versions of the art, with higher-resolution versions available as an extra. It is, of course possible to use alternative art using the same extensibility model.

1 Image credit the U.S. Naval Observatory and the Space Telescope Science Institute.

Filters Part 2

By
Dave
Project
Published
12 May 2012 18:29
Last Modified
13 Jan 2013 17:21

I previously discussed a method for rendering background images over specific areas of the screen using filters. In order to combine these images with the entire starfield, I followed a similar level-of-detail approach to planetary bodies. Figures 1 and 2 below show example screenshots.

H-ALpha

Figure 1. H-alpha emmision1 filter, 225,000km from Jupiter. The field-of-view is 90°.

WISE

Figure 2. Wide-field Infrared Survey2 (WISE) filter, 20,000km from Earth. The field-of-view is 70°.

Now that I am using an extensibility model, once I have generated the image tiles and added the configuration file, I simply need copy the data to the application data directory for use.

Check out the gallery for more images.

1Composite of the Virginia Tech Spectral line Survey (VTSS) and the Southern H-Alpha Sky Survey Atlas (SHASSA), Credit Douglas Finkbeiner
2Credit NASA/JPL-Caltech/WISE Team

Planetary Body Shader Part 3

By
Dave
Project
Published
22 Apr 2012 11:56
Last Modified
8 Dec 2012 16:23

It's been a while since I've updated my planet shader. One of the items on my list was to make the shader more modular so that I could cope with different "types" of planet or moon, without proliferating the number of techniques. This is particularly important now that I am using an extensibility model for adding or updating data.

The planetary shader currently deals with the following options:

  • Color map
  • Normal map
  • Specular map
  • Night map
  • Cloud map
  • Atmosphere
  • Rings (with shadows on both the body and ring)

One approach to support multiple combinations of these options is to use seperate shader techniques. The number of combinations when picking r items from n options is defined by Equation 1.

Combinations(1)

Since I could use any number of these options, the number is given by Equation 2.

Shader Combinations(2)

For 7 shader options, this therefore gives 127 shader techniques. Clearly this would not be viable, so I opted to use static shader branching. Since certain aspects of my rendering required Shader Model 3.0 anyway, this seemed a good approach, and resulted in acceptable performance.

Figures 1-7 below show individual shader options for color, specular mapping, normal mapping, night, atmosphere, and clouds, with a cumulative combination alongside.

Color

Figure 1. Color Map.

Specular Color, Specular

Figure 2. Specular Map.

Normals Color, Specular, Normals

Figure 3. Normal Map.

Night Color, Specular, Normals, Night

Figure 4. Night Map.

Atmosphere Color, Specular, Normals, Night, Atmosphere

Figure 5. Atmosphere.

Clouds Color, Specular, Normals, Night, Atmosphere, Clouds

Figure 6. Clouds.

Figures 7 and 8 below show individual shader options for color and rings with a cumulative combination alongside.

Color

Figure 7. Color.

Ring Color, Ring

Figure 8. Rings.

Camera Control

By
Dave
Project
Published
31 Jan 2012 17:58
Last Modified
13 Jan 2013 17:26

Currently the app supports a number of different types of manipulation, depending on the camera mode. These are demonstrated in the following short video and summarised below.

Video 1. Touch Manipulation used for camera control, recorded on a Microsoft Surface 2.0.

Free Camera

  • Pitch and Yaw by moving one or more touchpoints on the background
  • Roll by rotating two or more touchpoints on the background

Orbit Camera

  • Orbit body by moving one or more touchpoints on the background
  • Roll by rotating two or more touchpoints on the background
  • Adjust distance to body by pinch-zooming two or more touchpoints on the background

Geosync Camera

  • Roll by rotating two or more touchpoints on the background
  • Adjust distance to body by pinch-zooming two or more touchpoints on the background

Tracking Camera

  • Roll by rotating two or more touchpoints on the background

As shown in the video, when time is running, an orbit camera travels with the body, maintaining constant orientation so that the background stars remain fixed. The geosync camera always points at a specific coordinate on the body, and maintains a constant north bearing.

Smoothing

Touch resolutions oftern correspond to approximately the screen resolution in pixels, hence smoothing is necessary to avoid "jumps" in orientation or position. Also of importance is the use of momentum and inertia to provide a more "natural" touch experience.

I initially used a simple spring algorithm to add smoothing, momentum and inertia, and tracked manipulation speed to continue inertia when manipulation ended. This worked well at high framerates, but when using vertical sync (i.e. 60fps) the experience degraded.

Switching to a simple linear interpolation (LERP) of position and spherical linear interpolation (SLERP) for orientation bahaves well at lower frame-rates, and also gives the impression of momentum and inertia. I no longer track manipulation speed, nor continue inertia when the interpolation is complete.

Page