Tag Support

By
Dave
Project
Published
10 Jun 2012 22:24
Last Modified
13 Jan 2013 17:19

I currently use Surface Tags for menus, reticles, compasses, and filters.

There are both advantages and disadvantages to this approach. Advantages inlcude the ability of a tag to provide context of which control to add to the screen, and the ease at which a physical object can be moved, rotated, and removed (i.e. no requirement for "close" buttons). Disadvantages include the requirement to have the tags, and what to do in either Windows 7 multi-touch or vertical Surface deployments. With the latter in-mind, I was keen to explore a "tag-free" mode, which uses touch instead of tags.

In order to add a control to the screen without a tag, I needed a gesture which was easy to discover, and which had a low chance of being invoked accidentally. I opted for an analogy to sliding a physical tag from the bezel onto the edge of the screen. To add a default control, I drag a touch from the edge of the screen. In order to add different controls, I slide in a default "virtual tag" until the tag "selection bar" appears, then move in an orthogonal direction to cycle through a circular list of controls, then continue sliding in to place the control. These steps are illustrated below.

Tag Free Hint

Figure 1. Hint to "tag-free" default control.

Tag Free Menu

Figure 2. Slide-in "tag-free" default (menu) control.

Tag Free Compass

Figure 3. Cross-slide to "tag-free" compass control.

Tag Free Reticle

Figure 4. Cross-slide to "tag-free" reticle control.

In a similar way to physical tags, I can rotate the "virtual tags" using two or more touch points placed where the physical tag would normally reside. To remove a tag, I slide or flick it to the edge of the screen.

In order to best support multiple users and/or orientations, it is possible to slide in controls from each edge of the Surface, with the tag "selection bar" and controls being oriented appropriately.

NUIverse Video

By
Dave
Project
Published
16 Feb 2012 16:02
Last Modified
13 Jan 2013 17:48

One of my colleagues came up with the name 'NUIverse', and since a major motivation behind creating this app was the opportunity to explore the use of Natural User Interface (NUI), and the name is a nice play on the spelling of 'Universe' the name has stuck.

While there are still a heap of tweaks and new features on my TODO list, I though it was a good time to share a short video captured from the app.

Video 1. NUIverse on Microsoft Surface 2.0.

Note that rather than using an overhead camera, I used FRAPS to capture video. Touch-points are highlighed as translucent circular markers.

Credits:

  • Data provided by NASA
  • Music by iStockphoto®, ©AudioQuattro, Sky
  • Battlestar Galactica models (based on new TV series) by Coxxon

Update: I thought I'd also include a link to the other video mentioned in the comments, which films the app being used rather than capturing the screen.

This video has now moved, so I've udated the link below.

Video 2. NUIverse on Microsoft Surface 2.0.

Camera Control

By
Dave
Project
Published
31 Jan 2012 17:58
Last Modified
13 Jan 2013 17:26

Currently the app supports a number of different types of manipulation, depending on the camera mode. These are demonstrated in the following short video and summarised below.

Video 1. Touch Manipulation used for camera control, recorded on a Microsoft Surface 2.0.

Free Camera

  • Pitch and Yaw by moving one or more touchpoints on the background
  • Roll by rotating two or more touchpoints on the background

Orbit Camera

  • Orbit body by moving one or more touchpoints on the background
  • Roll by rotating two or more touchpoints on the background
  • Adjust distance to body by pinch-zooming two or more touchpoints on the background

Geosync Camera

  • Roll by rotating two or more touchpoints on the background
  • Adjust distance to body by pinch-zooming two or more touchpoints on the background

Tracking Camera

  • Roll by rotating two or more touchpoints on the background

As shown in the video, when time is running, an orbit camera travels with the body, maintaining constant orientation so that the background stars remain fixed. The geosync camera always points at a specific coordinate on the body, and maintains a constant north bearing.

Smoothing

Touch resolutions oftern correspond to approximately the screen resolution in pixels, hence smoothing is necessary to avoid "jumps" in orientation or position. Also of importance is the use of momentum and inertia to provide a more "natural" touch experience.

I initially used a simple spring algorithm to add smoothing, momentum and inertia, and tracked manipulation speed to continue inertia when manipulation ended. This worked well at high framerates, but when using vertical sync (i.e. 60fps) the experience degraded.

Switching to a simple linear interpolation (LERP) of position and spherical linear interpolation (SLERP) for orientation bahaves well at lower frame-rates, and also gives the impression of momentum and inertia. I no longer track manipulation speed, nor continue inertia when the interpolation is complete.

Orientation

By
Dave
Project
Published
17 May 2011 20:48
Last Modified
13 Jan 2013 17:34

Since I wanted to support running the application on a Microsoft Surface device, it was important to cater for multi-directional interaction. User interface (UI) components can eiher be directional or not. For example, the camera view is non-directional in the sense that it is valid for all users regardless of direction (it is perfectly acceptable to be "upside down" when rolling through 180° in a 3D environment). Text labels, on the other hand, are directional and most beneficial when directly facing a user.

The following screenshots illustrate how the user interface supports multiple directions and multiple users.

Orientation

Figure 1. User interface shown orientated for single user.

Orientation

Figure 2. User interface components are shown with varying orientation to support multiple users.

Figure 2 shows a menu and reticle for each of two users on opposite sides. While there can be many instances of menus and reticles at any one time, a given instance is typically used by one user at a time. It is therefore possible to orient them to the relevant user, either by using the orientation of a Surface tag, or by using the orienation reported by the API for a touch event.

For items such as background labels which are shared between multiple users, it is necessary to pick a consistent orientation for all instances. This "system orientation" can either be determined in code (e.g. by examining the orientation of other directional UI components) or by a user via a menu setting. In Figure 2 the orienation has been chosen to face one of the two users.

While the system orientation is an analogue value (i.e. the background labels, for example, can face any consistent direction), it makes sense to axis-align the orientation of items such as the clock to a side of the screen.

Surface 3D Demo Part 3

By
Dave
Project
Published
18 Mar 2011 20:32
Last Modified
13 Jan 2013 18:24

In Part 1 and Part 2 I described an interactive autostereogram viewer. In this post, I thought I start by including a link to a video.

The video shows how multitouch gestures on Microsoft Surface can be used to interact with the models. A depth-map shader then processes the model, and a second-pass converts the depth-map to an autostereogram. As described previously, to avoid distracting the eye when animating a tile-based autostereogram, a randon-dot pattern is used which is regenerated each frame. Unfortunately the video doesn't have the required resolution to see much detail when animating the autostereogram, but the 3D-effect seen when the animation stops is maintained when viewing the application.

Here are some further screenshots:

SIRDS

Figure 1. Depth map

SIRDS

Figure 2. SIRDS for animation

SIRDS

Figure 3. Texture for static image

Multitouch Game Loop Input Processing

By
Dave
Project
Published
21 Jan 2011 19:37
Last Modified
13 Jan 2013 18:13

There are multiple potential sources of input for an XNA game component, such as keyboard, gamepad, mouse, Surface multitouch, .NET 4 Touch, etc. Dealing with multpile approaches to gathering input in the Update loop of each component can become complex, and adding support for a new input type further adds to this complexity.

I divided my input handling into two different categories:

  1. Controller-based input such as keyboard and gamepad.
  2. Manipulation-based input such as mouse, Surface multitouch, and .NET 4 touch.

For the latter, I wanted to abstract input handling such that a given game component only had to deal with a custom Manipulator type. In this way, I would be free to change or add input source(s) without affecting the implementation in a game component.

Clearly, different input sources provide different types and degrees of information. Surface multitouch Contacts for example, provide information on size, orientation, type etc, whereas a mouse only provides position. In many cases only position information is necessary, however additional properties can easily be added to the Manipulator type and supported by a game component if available. In this case I decided to sub-class my Manipulator type to the following:

  1. Mouse and finger-touch
  2. Surface Tag contacts
  3. Surface blob contacts

In order to deal with multitouch manipulations, I could process input using my previosuly-discussed Manipulation classes and processor.

The following video demonstrates the use of multiple input sources:

Video 1. Input processing from multiple sources.

Since a mouse can only deliver single-touch input, when demonstrating mouse input in Video 1 I switch the pivot type to "Tracked" in order to be able to demonstrate rotation and scaling, as shown in Figure 1.

Manipulation Input

Figure 1. Input processing from multiple sources.

Of course, it is unlikely that a scenario mixing both Mouse and Surface input would ever be used in practice, however it serves to illustrate how a game component can handle input in a consistent way, without being aware of the input source. For example, the buttons on the left of the screen are "pressed" using either Surface v1 Contact or a mouse-click, however the buttons only tracking Manipulator state.

A useful application of this approach is the ability to write XNA applications which work with both Surface v1 multitouch input and .NET 4 multitouch (and mouse for single-touch if multi-touch hardware is not available) without any code changes, i.e. multi-platform targetting.

Multitouch Game Loop Inertia Processing

By
Dave
Project
Published
21 Sep 2010 18:11
Last Modified
8 Dec 2012 16:21

In Part 1 I described a lightweight class to process general multitouch manipulations using a game-loop model in XNA.

A key aspect in bringing realism to a multitouch experience is the use of momentum and inertia. The Microsoft.Surface.Core.Manipulations.Affine2DInertiaProcessor provides this support in an event-driven model. In order to continue the approach of using a game-loop model, I needed to add linear and angular velocities, and expansion rates to my manipulation processor.

Changes in manipulator position occur relatively rarely in relation to the game loop frequency, hence calculating rates of change requires averaging these changes over time. The choice of duration and quantity of these time periods is somewhat subjective, and I settled on 5 samples averaged over a maximum of 200ms. Clearly this leads to a small latency between measured and actual velocities, however the smoothing effect is beneficial.

In order to visualise these rates, both for debbugging and to settle on suitable sampling settings, I added visual indicators of direction and magnitude, as shown below in Figure 1.

Multitouch manipulation

Figure 1. Visualising linear and angular velocities and expansion rates during multitouch manipulation.

With the addition of linear and angular velocities and expansion rates, I could now add a simple inertial processing component which uses these properties as input parameters. This inertial processor uses a simple deceleration algorithm, with configurable rates for translation, rotation, and expansion. The solution is demonstrated in the following video.

Video 1. Multitouch game loop inertia processing.

Note that for gestures such as a flick, where velcity increases rapidly, a time-weighted average may be more suitable for calculating the inertia processor's input parameters, and I'll investigate this at a later date.

Multitouch Game Loop Manipulation Processing

By
Dave
Project
Published
12 Aug 2010 23:05
Last Modified
8 Dec 2012 16:21

Multitouch Surface support for XNA-based applications is provided via the Microsoft.Surface.Core.Manipulations namespace in an event-driven model. I was interested in investigating general multitouch processing within a game-loop model, i.e. not involving events, and thought it would be instructuve to develop my own class to process manipulations in this way.

In a similar manner to the Affine2DManipulationProcessor, I wanted to support arbitrary combinations of translation, rotation, and scale. Translation is trivial, since it is simply a matter of tracking the changes in each manipulation point between calls and taking the average.

Rotation and scale is a little more tricky, since each of these values are relative to a centre-point, or manipulation origin. Rotation is calculated from the average angle between each point and the origin, and scale is calculated from the average distance. Unlike a mouse cursor, manipulation points come and go, so the key is only to track changes to those points which persist between calls, and then update the positions, origin, distance and angles for the next call.

In order to generalise manipulation points from multiple sources such as mouse, gamepad, Surface Contacts and .NET 4.0 Touch, I created my own Manipulation structure to abstract properties such as ID and position. My input processing can then build a collection of these objects from any relevant source(s) and pass them to my manipulation processor as part of the Update call as follows:

myManipulationProcessor.ProcessManipulators(myManipulators);

The cumulative transforms for translation, rotation and scale (according to the supported manipulation types specified) are then immediately available as properties on the manipulation processor.

In order to test the solution, I wrote a small harness to visualise the manipulation points, and their effect on a reference cross-hair. While I also added support for a mouse (right-click to add/remove points, left-drag to move points), multi-touch hardware is required to test scenarios when multiple points are added, moved, or removed simultaneusly. A screenshot is shown in Figure 1.

Multitouch manipulation

Figure 1. Multitouch manipulation with averaged manipulation origin.

One of the complexities of working with multiple manipulation points is deciding how to determine the manipulation origin. One option is to simply take the average position of each point, as shown in Figure 1. Another option is to introduce a speicific "pivot point" as the origin and use this to calculate scale and rotation. This pivot point can either be fixed, or tracking the object being manipulated. The latter two options are shown in Figures 2 and 3.

Multitouch manipulation

Figure 2. Multitouch manipulation with fixed pivot point.

Multitouch manipulation

Figure 3. Multitouch manipulation with pivot point centered on manipulation object.

The solution is demonstrated in the following video. Each approach for determining the manipulation origin is illustrated using translation, rotation, and scaling manipulations, initially in isolation and then in combination.

Video 1. Multitouch game loop manipulation processing.

Surface 3D Demo Part 2

By
Dave
Project
Published
7 Aug 2010 11:03
Last Modified
13 Jan 2013 18:23

In Part 1 I discussed an simple shader which generated autostereograms from depth maps. The next step was to make the application interactive, so that instead of generating an autostereogram from a static depth-map, an interactive 3D model could be used to generate dynamic autostereograms.

To create a depth-map from a model, I added a pixel shader to return the depth value. A similar example can be found in How To: Create a Depth Texture. In order to obtain a sensible range of depth values, I need to calculate the bounds of the model using bounding spheres. I can then set the near and far planes in my projection matrix to correspond to the closest and furthest points in the model respectively. The depth texture can be extracted by rendering to a render target. It can then be passed as a shader parameter to allow autostereograms to be rendered in real time as the model is manipulated.

One of the main issues with animating an autostereogram is that manipulating the model results in changes across areas of the image which do not correspond to the location of the model itself.1 One way around this distracting side-effect is to use a SIRDS, or Single Image Random Dot Stereogram, and alter the pattern each frame in which the model is manipulated.

Instead of passing a pre-defined pattern texture, I generate a random texture on the CPU as follows:

private Texture2D CreateStaticMap(int resolution)
{
    Random random = new Random();
    Color[] colors = new Color[resolution * resolution];
    for (int x = 0; x < resolution; x++)
        for (int y = 0; y < resolution; y++)
            colors[x + y * resolution] = new Color(new Vector3((float)random.NextDouble()));

    Texture2D texture = new Texture2D(GraphicsDevice, resolution, resolution, 1, TextureUsage.None, SurfaceFormat.Color);
    texture.SetData(colors);
    return texture;
}

The resultant grayscale image is shown below in Figure 1.

SIRDS

Figure 1. SIRDS for animation

1This is exacerbated by my current simplistic stereogram shader, since it doesn't yet implmenent features such as hidden-surface removal (where a part of the model should only be visible by one eye) which in turn leads to additional "echoes" in the image.

Surface 3D Demo Part 1

By
Dave
Project
Published
1 Aug 2010 19:19
Last Modified
13 Jan 2013 18:22

I've always been fascinated by stereoscopic images and their ability to convey depth from just two dimensions, and I was interested to explore their effectiveness in implementing a Surface application for 3D visualisation.

As a multiuser and multidiretional platform, Microsoft Surface is ideal for viewing 2D content. Since stereoscropic images can viewed1 from angles orthogonal to an axis designed to align with the plane of the viewer's eyes, they enable depth perception from two opposing viewing positions on either side of a Surface device.

The first step was to generate a stereoscopic image from a depth-map by implementing a pixel shader to render an autostereogram. My initial algorithm was very basic, and produced images as per the example shown below in Figure 1.

Initial rendering

Figure 1. Initial autostereogram rendering from depth map.

The initial shader takes the following parameters:

  • Depth texture
  • Tile texture
  • Size of tile (number of vertical strips)
  • Depth factor

1In order to perceive a 3D image the viewer must decouple convergence and focusing of their eyes. In order to aid convergence, a disc and ring have been placed at the correct separation. Looking "through" the image results in four shapes (a disc and ring from each eye). The eyes are correctly converged when the two centre shapes "overlap" to give a disc within a ring. At this point the eyes must be refocussed without changing their convergence. Bright light can help, since a contracted iris gives a decreased depth of field.

Page