Overview: 2 Dimensional Gesture Based Extension to the Standard User Interface
==============================================================================
Author: Mike Bennett (smoog dot stressbunny at com)
Last Update: 14/08/2003


Contents
--------
- Abstract
- Objectives
- Background
- Basic Infrastructure
- Resources Needed
- Future Possibilities


Abstract
--------
A user is able to setup wayV to recognize gestures and associate actions
with these gestures. When one of the gestures is inputed and matched an
action associated with the gesture occurs, e.g. startup a program, send
keypress', etc.

The gestures are created via a 2 dimensional input device, in most cases
the standard mouse.

The gestures should be unique symbolic representations of the actions 
they represent, i.e. draw an N and Netscape starts, draw a circle and
Opera starts, draw an > and the keypress' to switch desktops occurs,
etc.


Objectives
----------
1. develop an extension to the standard user interface paradigm
2. explore concepts in human computer interaction, especially with regards
   to the way users interact with menus and desktop based shortcut icons 
   (which are too often hidden)
3. develop an easier and faster way of executing certain programs, i.e. 
   when a user interacts with a computer they usually repeatedly run the
   same programs - always having to navigate through the same various menus 
   and sub-menus


Background
----------
When using a standard desktop, we'll use Windows 98 as the example, a user
can have desktop icons which function as shortcuts to execute programs. 
These shortcut icons are often used to represent programs which are required 
regularly. A major failing of this concept is the simple fact that the icons
tend to be hidden when programs are running on the desktop, effectively 
rendering the shortcuts useless.

We are then left with the "Start" button - which while an adequate solution
fails for a few reasons:
- it has various menus and sub-menus which aren't easy to rapidly navigate 
  or find things in
- when using the mouse the user is forced to break their "flow" and go to
  a very specific part of the screen
- when using the keyboard the user is often forced to press the arrow keys
  way too many times and then confirm their choice


Basic Infrastructure
--------------------
Consist of 1 program and 2 configuration files.

The program (wayv) runs, usually in the background, watching for certain
X events. Once the appropriate events are received a transparent window
is overlaid (depends on your wayV configuration) on the desktop. The end 
user then makes a gesture with the mouse resulting in a pattern appearing
on the screen. The pattern is then copied into memory, broken into a grid
and the resulting grid is sent to the pattern matching system to check
does it have a match. If it matches then the suitable actions are carried
out. This program is written in C and links with various X libraries.

The first configuration file, wayv.conf, stores:
1. the X event(s) which activate the gesture capture
2. the type of input displays and user feedback
3. the relationship between patterns and actions
4. the patterns to recognize
5. the actions and how they're to be carried out
6. miscellaneous as of yet unknowns

The second configuration file, DEFAULT.wkey, stores:
1. first column is the keycodes
2. second column is an alias for the keycode
3. optional third column is an alias for the keycode when Shift is pressed


Resources Needed
----------------
Hardware
- Computer running Unix with a functional X server
- Functional mouse (pointer) that is recognized by the X server

Software
- Running X server
- X window manager (CDE is sufficient)
- X libraries with development headers
- gcc, make, etc (if compiling from code)


Future Possibilities
--------------------
An idea like this has a lot of potential in terms of further development.
Some of the interesting possibilities are:

- use a webcam instead of a mouse to do the gesture capture
- experiment with ways of gesture capture in 3D space
- capture gestures as more than merely paths through space, i.e. develop
  techniques to capture small symbolic movements
- develop possible fuzzy scaling/representations of gestures
- examine what types of gestures are "good", i.e. easy to remember and
  reproduce (actually this is one of the areas I'm REALLY interested in,
  i.e. people's different cognitive models and how to design user interfaces
  that work with them)
- build on the concept of "flow" and non-intrusive command interfaces
- examine implications of speed, direction and time in gesture creation
- invent and explore techniques for end user feedback that reduce the
  need to remember various gestures and the associated actions
- find out whether having commonly defined "dictionary" of gestures is better
  than letting/encouraging each person develop their own (obviously language
  and cultural factors need to be considered)
- examine the possibilities for use in non-graphical human computer
  interaction
- develop further ways of recognizing gestures (evolutionary computing could
  be really interesting for that)
- less research but still handy would be a Mac and Windows version. I've a
  fair few friends who'd loved to use it but aren't Unix heads