Recently, together with my friends, we’ve finished our engineering degree work  entitled “Controlling Windows OS using trained gestures with Microsoft Kinect.” at Gdansk University of Technology. I’ve prepared simple not-so-technical presentation of our solution on the YouTube

Core of our project was gesture recognition system. We’ve prepared Gesture Recognizer application which task was obviously recognizing gestures presented by user. Also we created Gesture Teacher app where you can record various kinds of gestures to be recognized by the first app. The gestures are represented by a set of 60 points in two-dimensional space (X and Y). To improve detection ratio gestures are recorded multiple times. Recorded gesture is then influenced by a process of normalization and rotation (little bit of trigonometry, blah :P). During the development we’ve faced a problem that a “swipe” gesture (simple move of hand from one side to another) after normalization becomes unrecognizable. We’ve resolved this issue by separating swipe gestures from remaining ones and writing different detection algorithm for them.

Gesture recognition algorithm is quite complex so I won’t describe it precisely (if you are interested in it contact me). It’s based on Golden Section Search to find local extremum of a function. Swipe gesture detection is quite primitive and straightforward, it checks if hand move was in correct direction, speed and threshold of error.

To organize gestures we’ve prepared application that matches gestures with existing Windows OS controlling functions. All the configurations are kept in XML files so they are easy modifiable. You can record your own set of gestures and share it with your friend.

During work we’ve decided to extend our subject of project to controlling Windows applications: Windows Media Player and Microsoft PowerPoint. To control them we’ve used WinAPI interface. All the functions are available on WCF server. This solution allows us not only to control computer where the Kinect is connected but also any computer connected to LAN or even over the Internet 🙂

In addition we’ve prepared simple Android application that consumes functions of WCF server. For example you can change songs or volume while sitting comfortable on the couch 😉

I’m not planning to publish source code openly but if you are interested in our solution contact me.