Natural User Interface Group

Web Page: http://nuigc.com/gsoc2011

Mailing List: http://groups.google.com/group/nuigroup-gsoc

Established in 2006, The Natural User Interface Group is an open source community that creates and shares interaction techniques & standards that benefit designers & developers throughout the world. We offer a collaborative environment for scientists that are interested in learning and developing modern Human/Computer Interaction methods and concepts. Our research includes topics such as: computer vision, touch computing, voice & gesture recognition, and experience design and information visualization. We have participated in every GSoC event since 2008.

Natural User Interfaces are becoming common in our everyday lives. Companies are rapidly introducing new product lines with NUI methodologies interwoven within; Google, Samsung, Apple, Microsoft and others… all with a major push towards enhanced interaction. The class most influenced by the emergence so far is the “smart mobile” as each unit contains 5-10 sensors paired with rich effectors. With 2011 seeming to be the year of the tablet both touch computing and our understanding of UX must scale once again as Android has shown with Fragments. Moving beyond mobile applications, entertainment devices like the Xbox Kinect Sensor give us new sensing abilities which officially registers on Windows 7 as NUI Audio, NUI Camera and NUI Motor devices, hinting towards the future.

By participating in GSoC, we would like to expand the NUI concepts to a wider audience, raising interest from students around the world that do not typically have the opportunity to work with such technologies. We aim to study emerging sensing solutions and translate to applicable scenarios, starting with computer vision, natural language, and signal processing. Beyond that students can ponder on innovative problems that include related developments in clustering algorithms, image processing and machine learning.

With over 12,000 members, our goals is to accelerate the exploration of sensing solutions and interaction scenarios, thus allowing us to find the most effective methods of constructing modern interface devices. Our members are a worldwide gathering of students, researchers, designers, developers and engineers all who share a similar passion for the creation and distribution of open systems.

An important aspect is to create and utilize open standards that allow development to flourish. An example is our Community Core Vision or CCV project which is built in OpenFrameworks and supports several communications modes (Binary, TUIO, OSC and XML). The project is nearing one million total downloads and has introduced computer vision to a whole new audience, lowering the entry bar for new researchers while remaining powerful for advanced use and content creators. We would like to thank Google Summer of Code Project and the amazing contributors for making this a possible.

Everyone in the community is really excited about GSoC 2011 as last year set the stage for our Core Platform, Which aims to manage voice and visual signals and route to clients build in AS3/Java/Python. This year we would like to continue our development with focus on several key projects:

Community Core Services (C/C++):

• Community Core Audio (CCA) (GSoC 2010 - Alpha) - A cross platform server which manages voice inputs, convert voice to text, and output resulting messages to network. Alpha shipped during GSoC 2010 including a new addon (ofxASR) abstracting signal processors and sources. 2011 Goals - Ship 1.0 release stable on all platforms and broaden natural language processor support (SAPI, Sphinx, OSX-SRS) / CCW Management
• Community Core Vision (CCV) (GSoC 2008/2009/2010 - Beta 1.4) - A cross platform computer vision server for high fidelity tracking of objects such as fiducials, hands or fingers. First release during GSoC 2008 and with great community response. In 2009/2010 we continued to develop new tracking techniques and enhanced stability. 2011 Goals - Ship 1.5 with Depth Sensor (Kinect Devices) / Augmented Reality / CCW Management
• Community Core Web (CCW) (GSoC 2010 - Alpha) - A cross platform light weight web server which supports plugins for web development standards such as: Moongoose/PHP/MySQL/SQLite etc. Manages C++/Javascript communications allowing CCA/CCV to be controlled via minimal web client. Alpha started during GSoC 2010 but needs further development until official community beta. 2011 Goals - Ship 1.0, optimize, stable on all platforms, continue development of HTML/JS based management utility and binding to core services (CCA/CCV)

Community Client Frameworks:

• Actionscript/Flash - A client UX framework and set of example applications written in Actionscript (2007+)
• C/C++ - A client UX framework and set of example applications written in C/C++ and OpenGL ES 2
• Java/MT4J - A client UX framework and set of example applications written in Java (GSoC 2009/2010 projects)
• Python/PyMT - A client UX framework and set of example applications written in Python (GSoC 2008/2010 projects) 2011

Framework Goals - Development and standardization of target frameworks including structure, documentation and release packages.

Community Tools:

• Signal Simulators - Utilities to generate signals when hardware is not available. (QMTSim GSoC 2008) 
• Signal Translators - Client/Server Translation Utilities (flosc, wm_touch2tuio, mtvista)

With ongoing developments for supporting projects:

• OpenFrameworks - Continue contributing to Addons/Framework through C/C++ developments
• Protocols – Socket support on clients and servers to broaden TUIO/OSC support UX Design - A general focus on emerging experience design topics related to NUI 
• Web Applications - Community Search, Publishing and Communication (LAMP Stack)

Other than the projects mentioned above, we have our Ideas Page with topics to help give students inspiration and insight when writing their proposals.

Please refer to our ideas page for a full overview of past projects and video showcase: http://nuigc.com/gsocvideos

Projects

• A high level UX Framework - implemented in MT4J , AS3 and C++ The purpose of this framework is to initiate the creation of frameworks in different languages, where the frameworks in different languages have some level of similarity in their API and Designs.
• Fusion - a multimodal interaction framework for task-oriented and object-oriented interfaces A multimodal interface provides the user a simultaneous combination of different interaction modes (i.e., keyboard, speech, touch, pen) to increase usability and interaction bandwidth. The product of this project is a framework for a modular but initially touch-and-speech-only multimodal interface, an engine to generate a grammar from a model of the application's objects and functions, and a demonstrative application that shows the concept in action with CCA/Sphinx and a traditional mouse.
• GPU implementation of CCV I am submitting my proposal on GPU implementation in CCV. In the latest release of CCV, GPU support was removed, hence I found it as a good opportunity to contribute to the community this summer. I will be using Nvidia CUDA to achieve this goal. The deliverables I will be providing as initial implementation this summer will be GPU filtering and GPU blob finding. I will be leaving the task of blob tracking to CPU because porting of knn algorithm on GPU might adversely affect CCV performance.
• Kivy: Tight Mac OS X Integration Kivy abstracts from platform specific APIs to allow it to run on many different platforms, such as Mac OS X, Windows, Linux and Android. For every platform, specific 'providers' can be implemented that provide a certain piece of functionality (video, audio, text rendering, etc). On OS X we currently ship things like gstreamer in our portable package, which is not really easy to handle. This proposal aims at improving the situation by adding support for native OS X APIs such as Core Video/Audio.
• Multiple Camera Support for CCV In this proposal, the multi-cam support module for CCV will be introduced. By integrating the methods related to Camera Calibration and 3D Reconstruction of OpenCV into CCV, we can make the CCV track the blobs with the relationship between the multi-cams and the screen, and then we can track the objects and fiducials by stitching the undistorted and rectified captured by multi-cams.
• nuiFrameworks This proposal has two goals: 1. Migrate CCV from openframeworks; 2. Make a new wxWidgets based frameworks with module structure. CCV has a huge number of users, however, it is weak in GUI. CCV is based on openframeworks, which just simulates a GUI, instead of use operation system’s native GUI API. Once CCV was migrated, a new frameworks based on wxWidgets will be extracted and be the general frameworks for all our Community Core Services.