The Social Signal Interpretation (SSI) framework offers tools to record, analyse and recognize human behavior in real-time, such as gestures, mimics, head nods, and emotional speech. It supports streaming from multiple sensors and includes mechanisms for their synchronization. In particularly SSI supports the machine learning pipeline in its full length and offers a graphical interface that assists a user to collect own training corpora and obtain personalized models. It also suits the fusion of multimodal information at different stages including early and late fusion.
SSI offers the following features:
(1) Patch-based pipeline design (C++-API or easy-to-use XML editor) and plug-in system to extend available tools with new function
(2) Parallel and synchronized processing from multiple sensor devices, e.g. microphone, asio audio interface, web-cam, dv-cam, wiimote, kinect and physiological sensors
(3) General filter and feature algorithms, such as image processing, signal filtering, frequency analysis and statistical measurements in real-time
(4) Trigger mechanism for event based signal processing and full support of pattern recognition pipeline and training of personalized models
(5) Support of external tools, such as OpenCV, ARToolKitPlus, SHORE, Torch, Speex, Watson, as well as our own tools AuBT, EmoVoice, WiiGLE and FUBI.
A description of the framework is provided in:
 J. Wagner, F. Lingenfelser, and E. Andre, The Social Signal Interpretation Framework (SSI) for Real Time Signal Processing and Recognitions,” in Proceedings of INTERSPEECH 2011, Florence, Italy, 2011.
- year: 2012
- month: Jun
- url: http://hcm-lab.de/ssi.html
- main_author: Johannes Wagner (lead), Ionut Damian, Felix Kistler, Florian Lingenfelser
- license: LGPL
- platform: Windows C++