Multimodal approaches combining voice, image, and gesture recognition must
necessarily acquire data from a variety of devices. The dedicated Multimodal
Middleware Protocol (MMP) provides the low level architecture to glue different
device modality components in a single user interface network. MMP’s goal is to
compose this network, abstracting details like underlying network protocols and
the meaning of custom messages, so that all higher layer semantics and logic can
relate to the composite multimodal interface. In the DIYSE concept the level
above the MMP is a powerful context reasoning system, providing context-aware
computing features, gathering information about users and their environment to
adjust the behaviour of applications. Through the natural interfaces provided by
multimodal devices such as the smart companion, context is seamlessly extended
to social expressivity.