Speech processing technology will move to a new level of social awareness to make interaction more intuitive, speech retrieval more efficient, and enhance computer-mediated communication and speech-analysis services. To reach this goal, rich speaker traits and personal states carried by the tone of the voice and the spoken words must be reliably identified by machines. In Schuller’s iHEARu project, ground-breaking methodology including novel techniques for multi-task and semi-supervised learning will deliver for the first time intelligent holistic and evolving analysis in real-life condition of universal speaker characteristics which have been considered only in isolation so far. Today's sparseness of annotated realistic speech data will be overcome by large-scale speech and meta-data mining from public sources such as social media, crowd-sourcing for labelling and quality control, and shared semi-automatic annotation.