Visual Scene Understanding for IntCDC Wood Building System

VISUAL SEMANTIC SCENE UNDERSTANDING FOR COLLABORATIVE INTCDC WOOD BUILDING SYSTEM

Image processing and computer vision enable the automatic digitalisation and advanced analysis of the construction prefabrication process, in particular improving the safety and efficiency of human-robot collaboration. However, the data-intensive nature of modern visual recognition methods, their vulnerability to changes in data distribution, and their primary focus on recognising large structures pose significant challenges in practical Co-Agency scenarios. Furthermore, data collection and annotation for all possible situations is challenging, and data quality is often compromised by occlusion or sensor failure.

This project aims to develop a comprehensive deep learning based visual recognition and post-analysis approach for complex, multi-party prefabrication environments in timber construction prefabrication. The objectives are to:

• Develop transfer learning based computer vision algorithms for fine-grained recognition of humans, materials and their interactions in prefabrication environments;
• Implement methods to accurately assess algorithmic limitations through reliable estimates of aleatory and epistemic uncertainty at different levels of granularity;
• Detect and signal unexpected events and anomalies, and verify the digital twin on-the-fly against the visually captured state;
• Perform in-depth post-analysis of Co-Agency assembly, focusing on human-robot-material interactions, movement trajectories, and undefined anomalous behaviours, thereby enhancing automatic exploration of worker activities in highly collaborative dynamic scenes.

The ultimate goal of our visual recognition system is to enhance the safety, operational efficiency and adaptability of human-robot Co-Agency in the IntCDC wood building system.

PRINCIPAL INVESTIGATOR

Jun.-Prof. Dr.-Ing. Alina Roitberg
Institute for Artificial Intelligence (KI), University of Stuttgart 

TEAM

Thinesh Thiyakesan Ponbagavathi (KI)