Blue Sky Project Grant Winner 2022

Gili Ron – ICD/CA, University of Stuttgart; IMPRS-IS, MPI

The project investigates possibilities of agile responses to uncertainties in construction, developing novel Human-Robot Collaboration (HRC) methods, evaluated with a demonstrator. A team of a skilled human and an Artificial Intelligence (AI) trained UR10 Collaborative Robot (CoBot) assemble a wooden structure of an architectural scale, while responding to real-life scenarios: error, rework and skill shortage. To overcome the challenges of speedy and accurate assembly, deep reinforcement learning with novel reward functions (DL, RL) combined with physical feedback from the human agent are proposed. This small-scale demonstrator for HRC aims for Co-Design, reduced fabrication errors and material waste and foster human-robot kinship.

Supervisor: Tenure-Track Prof. Dr. Thomas Wortmann

News on the Blue Sky project grant 2022 announcement.

First, please tell us briefly about your research area and/or project

My name is Gili Ron. I am a researcher at the Institute for Computational Design and Construction, within the Department for Computing in Architecture (ICD/CA). In Associated Project 30 "Towards Human-Robot Co-Agency" (AP 30), I investigate agile responses to construction uncertainties by developing Human-Robot Collaboration (HRC) methods for inclusive practice, addressing the needs of both the industry and workers. This work tackles challenges in the construction industry that exacerbate low productivity and sustainability: the shortage of skilled labor and low automation implementation. The project aims to develop novel HRC methods with industrial robots (cobots; “collaborative robots”) that use AI, sensing, and user feedback. AI acts as a mediator between workers and cobots, improving collaboration with diverse demographics and ensuring worker agency and satisfaction. Specifically, I focus on creating a theoretical framework to guide diverse and inclusive HRC, increasing cobot agency using depth and RGB images, and deep reinforcement learning (RL) algorithms in collaborative assembly. Collaborative timber assembly ("pick-and-place") is then tested with diverse demographics and reviewed for its success, incorporating user feedback on comfort, trust, and safety.

The first steps in the research included studying current HRC research in construction and reviewing it from feminist technoscience perspectives (FTS) for diversity, inclusivity, and design biases. This led to developing a theoretical framework to review existing HRC, assisting in designing new HRC, sensitizing researchers to the realities of workers and their needs, as well as to their own biases. This was followed by design research in collaborative timber assembly, comparing the performance of a non-responsive cobot to a responsive one. This included purchasing and installing a LARA8 cobot, camera, F/T sensor, and gripper. The equipment is used for user studies in collaborative HRC, the first of which had 25 participants, including timber manufacturers Müller-Blaustein. The research was conducted in collaboration with Prof. Dr. Cordula Kropp and Ph.D. candidate Amelie Schreck from the Social Sciences department at the University of Stuttgart (SOWI).

Future steps include a user study with a cobot informed by vision (depth and RGB images) and trained in RL to acknowledge human choice and perform close collaboration safely. While the scope of this research is in construction, its proposed cyclical design process that includes user feedback, as well as a user-responsive robotic path, has merit in various fields where HRC is employed.

What were your objectives in applying for the award / why did you apply for the IntCDC Blue Sky Project Grant?

At the outset of my research, I applied for this award to obtain critical equipment for developing innovative communication and control methods for cobots in Human-Robot Collaboration (HRC). This acquisition supports my research endeavors for two years now and those of my thesis students in the ITECH master program from 2023 to 2025.

What did you use the award money for? 

The funding was used to purchase a research desktop and a tracking system; for simulated studies, and a visual input for robot control, respectively. Since a required research cobot exceeded the initial budget, I sought additional funding through the IRIS-3D program. I also partnered with the Research Project 4-1 "Cyber-Physical Wood Fabrication Platform"  (RP 4-1) team, to acquire a LARA8 cobot and end-effectors (gripper, force torque sensor, camera). This equipment is crucial for my research in AI-assisted HRC, enabling me to train cobots in simulation, implement policies on real robots, and test their effects in HRC processes.

What were you able to achieve with it for your research?

The purchased equipment runs simulated studies with reinforcement learning algorithms (RL), informs the robot live with visual inputs, and tests this training in collaborative pick-and-place sequences with varied users. A future step is forming a data set calculating all possible path plans in response to users’ choices in pick-and-place that will be deployed on the cobot to inform its trajectory. Combining the simulated studies with live visual input will be used in subsequent iterations of the user study, later to be combined with user input and preferences in close collaboration.

Tests in HRC have been insightful so far: the first user study in November 2023 had 25 diverse participants, including timber manufacturer Müller-Blaustein; a workshop for master students at the Institute for Advanced Architecture of Catalonia (IaaC) explored different modes of interactive robotic control; and a second user study is planned for November 2024. As my research continues, the cobots’ responsiveness grows with every development, and user feedback informs design decisions.

Since the grant is explicitly a prize for a risky research idea, a negative result would also be worth reporting. What important finding did it represent in your field of research?

The grant allowed me to explore an interdisciplinary topic, encompassing aspects of digital and robotic fabrication and FTS critique: the interrelations between humans and machines and their effect on making architecture. These themes continue to inform my work, and I’m grateful for having had the opportunity and freedom to explore them.

A challenge realized over time was handling inventory: purchasing multiple pieces of equipment meant spending much time, in addition to research work and related tutorials, managing the budget, inventory, quality tests, and maintaining responsive communication with suppliers and manufacturers. Consequently, the research’s planned timeline shifted. A lesson learned is to have collaborators or HiWis employed when new equipment arrives to integrate it seamlessly into the research pipeline.

What was your personal highlight among the things you were able to achieve with the award money?

The research’s interdisciplinary topic led to multiple collaborations, bridging and enriching research themes:

• On the subject of feminist technoscience (FTS), a collaboration with Prof. Dr. Cordula Kropp (Institute for Social Sciences, University of Stuttgart) led to my participation in the AHRA-1 workshop at Manchester University (April 2023),; a book chapter publication on the relevance of FTS critique to digital fabrication (Ron G., Kropp C., Menges A. & Wortmann T., 2024); and a successful funding application to the IRIS-3D fund from the Stuttgart Research Focus “Interchange Forum for Reflection on Intelligent Systems (IRIS).” The application led to a 2-year collaboration with Ph.D researcher Amelie Schreck, developing human-focused user studies in HRC.
• Regarding reinforcement learning (RL), a current collaboration with Prof. Dr. Georg Martius and Ph.D. candidate Boya Zhang (Autonomous Learning Department, University of Tübingen) on vision-informed RL for path planning is being further into a research proposal and joint funding application.
• Research on sensor-responsive robotic behavior was further explored in a workshop dedicated to HRC, FTS, and ABM for the master students at the Institute for Advanced Architecture of Catalonia (IaaC), in collaboration with Ph.D. candidates Samuel Leder and Lasath Siriwardena (Institute for Computational Design and Construction, University of Stuttgart).

Why would you advise others to apply for this prize?

The application helped me formalize my research ambitions, work packages, and timeline at an early research stage. The preparation expedited my familiarity with the state-of-the-art in HRC research in prefabrication in construction, advancing a novel contribution. The experience gained from the application assisted me in a subsequent application (IRIS-3D; funded by the Ministry of Science, Research, and Arts Baden-Wuerttemberg Az. 33-7533-9-19/54/5), to purchase equipment and support a position for my research collaborator.