Digital twin improved via visual question answering for vision-language interactive mode in human–machine collaboration

Tian Wang (1) , Jiakun Li (2) , Zhaoning Kong , Xin Liu (3) , Hichem Snoussi (4) , Hongqiang Lv (5)

1 Shaanxi University of Science and Technology
2 BUAA - Beihang University
3 School of Economics and Management
4 LM2S - Laboratoire Modélisation et Sûreté des Systèmes
5 School of Automation and Electrical Engineering [Beijing] (University of Science and Technology Beijing)

The human–machine collaboration system is a key means of manufacturing. Its surveillance, prognostic, and health management are related to safety and manufacturing persistence. This paper begins with the mission requirements of intelligent manufacturing. The study is based on the visual question answering (VQA) technology with a digital twin to increase efficiency. The research contents are as follows: (1) A method of modeling human–machine collaboration based on digital twins is proposed. (2) A VQA is adopted in the digital twin. The video and neural language are considered. (3) VQA technology is introduced into the modeling of the human–machine collaboration system for consistent integration. With VQA technology, humans and machines can collaborate. Human–machine interaction and product counting are implemented in a case study to provide a comprehensive perception.

Domaines: Traitement du signal et de l'image [eess.SP]

Tian Wang, Jiakun Li, Zhaoning Kong, Xin Liu, Hichem Snoussi, et al.. Digital twin improved via visual question answering for vision-language interactive mode in human–machine collaboration. Journal of Manufacturing Systems, 2021, 58 (PART B), pp.261-269. ⟨10.1016/j.jmsy.2020.07.011⟩. ⟨hal-03320661⟩