Hi4D: 4D Instance Segmentation of Close Human Interaction

ETH Zürich
CVPR 2023

Hi4D (Humans interacting in 4D), a dataset of humans in close physical interaction that contains (A) 4D textured scans, (B) instance meshes with vertex-level contact annotations, (C) instance masks in 2D and 3D, (D) registered parametric body models with contact annotations.

Abstract

We propose Hi4D, a method and dataset for the automatic analysis of physically close human-human interaction under prolonged contact. Robustly disentangling several in-contact subjects is a challenging task due to occlusions and complex shapes. Hence, existing multi-view systems typically fuse 3D surfaces of close subjects into a single, connected mesh. To address this issue we leverage i) individually fitted neural implicit avatars; ii) an alternating optimization scheme that refines pose and surface through periods of close proximity; and iii) thus segment the fused 4D raw scans into individual instances. From these instances we compile a dataset Hi4D, which contains rich interaction centric annotations in 2D and 3D alongside accurately registered parametric body models. We define varied human pose and shape estimation tasks on this dataset and provide results from state-of-the-art methods on these benchmarks.

Video

Method

Vision-based disentanglement of in-contact subjects is a challenging task due to strong occlusions and a-priori unknown geometries. Hence, multi-view systems typically fuse 3D surfaces of close subjects into a single, connected mesh. In this paper, we aim to segment 4D scans of closely interacting people to obtain instance-level annotations. Our method makes use of two main components:

Dataset

Hi4D is the first dataset containing rich interaction centric annotations and high-quality 4D textured geometry of closely interacting humans.

4D Textured Scans

Instance Meshes with Contact Annotations

Instance Segmentation Masks in 2D and 3D

Parametric Body Models with Contact Annotations

Download

Download instructions will be available soon.

BibTeX

@inproceedings{yin2023hi4d,
      author = {Yin, Yifei and Guo, Chen and Kaufmann, Manuel and Zarate, Juan and Song, Jie and Hilliges, Otmar}, 
      title = {Hi4D: 4D Instance Segmentation of Close Human Interaction}, 
      booktitle = {Computer Vision and Pattern Recognition (CVPR)},
      year = {2023}
      }