NToP

News

2024-10-30 - Code released on Github.
2024-08-25 - Paper accepted by ECCV 2024 Synthetic Data for Computer Vision Workshop.
2024-08-15 - OmniLab with images, bounding boxes and keypoints is published.
2024-04-24 - NToP paper v2 is published on arXiv.

Abstract

Due to restrictions by the origin dataset licenses, we cannot publish the subsets of NToP that used Human3.6M and Genebody as origin datasets. To download the NToP ZJU-MoCap subset, please first fill out the google form (which is shared with Qing Shuai) and write an email to Jingrui Yu. The link for dataset download will then be sent to you shortly. Alternatively, You can use our git repo to train and render your own dataset.

Human pose estimation (HPE) in the top-view using fisheye cameras presents a promising and innovative application domain. However, the availability of datasets capturing this viewpoint is extremely limited, especially those with high-quality 2D and 3D keypoint annotations. Addressing this gap, we leverage the capabilities of Neural Radiance Fields (NeRF) technique to establish a comprehensive pipeline for generating human pose datasets from existing 2D and 3D datasets, specifically tailored for the top-view fisheye perspective. Through this pipeline, we create a novel dataset NToP (NeRF-powered Top-view human Pose dataset for fisheye cameras) with over 570 thousand images, and conduct an extensive evaluation of its efficacy in enhancing neural networks for 2D and 3D top-view human pose estimation. Extensive evaluations on existing top-view 2D and 3D HPE datasets as well as our new real-world top-view 2D HPE dataset OmniLab prove that our dataset is effective and exceeds previous datasets in this field of research.

The render quality of NToP is compared with the only other top-view HPE datasets available (as of August 2024) in the figure below.

Paper

arXiv

Code

Github

BibTeX

If you use the data set in your work, please don't forget to cite:

@misc{yu2024ntop,
      title={NToP: NeRF-Powered Large-scale Dataset Generation for 2D and 3D Human Pose Estimation in Top-View Fisheye Images}, 
      author={Jingrui Yu and Dipankar Nandi and Roman Seidel and Gangolf Hirtz},
      year={2024},
      eprint={2402.18196},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2402.18196}, 
}