Our main research lines are Multimodal Deep Learning, Computer Vision, Synthetic Data Generation. Although our research usually focus on the intersection of those three fields, we are also open to applying our knowledge to cross-disciplinary problems that can be solved by applying artificial intelligence or computer vision techniques.
Augmented reality, 3D reconstruction, synthetic data, digital twins
Projects
Our projects
in2IN
Leveraging individual Information to Generate Human Interactions
UnrealFall
Synthetic Action Video Generation
HoloDemtect
A mixed reality framework for cognitive stimulation through interaction with objects
Holo4Care
A MR framework for assisting in activities of daily living by context-aware action recognition
UnrealHandGrasp
Interaction with objects in Virtual Reality using hand tracking
UnrealROX
An Improved Tool for Acquiring Synthetic Data from Virtual 3D Environments
in2IN
Leveraging individual Information to Generate Human Interactions
Generating human-human motion interactions conditioned on textual descriptions is a very useful application in many areas such as robotics, gaming, animation, and the metaverse. Alongside this utility also comes a great difficulty in modeling the highly dimensional inter-personal dynamics. In addition, properly capturing the intra-personal diversity of interactions has a lot of challenges. Current methods generate interactions with limited diversity of intra-person dynamics due to the limitations of the available datasets and conditioning strategies. For this, we introduce in2IN, a novel diffusion model for human-human motion generation which is conditioned not only on the textual description of the overall interaction but also on the individual descriptions of the actions performed by each person involved in the interaction. To train this model, we use a large language model to extend the InterHuman dataset with individual descriptions. As a result, in2IN achieves state-of-the-art performance in the InterHuman dataset. Furthermore, in order to increase the intra-personal diversity on the existing interaction datasets, we propose DualMDM, a model composition technique that combines the motions generated with in2IN and the motions generated by a single-person motion prior pre-trained on HumanML3D. As a result, DualMDM generates motions with higher individual diversity and improves control over the intra-person dynamics while maintaining inter-personal coherence.
UnrealFall
Synthetic Video Data Generation Framework
Humans perform a variety of actions, some of which are infrequent but crucial for data collection. Synthetic generation techniques are highly effective in these situations, enhancing the data for such rare actions. In response to this need, we present UnrealFall, a robust framework developed within Unreal Engine 5, designed for the generation of human action video data in hyper-realistic virtual scenes. It addresses the scarcity and limited diversity in existing datasets for actions like falls by leveraging synthetic motion generation through text-guided generative models, Gaussian Splatting technology, and MetaHumans. The usefulness of the framework is demonstrated by its capability to produce a synthetic video dataset featuring elderly individuals falling in various settings. The value of the dataset is demonstrated by its successful use in training a VideoMAE model, in conjunction with the UCF101 and various fall-specific datasets. This versatility in generating data across a spectrum of actions and environments positions our framework as a valuable tool for broader applications such as digital twin creation and dataset augmentation. The code and data are available for research at project website,
darkviid.github.io/UnrealFall/.
HoloDemtect
A mixed reality framework for cognitive stimulation through interaction with objects
The evolution of augmented reality (AR), with an improvement in its usability, allows more users to interact with this type of devices such as the HoloLens 2. One of the main fields benefiting greatly of these advances is healthcare, particularly for rehabilitation and well being. In AR, users are able to see the real world, reducing potential dizziness caused by other devices. In this paper we present HoloDemTect, an AR application to stimulate the interaction of patients with objects while performing different tasks. They are asked to perform tasks in which their motor and cognitive abilities are stimulated, while the system collects relevant information (e.g. eye tracking, hand positions and head movements) to reconstruct their actions. Our system creates a realistic experience with high immersion that we evaluated with a set of exercises performed by volunteers by a qualitative and quantitative analysis.
Holo4Care
A MR framework for assisting in activities of daily living by context-aware action recognition
The evolution of virtual and augmented reality devices in recent years has encouraged researchers to develop new systems for different fields. This paper introduces Holo4Care, a context-aware mixed reality framework designed for assisting in activities of daily living using the HoloLens 2. By leveraging egocentric cameras embedded in these devices, which offer a close-to-wearer perspective, our framework establishes a congruent relationship, facilitating a deeper understanding of user actions and enabling effective assistance. In our approach, we extend a previously established action estimation architecture after conducting a thorough review of state-of-the-art methods. The proposed architecture utilizes YOLO for hand and object detection, enabling action estimation based on these identified elements. We have trained new models on well-known datasets for object detection, incorporating action recognition annotations. Unlike previous systems that focus primarily on user evaluation, Holo4Care emphasizes assistance by providing a set of global actions based on the user’s field of view and hand positions that reflect their intentions. Experimental results demonstrate Holo4Care’s ability to assist users in activities of daily living and other domains.
UnrealHandGrasp
Interaction with objects in Virtual Reality using hand tracking
In this project has been developed an object grasping system in Unreal Engine 4. For the creation of this system, we have started from a base project named UnrealGrasp that has been modified to use the hand detection framework offered by Oculus and to be able to interact with the objects in the virtual environment only with our own hands.The work consisted in the implementation of an algorithm that decides when to move and lock the fingers of our virtual hand, as well as when we are grasping an object and when it has been released. Once this was done, it can be used to interact in an environment withmany objects. Different people have been asked to grasp different objects as they spawn in the environment. These objects appear sequentially and in different orientations to encourage differenttypes of grasping. This information has been collected in the format of a heat map, creat-ing a dataset of 12 objects that can be used in different fields, such as training an artificial intelligence system or making a more exhaustive study on grasping. Finally, an environment has been created in which the user can move and interact with objects of all kinds: toys, technical tools, laboratory utensils or foodstuffs, among others. This is intended to demonstrate the wide range of applications of this new technology.
UnrealROX
An Improved Tool for Acquiring Synthetic Data from Virtual 3D Environments
Data-driven algorithms have surpassed traditional techniques in almost every aspect in robotic vision problems. Such algorithms need vast amounts of quality data to be able to work properly after their training process. Gathering and annotating that sheer amount of data in the real world is a time-consuming and error-prone task. Those problems limit scale and quality. Synthetic data generation has become increasingly popular since it is faster to generate and automatic to annotate. However, most of the current datasets and environments lack realism, interactions, and details from the real world. UnrealROX is an environment built over Unreal Engine 4 which aims to reduce that reality gap by leveraging hyperrealistic indoor scenes that are explored by robot agents which also interact with objects in a visually realistic manner in that simulated world. Photorealistic scenes and robots are rendered by Unreal Engine into a virtual reality headset which captures gaze so that a human operator can move the robot and use controllers for the robotic hands; scene information is dumped on a per-frame basis so that it can be reproduced offline to generate raw data and ground truth annotations. This virtual reality environment enables robotic vision researchers to generate realistic and visually plausible data with full ground truth for a wide variety of problems such as class and instance semantic segmentation, object detection, depth estimation, visual grasping, and navigation.
About Us
our story
The 3D Perception Lab at the University of Alicante consists of a team of researchers dedicated to exploring the intersection of deep learning and computer vision. Our research mission covers various aspects of perception, utilizing video data as our primary source of information. Our work primarily focuses on multimodal deep learning techniques and applications in virtual reality (VR) and augmented reality (AR) using devices such as the Oculus Quest and Hololens. Among our numerous projects, we highlight two in particular: the detection of dementia symptoms in elderly individuals and the analysis of bird behavior in natural parks. Another key area of our research is the synthetic generation of data using generative models and graphical engines like Unreal Engine 5.
