Beauty in Details: HSE University and AIRI Scientists Develop a Method for High-Quality Image Editing
Researchers from the HSE AI Research Centre, AIRI, and the University of Bremen have developed a new image editing method based on deep learning—StyleFeatureEditor. This tool allows for precise reproduction of even the smallest details in an image while preserving them during the editing process. With its help, users can easily change hair colour or facial expressions without sacrificing image quality. The results of this three-party collaboration were published at the highly-cited computer vision conference CVPR 2024.
Artificial intelligence is already able to generate and edit images using generative adversarial networks (GANs). The architecture consists of two independent networks: a generator that creates images and a discriminator that distinguishes between real and generated samples. These networks compete with each other, and a new stage in their development is the StyleGAN model. This model can generate images and modify specific parts based on user requests, but it has not been able to work with real photos or images before.
Researchers from the HSE AI Research Centre, the Artificial Intelligence Research Institute (AIRI), and the University of Bremen have proposed a method to quickly and efficiently edit real images. This StyleFeatureEditor approach consists of two modules: the first inverts (reconstructs) the original image, and the second edits this reconstruction. The results of these two steps are passed to StyleGAN, which generates the edited image based on the internal representations. The developers addressed some challenges that had been encountered in previous research. With a small set of representations, the network could edit the image well, but it lost some details from the original. However, with a larger set, all the details were preserved, but the network had difficulty transforming them correctly according to the task.
To solve this, the researchers proposed a new solution: the first module finds both large and small representations, while the second learns how to edit the larger ones using the smaller ones as reference.
However, to train these modules to accurately edit the representations, the neural network requires both real images and their edited versions.
‘We needed examples, such as the same face with different expressions, hairstyles, and details. Unfortunately, such image pairs do not exist at the moment. So, we came up with a trick: using a method that works with small representations, we created a reconstruction of a real image and an example of editing this reconstruction. Although the examples were relatively simple and without details, the model clearly understood how to make the edits,’ explains Denis Bobkov, one of the authors of the article, a research intern at the Centre of Deep Learning and Bayesian Methods of the AI and Digital Science Institute (part of the HSE Faculty of Computer Science), and a Junior Research Fellow at AIRI’s Fusion Brain Lab.
However, training only on generated (simple) examples leads to a loss of detail when working with real (complex) images. To prevent this, the researchers added real images to the training dataset, and the neural network learnt to reconstruct them in detail.
Thus, by showing the model how to edit both simple and complex images, the scientists created conditions under which the network could edit complex images more effectively. In particular, the developed approach handles adding new elements of style while preserving the details of the original image better than other existing methods.
In the case of simple reconstruction (first row), StyleFeatureEditor accurately reproduced a hat, while most other methods almost completely lost it. The developed method showed the best results with additional accessories (third row): most methods could add glasses, but only the StyleFeatureEditor retained the original eye colour.
‘Thanks to this training technique on generated data, we have obtained a model with high editing quality and a fast processing speed due to the use of relatively lightweight neural networks. The StyleFeatureEditor framework requires only 0.07 seconds to edit a single image,’ says Aibek Alanov, Head of the Centre of Deep Learning and Bayesian Methods of the AI and Digital Science Institute (part of the HSE Faculty of Computer Science), and leader of the research group ‘Controlled Generative AI’ at AIRI's Fusion Brain Lab.
The research was funded by a grant from the Analytical Centre under the Government of the Russian Federation for AI research centres.
The research results will be presented at the Fall into ML 2024 conference on artificial intelligence and machine learning, which will take place at HSE University on October 25–26, 2024. Leading AI scientists will discuss the best papers published at top-tier (A*) flagship AI conferences in 2024. A demo of the developed method can be tried out on HuggingFace, and the source code is available on GitHub.
See also:
HSE Scientists Have Examined Potential Impact of Nuclear Power on Sustainable Development
Researchers at HSE University have developed a set of mathematical models to predict the impact of nuclear power on the Sustainable Development Index. If the share of nuclear power in the global energy mix increases to between 20% and 25%, the global Sustainable Development Index (SDI) is projected to grow by one-third by 2050. In scenarios where the share of nuclear power grows more slowly, the increase in the SDI is found to be lower. The study has been published in Nuclear Energy and Technology.
HSE Scientists Have Developed a New Model of Electric Double Layer
This new model accounts for a wide range of ion-electrode interactions and predicts a device's ability to store electric charge. The model's theoretical predictions align with the experimental results. Data on the behaviour of the electric double layer (EDL) can aid in the development of more efficient supercapacitors for portable electronics and electric vehicles. The study has been published in ChemPhysChem.
Psychologists from HSE University Discovered How Love for Animals Affects Relationships with People
Researchers from HSE University have identified a connection between attachment to pets and attitudes toward nature and other people. The study found that the more joy people derive from interacting with their pets, the more they want to help others. However, love for animals is not always associated with concern for nature. The findings were published in the Social Psychology and Society journal.
HSE Scientists Propose Using Heart Rate Analysis to Diagnose Anxiety and Depression
A group of scientists at HSE University have discovered how anxiety and depression can be diagnosed by analysing heart rate. It turns out that under mental stress, the heart rate of individuals with a predisposition to mental health disorders differs from that of healthy individuals, especially when performing more complex tasks. These changes in cardiovascular parameters can even be detected using a pulse oximeter or a smartwatch. The study findings have been published in Frontiers in Psychiatry.
Researchers at HSE in St Petersburg Develop Superior Machine Learning Model for Determining Text Topics
Topic models are machine learning algorithms designed to analyse large text collections based on their topics. Scientists at HSE Campus in St Petersburg compared five topic models to determine which ones performed better. Two models, including GLDAW developed by the Laboratory for Social and Cognitive Informatics at HSE Campus in St Petersburg, made the lowest number of errors. The paper has been published in PeerJ Computer Science.
Narcissistic and Workaholic Leaders Guide Young Firms to Success
Scientists at HSE University—St. Petersburg studied how the founder's personal characteristics impact a young firm's performance. It turns out that a narcissist and workaholic who also fosters innovation will effectively grow their company. The paper has been published in IEEE Transactions on Engineering Management.
Biologists at HSE University Warn of Potential Errors in MicroRNA Overexpression Method
Researchers at HSE University and the RAS Institute of Bioorganic Chemistry have discovered that a common method of studying genes, which relies on the overexpression of microRNAs, can produce inaccurate results. This method is widely used in the study of various pathologies, in particular cancers. Errors in experiments can lead to incorrect conclusions, affecting the diagnosis and treatment of the disease. The study findings have been published in BBA.
Green Energy Patents Boost Company Profitability
An ESG strategy—Environmental, Social, and Corporate Governance—not only helps preserve the environment but can also generate tangible income. Thus, the use of renewable energy sources (RES) and green technologies in the energy sector enhances return on investment and profitability. In contrast, higher CO2 emissions result in lower financial performance. This has been demonstrated in a collaborative study by the HSE Faculty of Economic Sciences and the European University at St. Petersburg. Their findings have been published in Frontiers in Environmental Science.
HSE Scientist Optimises Solution of Hydrodynamics Problems
Roman Gaydukov, Associate Professor at the MIEM HSE School of Applied Mathematics, has modelled the fluid flow around a rotating disk with small surface irregularities. His solution allows for predicting fluid flow behaviour without the need for powerful supercomputers. The results have been published in Russian Journal of Mathematical Physics.
Neuroscientists from HSE University Learn to Predict Human Behaviour by Their Facial Expressions
Researchers at the Institute for Cognitive Neuroscience at HSE University are using automatic emotion recognition technologies to study charitable behaviour. In an experiment, scientists presented 45 participants with photographs of dogs in need and invited them to make donations to support these animals. Emotional reactions to the images were determined through facial activity using the FaceReader program. It turned out that the stronger the participants felt sadness and anger, the more money they were willing to donate to charity funds, regardless of their personal financial well-being. The study was published in the journal Heliyon.