[ad_1]
Zoom calls have exploded during the current coronavirus crash. But if you have already exhausted the possibilities of different angles and backgrounds for your video calls, why not try to be a completely different person? You know, like former Apple CEO and co-founder Steve Jobs. ->
At least, that’s the tech demo / experiment created by coders Ali Aliev and Karim Iskakov. They have developed an Animoji or Memoji-style tool called Avatarify that allows users to overlay themselves a mask in real time during video calls. ->
->How does Avatarify work?
->Avatarify is based on an artificial neural network called First Order Motion Model, developed by researchers in Italy. Consisting of more than 12,000 videos, it makes it possible to animate an avatar image without manual adjustment. That means users only need to add a still image of one face. Avatarify will turn it into an animated mask.
It works by extracting key points from the webcam video of your real face, and then tracking key points as you move. Then apply that information to the avatar image to move, for example, their nose at the same time as yours. The animated image is transmitted to Zoom, Skype, or any other video calling service you want to use.
To cheer up a person, simply place their image in a specific folder, start the Avatarify app, and then start a video call.
“That idea came up spontaneously when I came across the first-order movement model,” said Ali Aliev. Cult of mac. “I was surprised by his performance in terms of animation quality … I decided to make fun of my colleagues, [and] quickly prototyped and entered our weekly Zoom call with [the] face of the famous MMA fighter Khabib Nurmagomedov. They appreciated the joke. Karim, who is also my colleague, had an idea. He ported it to Mac and wrote the video with the fake Elon Musk. “
The surprisingly convincing results can be seen below.
The Steve Jobs Preset
When they decided to post Avatarify, the couple made the decision to include an avatar preset. “We admire Steve for his commitment to great ideas, so I definitely wanted to have him in the avatar preset,” Aliev said.
The results are not perfect, of course. The model was trained in 256×256 image cropping, meaning the quality, while perfectly acceptable for Zoom calls, won’t exactly stand up to HD scrutiny. Better AI training, particularly at extreme facial angles, would help improve that.
More significantly, unlike the famous deepfake videos you’ve probably seen, Avatarify works with 2D images. As Aliev said, he knows nothing about the 3D world. That’s more noticeable when it comes to head rotations. (Again, think of this as wearing a flat cardboard mask, rather than a 3D mask.)
Photo: Karim Iskakov / Ali Aliev
However, Avatarify does have some limitations.
“The other side of the problem is performance,” he said. “Right now you need a GPU computer to run Avatarify with a reasonable 30 FPS. Run it on a single CPU device [is insufficient] for a pleasant video conference. We believe that it is possible to speed up the model to work in real time on a CPU machine [such as a] MacBook but it is a research problem that requires a lot of resources and requires a lot of effort and time. Another option to improve the performance we are analyzing is to use GPU in the cloud. [That would mean] all heavy calculations are done elsewhere, but not on your laptop. “
->The last problem, inescapable for anyone watching the Elon Musk demo, is that the voice sounds muffled. At the end of the day, it’s still your voice coming out of someone else’s face. In fact, there have been some pretty impressive (and terrifying) deepfake audio demos recently, able to replicate the voices of famous people. But those couldn’t easily be used here. So maybe he’ll start practicing his impressions!
Avatarify can be downloaded from the Github online code repository. It is free and open source, although you will need some basic coding knowledge to get it working. After that? It’s just a matter of waiting for your next Zoom call …
[ad_2]
