Google has been doing some pretty cool things with AI, but one of the coolest applications of the technology has to be the newly revealed Project Gameface, a hands-free AI-powered gaming mouse that the company announced at Google I/O 2023.
With the help of a webcam, one could use Project Gameface to play PC games that would normally require a mouse for movement, thanks to the AI tech. It works by utilizing the webcam to scan the face and look for head movements and facial gestures. Those movements and gestures then translate to in-game movements of one’s character.
The controls are rather robust and intuitive too. The Gameface software allows you to set up key and mouse bindings for pretty much everything. Opening your mouth for instance could be set to the mouse’s left click. Though there are all kinds of other movement or gesture options as well. Including smiling, looking left, or raising your left eyebrow. You can even set how prominent those movements or gestures would need to be to relay the input. Which should help the software be more adaptable to the needs of different users.
Project Gameface uses Google’s open-source MediaPipe solution, which links together different AI models to give you an output. The output is “a mesh of 468 points on your face that are then converted into telemetry like mouse movement or clicks,” Google’s Lead AI Advocate Laurence Moroney says.
Moroney states that it runs natively on devices and “one of the most amazing things about the technology is that only needs an input from a webcam.”
Project Gameface is co-designed with game streamer Lance Carr
You might be wondering who Lance Carr is. Well, let us introduce you. Lance Carr is a game streamer who lives with muscular dystrophy. If you’re familiar at all with the disease, then you know it weakens your muscles and can make it more challenging to do almost everything. Including gaming. This is where Google comes in. Lance’s house burned down back in 2021 while on stream playing Diablo II. Along with all his expensive equipment, including the adaptive head-tracking mouse that allowed him to play games.
Google caught wind of the story and reached out to Lance. Work soon began between Google and Lance to co-design Project Gameface. Lance’s hope is that this technology can be given to everybody who could use it to make their lives easier.
Google has made the code for the technology open-source as well, and it’s even available in a preview from today for those who want to check it out over at Google’s GitHub repository.