This is how Microsoft built its smart Surface camera

“From day one of the Surface Hub 2, we knew we were going to make our cameras smart,” explains Steven Bathiche, who oversees all hardware innovation for Microsoft devices, in an interview with The edge. Microsoft’s surprise $799.99 Surface Hub 2 smart camera was introduced last week and offers automatic reframing without the distortion and distortion you usually see with other conference room cameras.

It can recognize faces and bodies to ensure everyone in a room is visible during meetings, whether they’re close to the camera or up to eight meters away. The Surface Hub 2 Smart Camera can cover virtually an entire conference room thanks to its 136-degree field of view that keeps people in front in focus next to those behind.

Microsoft had always planned to upgrade its Surface Hub 2 camera before the pandemic brought hybrid meetings into focus, so it’s modular and detachable from the top of either 55- or 85-inch displays. “We knew we would continue to develop the experience. We didn’t know exactly how, but we knew that was going to change and would change with people’s needs, the conference space evolving, and even the way our culture will essentially adapt to meetings,” says Bathiche.

Large devices like the 85-inch Surface Hub 2 presented challenges when it came to capturing everyone in a meeting room with a traditional camera. “We needed a camera for larger spaces,” says Bathiche, so Microsoft got to work.

Bathiche and his team developed Microsoft’s own optics, AI model, and edge computer to step into the Surface Hub 2 smart camera and power their computational photography. “It has built-in processing power, 1 teraflop of processing power, which essentially houses a really big AI model that we built,” says Bathiche. “It contains the autoframing application, it’s in the camera, so what comes out is just a 4K image, so it literally looks like a webcam to the Surface Hub.”

This means that all the AI ​​work is done on the camera itself and never sent to the cloud for processing or even to the Surface Hub 2 over the cable. The camera runs the AI ​​model, processes all the data and makes the decision to crop the image accordingly. While auto-framing can automatically capture everyone in a room, the smart camera also uses tilt compensation to adjust the image to the camera position and create more natural eye contact instead. It’s also able to remove the fisheye effect from wide-angle lenses so people in meeting rooms don’t look distorted or stretched.

“We created an 11-element, all-glass lens that was super sharp in focus and basically close to the refractive limit,” explains Bathiche. Behind the lens is a 12-megapixel (4000 x 3000) sensor with an f/1.8 aperture, all of which produce the cropped 4K image. “The actual lens has a 184-degree field of view, allowing the camera to look behind you.”


Microsoft has developed custom parts for its Surface Smart Camera.
Image: Microsoft

However, all that hardware is nothing without the AI ​​models powering the Surface Smart Camera. Microsoft started this project before the pandemic, but needed to train its AI models while the pandemic, which presented the obvious challenges of filling meeting rooms with people.

“We went to New Zealand because there were no COVID-19 cases there and we had offices there,” explains Bathiche. “We hired actors and actresses to collect data in all sorts of spaces. Our dataset is absolutely massive.”

Microsoft has trained its AI model on faces and bodies to ensure it’s fully integrated and recognizes people who aren’t always facing the camera. It even used synthetic people and faces to enhance its diversity across situations and people. “We have some really cool in-house technology that can generate synthetic data, so we were able to generate synthetic people and faces,” adds Bathiche.

However, the smart camera is not trained to recognize pets or animals. So that should mean that it doesn’t attempt to automatically rearrange a meeting if an office cat or dog comes into view. Microsoft has also applied its responsible AI principles on this project, which includes a committee and a set of tools to ensure the fairness and inclusivity of AI.


The Surface Smart Camera has a built-in computer.
Image: Microsoft

“If you look at our dataset, it’s absolutely amazing in terms of the differences between different groups: race, gender, skin color, hairstyles, etc.,” explains Bathiche. “I think one of the things that’s built into the camera that people might not see on the box is the ruggedness and inclusivity that the model has.”

Bathiche says Microsoft has “sat and the hell tuned” its smart camera’s autoframing abilities over the past year to make sure it’s not too erratic or too slow to miss content. “Each image that the camera receives decides whether it’s worth shifting or recropping the image.”

You might be wondering if you could use this $799.99 camera on a regular Windows PC, but it’s not quite that simple. While all the computing and AI models are housed in the Surface Hub 2 smart camera, it’s not really designed as a regular webcam. “His design point was specific to Hub. The altitude, angles, and AI are designed for multiple people, both near and far,” explains Bathiche. “While you could technically design a mount and plug it into a PC, I don’t think it’s going to work as well as you’d like.”

This isn’t the first time Microsoft has focused on improving its webcams and cameras, either. The Surface Pro X already has one AI-assisted eye contact function This makes it seem like you’re always making eye contact during a video call, no matter what you’re looking at. Apple added a similar FaceTime attention correction feature to iOS 13. “The algorithms we used on eye contact [for the Surface Pro X] are the same algorithms for the faces that we use in this camera,” says Bathiche.

Microsoft clearly designed this smart camera for the Surface Hub 2, but with persistent rumors surrounding Surface-branded webcams, it’s possible that one day we’ll see a powerful webcam from Microsoft affordable ones available today. “This area of ​​using computers to bring people together and make them feel like they’re in the same room… I think it’s something we’ve always been passionate about and will continue to do, and we will.” we continue to develop devices like the ones you see in the Surface Pro X,” says Bathiche. This is how Microsoft built its smart Surface camera

Fry Electronics Team

Fry is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – The content will be deleted within 24 hours.

Related Articles

Back to top button