AI News, ROS 3D Kinect Contest Winners

ROS 3D Kinect Contest Winners

This is maybe only peripherally (ha!) related to robotics, but it’s cool enough that I thought it was worth sharing… Besides, it’s Friday, and you deserve some nifty videos to watch.

Anyway, we’ve posted before on all the cool things that roboticists have been able to do with Microsoft’s stupidly cheap and effective 3D camera system, and Willow Garage took some initiative and sponsored a contest to try and kick start even more open source Kinect innovation.

The second and third places went to robots we’ve covered before, namely UC Berkeley’s quadrotors and Taylor Veltrop’s teleoperation, while a real-time color 3D mapping tool from the University of Freiburg took home the most useful award.

March 2011 Archives

Now that quality 3D point cloud sensors like the Kinect are cheaply available, the need for a stable 3D point cloud-processing library is greater than ever before.

PCL contains numerous state-of-the art algorithms for 3D point cloud processing, including filtering, feature estimation, surface reconstruction, registration, model fitting and segmentation.

These algorithms can be used, for example, to filter outliers from noisy data, stitch 3D point clouds together, segment relevant parts of a scene, extract keypoints and compute descriptors to recognize objects in the world based on their geometric appearance, and create surfaces from point clouds and visualize them -- to name a few.

Our goal was to create a library that can support the type of 3D point cloud algorithms that mobile manipulation and personal robotics need, and try to combine years of experience in the field into coherent framework.

We are now proud to announce that the upcoming release of PCL features a complete Kinect (OpenNI) camera grabber, which allows users to get data directly in PCL and operate on it.

Our development team spawns over three continents and five countries, and it includes prestigious engineers and scientists from institutions such as: AIST, University of California Berkeley, University of Bonn, University of British Columbia, ETH Zurich, University of Freiburg, Intel Research Seattle, LAAS/CNRS, MIT, NVidia, University of Osnabrück, Stanford University, University of Tokyo, TUM, Vienna University of Technology, Willow Garage, and Washington University in St. Louis.


Based around a webcam-style add-on peripheral, it enables users to control and interact with their console/computer without the need for a game controller, through a natural user interface using gestures and spoken commands.[12] The first-generation Kinect was first introduced in November 2010 in an attempt to broaden Xbox 360's audience beyond its typical gamer base.[13] A version for Microsoft Windows was released on February 1, 2012.[6] A newer version, Kinect 2.0, was released with the Xbox One platform starting in 2013.[14] Microsoft released the first Beta of the Kinect software development kit for Windows 7 on June 16, 2011.[15][16][17] This SDK was meant to allow developers to write Kinecting apps in C++/CLI, C#, or Visual Basic .NET.[18][19] Ultimately, the Kinect for either Xbox console did not have long-term popularity, and the two versions were discontinued by April 2016 and October 2017, respectively.[9] The Windows version had been discontinued by April 2015.[7]

Microsoft indicated that the company considers it to be a significant initiative, as fundamental to Xbox brand as Xbox Live,[32] and with a launch akin to that of a new Xbox console platform.[33] Kinect was even referred to as a 'new Xbox' by Microsoft CEO Steve Ballmer at a speech for Executives' Club of Chicago.[34][35] When asked if the introduction will extend the time before the next-generation console platform is launched (historically about 5 years between platforms),[36] Microsoft corporate vice president Shane Kim reaffirmed that the company believes that the life cycle of Xbox 360 will last through 2015 (10 years).[24] During Kinect's development, project team members experimentally adapted numerous games to Kinect-based control schemes to help evaluate usability.

Instead, processing would be handled by one of the processor cores of Xbox 360's Xenon CPU.[39] According to Alex Kipman, Kinect system consumes about 10-15% of Xbox 360's computing resources.[40] However, in November, Alex Kipman made a statement that 'the new motion control tech now only uses a single-digit percentage of Xbox 360's processing power, down from the previously stated 10 to 15 percent.'[41] A number of observers commented that the computational load required for Kinect makes the addition of Kinect functionality to pre-existing games through software updates even less likely, with concepts specific to Kinect more likely to be the focus for developers using the platform.[39] On March 25, 2010, Microsoft sent out a save the date flier for an event called the 'World Premiere 'Project Natal' for Xbox 360 Experience' at E3 2010.

David Dennis, Product Manager at Microsoft, said, 'There are hundreds of organizations we are working with to help them determine what's possible with the tech'.[48] On February 1, 2012, Microsoft released the commercial version of Kinect for Windows SDK and told that more than 300 companies from over 25 countries are working on Kinect-ready apps.[49] The tabloid the New York Post claimed Microsoft had a $500 million budget for advertising the launch of Kinect.[50] While this claim was widely re-reported,[51][52][53] an examination of the 10-Q from Microsoft reveals a 20% year-over-year increase ($85 million) for sales and marketing the quarter Kinect was launched for all of the Entertainment and Devices division, making the total sales and marketing spend $425 million for the entire division.[54] The marketing campaign You Are the Controller, aiming to reach new audiences, included advertisements on Kellogg's cereal boxes and Pepsi bottles, commercials during shows such as Dancing with the Stars and Glee as well as print ads in various magazines such as People and InStyle.[55] On October 19, Microsoft advertised Kinect on The Oprah Winfrey Show by giving free Xbox 360 consoles and Kinect sensors to the people in the audience.[56] Two weeks later, Kinect bundles with Xbox 360 consoles were also given away to the audience of Late Night with Jimmy Fallon.[57] On October 23, Microsoft held a pre-launch party for Kinect in Beverly Hills.

Kinect sensor's microphone array enables Xbox 360 to conduct acoustic source localization and ambient noise suppression, allowing for things such as headset-free party chat over Xbox Live.[12] The depth sensor consists of an infrared laser projector combined with a monochrome CMOS sensor, which captures video data in 3D under any ambient light conditions.[12][23] The sensing range of the depth sensor is adjustable, and Kinect software is capable of automatically calibrating the sensor based on gameplay and the player's physical environment, accommodating for the presence of furniture or other obstacles.[27] Described by Microsoft personnel as the primary innovation of Kinect,[24][75][76] the software technology enables advanced gesture recognition, facial recognition and voice recognition.[25] According to information supplied to retailers, Kinect is capable of simultaneously tracking up to six people, including two active players for motion analysis with a feature extraction of 20 joints per player.[77] However, PrimeSense has stated that the number of people the device can 'see' (but not process as players) is only limited by how many will fit in the field-of-view of the camera.[78] Reverse engineering[79] has determined that the Kinect's various sensors output video at a frame rate of ≈9 Hz to 30 Hz depending on resolution.

For example, Philipp Robbel of MIT combined Kinect with iRobot Create to map a room in 3D and have the robot respond to human gestures,[124] while an MIT Media Lab team is working on a JavaScript extension for Google Chrome called depthJS that allows users to control the browser with hand gestures.[125] Other programmers, including Robot Locomotion Group at MIT, are using the drivers to develop a motion-controller user interface similar to the one envisioned in Minority Report.[126] The developers of MRPT have integrated open source drivers into their libraries and provided examples of live 3D rendering and basic 3D visual SLAM.[127] Another team has shown an application that allows Kinect users to play a virtual piano by tapping their fingers on an empty desk.[128] Oliver Kreylos, a researcher at University of California, Davis, adopted the technology to improve live 3-dimensional videoconferencing, which NASA has shown interest in.[129] Alexandre Alahi from EPFL presented a video surveillance system that combines multiple Kinect devices to track groups of people even in complete darkness.[130] Companies So touch and Evoluce have developed presentation software for Kinect that can be controlled by hand gestures;

Researchers at the University of Minnesota have used Kinect to measure a range of disorder symptoms in children, creating new ways of objective evaluation to detect such conditions as autism, attention-deficit disorder and obsessive-compulsive disorder.[134] Several groups have reported using Kinect for intraoperative, review of medical imaging, allowing the surgeon to access the information without contamination.[135][136] This technique is already in use at Sunnybrook Health Sciences Centre in Toronto, where doctors use it to guide imaging during cancer surgery.[137] At least one company, GestSure Technologies, is pursuing the commercialization of such a system.[138] NASA's Jet Propulsion Laboratory (JPL) signed up for the Kinect for Windows Developer program in November 2013 to use the new Kinect to manipulate a robotic arm in combination with an Oculus Rift virtual reality headset, creating 'the most immersive interface' the unit had built to date.[139] Another application of Kinect is for multi-touch displays.

IGN gave the device 7.5 out of 10, saying that 'Kinect can be a tremendous amount of fun for casual players, and the creative, controller-free concept is undeniably appealing', though adding that for '$149.99, a motion-tracking camera add-on for Xbox 360 is a tough sell, especially considering that the entry level variation of Xbox 360 itself is only $199.99'.[144] Game Informer rated Kinect 8 out of 10, praising the technology but noting that the experience takes a while to get used to and that the spatial requirement may pose a barrier.[143] Computer and Video Games called the device a technological gem and applauded the gesture and voice controls, while criticizing the launch lineup and Kinect Hub.[142] CNET's review pointed out how Kinect keeps players active with its full-body motion sensing but criticized the learning curve, the additional power supply needed for older Xbox 360 consoles and the space requirements.[145] Engadget, too, listed the large space requirements as a negative, along with Kinect's launch lineup and the slowness of the hand gesture UI.

The review praised the system's powerful technology and the potential of its yoga and dance games.[146] Kotaku considered the device revolutionary upon first use but noted that games were sometimes unable to recognize gestures or had slow responses, concluding that Kinect is 'not must-own yet, more like must-eventually own.'[153] TechRadar praised the voice control and saw a great deal of potential in the device whose lag and space requirements were identified as issues.[148] Gizmodo also noted Kinect's potential and expressed curiosity in how more mainstream titles would utilize the technology.[154] Ars Technica's review expressed concern that the core feature of Kinect, its lack of a controller, would hamper development of games beyond those that have either stationary players or control the player's movement automatically.[155] The mainstream press also reviewed Kinect.

Top 10 Robotic Kinect Hacks

At a mere $150, it's a dirt-cheap way to bring depth sensing and 3D vision to robots, and while open-source USB drivers made it easy, a forthcoming Windows SDK from Microsoft promises to make it even easier.

Kinect, which is actually hardware made by an Israeli company called PrimeSense, works by projecting an infrared laser pattern onto nearby objects.

A dedicated IR sensor picks up on the laser to determine distance for each pixel, and that information is then mapped onto an image from a standard RGB camera.

What you end up with is an RGBD image, where each pixel has both a color and a distance, which you can then use to map out body positions, gestures, motion, or even generate 3D maps.

Bilibot The great thing about Kinect is that it can be used to give complex vision to cheap robots, and Bilibot is a DIY platform that gives you mobility, eyes, and a brain in a package that costs just $650.

Car Navigation Back when DARPA hosted their Grand Challenge for autonomous vehicles, robot cars required all kinds of crazy sensor systems to make it down a road.

Willow Garage's TurtleBot Brings Mobile 3D Mapping and ROS to Your Budget

Just a year or two ago, if you'd wanted to buy yourself a mobile robot base with an on-board computer and 3D vision system, you'd probably have been looking at mid-four to five figures.

know I've beaten this to death with respect to Willow Garage and ROS before, but remember that the whole point (or much of the point) of this kind of open source hardware and software is to keep hard working roboticists like you from having to start from scratch every time you want to invent something.

The core kit is $500, which includes: The complete TurtleBot (which is what's in the pictures and video) is $1200, and adds the following to the core kit: The reason that they're doing it this way is to make it as cheap as possible for you to put this kit together yourself, if (say) you have your own laptop already, or even your own iRobot Create.