Hands on: Structure Sensor turns your iPad into a 3D scanning machine

Last year, California-based Occipital closed out a wildly successful Kickstarter for its Structure Sensor, an iPad-mountable structured light scanner that can perceive the world in three dimensions by projecting infrared dots on things. The company is nearly done shipping the $399 devices to its crowdfunding backers (non-backers who buy one now can expect delivery in about a month), but we managed to finagle a couple of days with an early production version of the Structure Sensor to get a peek at how it performs in the real world. We learned a lot about what the Structure Sensor is—and we learned even more about what it isn’t.

We received a pair of boxes from Occipital: one contained the Structure Sensor and its dedicated Lightning cable, and the other contained the bracket necessary to attach the Structure to our iPad mini (the Structure works with any iPad with a Lightning connector, but you need the correct bracket to mount it). The device itself feels agreeably solid—it's wrapped in an anodized aluminum housing with a glass front, through which are visible two infrared LEDs, the infrared structured light projector, and the infrared camera. It screws into the provided mounting bracket, then slips onto your iPad and is secured in place with a clamp. Once attached, you then connect a custom Lightning cable from the Structure to the iPad’s port on its base. The entire assembly of sensor, bracket, and cable add about 124 grams of weight onto the back of the iPad.

The Structure, though, isn’t so much a scanner as it is a platform. There are mobile and desktop apps you can use to make it work, but at least at this stage in the product’s development those apps are still a bit sketchy. They function more as demonstrations of the scanner’s potential uses than actual apps you’d want to use. The biggest draw about the Structure is its openness: the iOS SDK is available right now, which can be used to create Structure-compatible iOS apps immediately. Other platforms (Windows, OS X, and Android) will be getting open source drivers in the near future, though creating Structure apps for anything other than iOS will require the use of a special "hacker cable" to connect to the Structure.

A quick note before some reactions to the Structure: performing a full in-depth review of the scanner would mean dedicating time to actually exploring the SDK and creating a few apps; unfortunately, your humble reviews editor lacks the skill to walk fully down that path. As such, this is more of a quick hands on with the device.

Seeing through its eyes

The Structure unit we were provided with came with links to download a number of prerelease iOS apps with which to try out the scanner. The main "Structure" app shows what the Structure Sensor sees—it can provide you with a pure IR image from the sensor, a "depth" view showing the real-time interpretation of the structured light projection data from the sensor, and a "depth + color" view that combines the depth view with the image from the iPad’s standard camera. The app also enables you to link the sensor via Wi-Fi to a desktop application like Skanect and scan directly to a computer instead of to the iPad. The IR-only mode also lets you toggle on the two bright infrared LEDs, thus giving your iPad a "night vision" mode similar to that on some consumer camcorders.

On our test second-generation iPad mini (the retina model), the output from the scanner in the main Structure app was snappy and low-lag. Occipital’s marketing materials claim that the scanner can generate a "VGA depth stream" at 30 fps, and that jives with what we observed on the iPad’s screen.

How it works

The principles are mostly the same with the Structure Sensor. The Structure projects a unique infrared pattern of dots out in front of it, and the infrared camera uses that pattern of dots to visualize the shape and distance of objects. The projection is done with a dedicated infrared laser projector (the two bright infrared LCDs on the scanner aren’t used for 3D scanning). This type of structured light projection and detection is also what Microsoft’s Kinect sensors use to work.This isn’t the first time Ars has gotten some hands-on time with structured light scanning. Previously, we got to fiddle around with a much larger-scale structured light scanning operation at NASA’s Marshall Space Flight Center in Huntsville, Alabama, where engineers were using large mobile structured light scanning rigs to visualize massive Rocketdyne F-1 rocket engines of the same type that powered Saturn V rockets off of the pad in the 1960s and '70s.

Users interested in actually developing their own apps will no doubt get a lot out of the main Structure app, since it provides a good overview of the device’s capabilities. However, aside from showing the sensor output and letting you update your Structure Sensor’s firmware, the app doesn’t really do anything. To actually make the scanner produce something, we need to either link it up to a desktop computer, or we need to fire up some additional apps.

Scanning stuff

Occipital provided us with several additional iOS apps to make the scanner actually do stuff. One, simply called "Scanner," allows you to slowly pan the Structure around an object to capture a 3D model of it. As you move, the iPad’s display shows you the object you’re scanning and a real-time point cloud of the object being scanned; the more you scan, the more the point cloud fills in with actual details. The app can then deliver (via e-mail) a Wavefront .obj file of the scanned object, which can then be cleaned up in any 3D application that supports .obj files. From there, you can use the scanned .obj in another 3D project or convert it to a format suitable for 3D printing.

However, by itself the scanner has no way to detect standard visible colors—it operates purely in the infrared. The .obj files it produces do not include any accompanying textures. To actually get a fully detailed scanned object, we need to turn to a different app: itSeez3D.

Like the default Structure app, itSeez3D combines the output of the Structure’s infrared camera with the iPad’s built-in color camera to simultaneously scan both an object’s shape and also the visual details of its surface. Actually pulling a good scan requires a carefully prepared scanning surface with lots of details for the app to use for reference (the tutorial video recommends putting some magazines down around your object to create an appropriately busy visual area), but the results we got in several test scans were actually not terrible!

The most complex object we tried to scan was our loaner DJI Phantom 2 Vision+ quadcopter. The ventral surface of the drone has a lot of intricate detail, being taken up by the drone’s integrated gimbal and video camera. itSeez3D managed to produce a somewhat-blobby but still clearly recognizable model of the drone.

We had less-stellar luck with the joystick from our Thrustmaster Warthog HOTAS setup. The joystick’s black metallic surface didn’t appear to throw back enough of an infrared reflection for the camera to see its projected infrared laser mesh. In spite of a lot of time spent scanning, we ended up with only a partial object.

We also found that the structured light scanning mesh can have a hard time picking up on small details, like the gooseneck on our Bona Vita electric kettle. Most of the rest of the kettle is modeled and textured correctly, but the neck appears to have been small enough to slip through the mesh and go unscanned.

We also tried wirelessly linking the main Structure app to Skanect, an application running on our desktop computer. When linked, Skanect was able to use the Structure sensor to capture an extremely wide and detailed swath of the real world—including my desk, my keyboard, my mouse, and some of my legs.

Leaving iOS and the nature of openness

Out of the box, you can immediately hook the Structure Sensor up to an iOS device, but what if Apple isn’t your platform of choice? The Structure Sensor has a proprietary plug, and if you want to connect the Structure to an Android device (or to a Windows or OS X desktop), you’ll need the $39 "Hacker Cable" accessory.

There appears to be a bit of a disconnect between the Structure Sensor’s positioning as an open development platform and the requirement to purchase a proprietary cable to actually use the Structure on non-iOS devices. When we contacted Occipital and asked about this, a spokesperson sent us the following response:

The Hacker Cable is a USB 2.0 cable, but terminated on one end with the special connector that the Structure Sensor uses....We thought about using a micro USB connector for the sensor's data port, but unfortunately, these connectors simply don't have enough wires for us to support both iOS devices and USB devices, since the requirements are different for both. In order to make both work, we needed more than the four wires that micro USB connector is able to carry, and so we had to use a connector with more pins.

Another option could have been to add two different data ports (one for Lightning, one for USB) on the sensor, but we felt this would have been confusing to users, and it would have added cost to the sensor itself, which everyone would have needed to pay, even if they only needed one of the two ports.

In the end, we created a single data port that (we think) elegantly supports both Apple Lightning devices as well as standard USB devices.

The spokesperson also noted that the Hacker Cable will let the Structure Sensor parasitically charge while plugged in—something that the Lightning cable doesn't allow. Without the Hacker Cable, you need to plug the Structure into its included AC adapter to charge. This isn't a huge problem—after several hours of usage, the Structure was still reading over 50% of its battery capacity available. We were able to do the entire review on a single charge.

The takeaway: needs more apps

The Structure Sensor is really more of a greenfield development tool right now than it is a consumer device. It’s got a huge amount of promise, but there aren’t yet a lot of ways for normal folks to pick up and use the device (though if you’ve got access to a 3D printer, it’s a cool way to quickly create 3D meshes from small objects).

Fortunately for Occipital, their smashing Kickstarter success has already shown that they have a large audience out there. The company is going to be somewhat dependent on that audience to drive further adoption—it seems, at least for now, that consumer-produced apps are going to be responsible for the bulk of the Structure Sensor’s uses.

If you want to get your hands on one, they’re available for sale at Structure.io. A bundle containing the Structure Sensor, a license for the pro version of the desktop Skanect application, an iPad bracket, and the Hacker Cable costs $499, while the sensor by itself costs $399. The devices are shipping now, but delivery times for non-Kickstarter customers are currently about 30 days.