Skip to main content
Skin and Bones Augmented RealityMobile APP

After the Scan: Augmented Reality in the Hall of Bones and Reptiles

By Rebecca Perry, NMNH on Fri, 11/21/2014 - 16:48

Robert Costello, the National Outreach Program Manager at the Smithsonian’s National Museum of Natural History (NMNH), has been busy time-traveling a 19th-century, 150-pound Mississippi catfish back to life on his digital tablet.

The catfish came to the Smithsonian at the request of Spencer Fullerton Baird, the first curator and later Secretary of the Smithsonian. He asked a fisheries commissioner in Missouri for some “big Mississippi river catfish.” Upon receiving Baird’s letter, the commissioner paid a visit to the St. Louis fish market. This was in November, 1879. He purchased the big catfish, packed it in ice and shipped it express to Washington, DC. Baird received sushi-grade catfish so well preserved he had a cast made. The cast became part of a fisheries exhibition in the late 1800s, but not the skeleton. The skeleton, displayed in a glass case at NMNH’s Bone Hall since 1966, has never been united with its life-like image.

This summer the cast, which was rediscovered as belonging to the skeleton on display, was 3D-scanned by Smithsonian Fellow Diana Marques with the help of Jon Blundell of the Smithsonian’s Digitization Program Office. The cast had some damage, so summer intern, Phat Nguyen, digitally repaired the 3D model of the catfish using ZBrush software. The big Mississippi river blue catfish cast is now reunited with its skeleton in 3D as part of a multimedia and augmented reality (AR) app for museum visitors called Skin & Bones, which was developed for iPads and iPhones.

For the past two years, Costello has been working with a small team of students and media production experts to reinvent the Bone Hall experience with mobile technology Skin and Bones is a reimagining of the NMNH’s Bone Hall using 3D scanning, 3D modeling and animation, video and games to create a fresh interpretation of one of the Smithsonian’s oldest exhibits. Skin & Bones will bring a number of specimens in the gallery of bones to life in the visitor’s hands. Costello, who wanted to preserve the hall’s Victorian Age experience but to overlay it with a new 21st-century experience, chose augmented reality as a cost-effective technology for creating a compelling, immersive environment to supplement existing curatorial narratives told with text and illustrations. As Robert notes, “…we’re not replacing the authentic object, we’re augmenting it.”

Costello is an expert at crafting educational museum experiences using a variety of media. The long list of his previous projects includes a mobile app called MEanderthal; a website for Human Origins; a web comic called Secret in the Cellar; and he leads on cyberinfrastructure development for eMammal, a large-scale citizen science project focused on understanding the effects of human caused rapid environmental change on mammal populations.

Skin and Bones Augmented RealityMobile APPWith Skin & Bones, visitors can experience the richness of the hall using a mobile device. They can ponder big ideas in evolution and ecology, meet the people, living and dead, who contribute to the legacy of the Smithsonian, see the objects on display in unexpected ways, and become familiar with the life of animals. Costello hopes that balancing streaming video “…which takes you away from where you are,” with the augmented reality part of the experience, which “…focuses you on where you are,” will create a memorable experience of the hall.

Though Costello says predicting the trajectory of any technology is difficult, he feels that technologies for bringing 3D to the museum are getting better all the time, and will be increasingly part of the way museums present compelling narratives. Museums such as the Royal Ontario Museum in Toronto have already begun to incorporate AR technologies with exhibits like “Ultimate Dinosaurs.” When the NMNH’s exhibit launches, it will offer visitors one of the largest and most complex museum experiences of 3D AR.

While some may think that the benefits of using 3D and animation to communicate scientific ideas are inherently obvious, there are few rigorous studies identifying the specific contributions of 3D to learning. Costello is seeking to craft more meaningful visitor experiences building on a research-based understanding of human learning styles. Diana Marques, who designed the Skin & Bones user interface, is also leading the research on the visitor experience with mobile.

Marques is using the latest data-gathering software to track visitor choices and she is using an identical mobile app that does not use AR to isolate this variable in the visitor experience. Costello explains that, “the conceptual framework we use for content is borrowed from Andrew Pekarik’s IPOP model, based on years of research on museum audiences and the preferences people have for engaging with the exhibits. …There are “Idea people”, “People people”, “Object people” and “Physical people”. We added one more category to that conceptual framework, “Animal people”… augmented reality supports the object category of user preferences under the IPOP framework.” The mobile app will have content that appeals to people within each of the preference categories. Robert and Diana plan to test the content as it is rolled out to determine if and how AR is adding value to the visitor experience.

Robert CostelloRobert has already determined that mobile and AR are cost-effective technologies that can add significant value to existing exhibits. The costs and logistical problems of redoing older exhibits are prohibitive, or time-delaying, and as he notes, “The osteology hall has over 300 skeletons that are mounted. Where do you store them? How do you move them? How do you replace them? It’s enormously expensive. If we can successfully apply technology to reinvent the visitor experience, and make it a much more engaging, enjoyable, memorable experience, then technology has served the visitor well. And we can also preserve something from the past.”