The Virtual Factory

Virtual and mixed-reality in an industrial setting

We are investigating virtual and mixed-reality systems for collaboration, particularly in industrial settings.

This research project focuses on virtual and mixed-reality systems for collaboration, particularly in industrial settings. In collaboration with TCHO, a chocolate maker with a factory in San Francisco,* we built virtual mirror world representations of a real-world chocolate factory and its processes, and importing real-time sensor data into the resulting virtual factory. This research is also an exploration into an area sometimes called “Serious Games”  which combines techiniques from gaming and high-end game engines with robust multimedia and database imports for real-world purposes. We run several parallel projects in our exploration of mixed reality applications for manufacturing:

Virtual Factory 3D world:

This 3D/mixed reality environment is designed for simulation, visualization, and collaboration, using a set of interlinked, real-time 3D and 2D layers of information about the TCHO chocolate factory and its processes. In the Virtual Factory environment, users can view real-time high-definition video streaming from a number of camera viewpoints in the factory, click on a machine’s “DataSpot” to read its sensor status, or move closer to a machine to trigger an in-world video overview of its function. The virtual environment imports real-time data from a network of sensors on the factory floor and displays it when appropriate in the virtual world. The system can track processes in detail over extended time periods and play them back on request. The mixed reality system is also multi-user, allowing remote collaboration and communication for collaborative tasks such as factory observation, virtual inspections, customer visits, employee education and training, process monitoring, and inventory tracking. The Virtual Factory is now deployed on a large display at the real-world TCHO factory, where it is in daily use. FXPAL sees this mashup of real and virtual factories as a way to enhance collaboration and communication between physical-plant operators and engineers and remote managers; for example, factories in China with managers in Japan. It also creates efficiencies for local operators.

Mobile Mixed Reality: an iPhone/iPad app for laboratory monitoring and control

Integrating mobile with virtual and mixed reality systems is a logical extension for industrial smart environments. With this in mind, we have built an experimental app for iPhone and iPad that provides mobile laboratory monitoring and control. In the TCHO recipe development lab, accurate tracking of time and temperature are essential. The mobile application gives the lab users a real-time view into this lab (via PTZ or Pan-Tilt-Zoom steerable camera) and imports real-world sensor data as well. Using the mobile app on either iPhone or iPad (or Android devices), machines can be turned on/off, temperatures and timer settings can be read and changed, and lab environmental controls can be changed. The mobile system is integrated with both the multi-camera surveillance system and the database underlying the virtual factory worlds. It has been in daily use at TCHO since 2009. Our Mobile Virtual Factory shows the Virtual Factory’s 3D environment on iPhone and Android mobiles, along with real-time data from the hundreds of sensors on the factory floor.

*In 2014. the TCHO factory moved to Oakland, CA.

Technical Contact

Related Publications

2012
Publication Details
  • DIS (Designing Interactive Systems) 2012 Demos track
  • Jun 11, 2012

Abstract

Close
We will demonstrate successive and final stages in the iterative design of a complex mixed reality system in a real-world factory setting. In collaboration with TCHO, a chocolate maker in San Francisco, we built a virtual “mirror” world of a real-world chocolate factory and its processes. Sensor data is imported into the multi-user 3D environment from hundreds of sensors and a number of cameras on the factory floor. The resulting virtual factory is used for simulation, visualization, and collaboration, using a set of interlinked, real-time layers of information. It can be a stand-alone or a web-based application, and also works on iOS and Android cell phones and tablet computers. A unique aspect of our system is that it is designed to enable the incorporation of lightweight social media-style interactions with co-workers along with factory data. Through this mixture of mobile, social, mixed and virtual technologies, we hope to create systems for enhanced collaboration in industrial settings between physically remote people and places, such as factories in China with managers in the US.
2010

The Virtual Chocolate Factory:Mixed Reality Industrial Collaboration and Control

Publication Details
  • ACM Multimedia 2010 - Industrial Exhibits
  • Oct 25, 2010

Abstract

Close
We will exhibit several aspects of a complex mixed reality system that we have built and deployed in a real-world factory setting. In our system, virtual worlds, augmented realities, and mobile applications are all fed from the same infrastructure. In collaboration with TCHO, a chocolate maker in San Francisco, we built a virtual “mirror” world of a real-world chocolate factory and its processes. Sensor data is imported into the multi-user 3D environment from hundreds of sensors on the factory floor. The resulting virtual factory is used for simulation, visualization, and collaboration, using a set of interlinked, real-time layers of information. Another part of our infrastructure is designed to support appropriate industrial uses for mobile devices such as cell phones and tablet computers. We deployed this system at the real-world factory in 2009, and it is now is daily use there. By simultaneously developing mobile, virtual, and web-based display and collaboration environments, we aimed to create an infrastructure that did not skew toward one type of application but that could serve many at once, interchangeably. Through this mixture of mobile, social, mixed and virtual technologies, we hope to create systems for enhanced collaboration in industrial settings between physically remote people and places, such as factories in China with managers in the US.
Publication Details
  • ICME 2010, Singapore, July 19-23 2010
  • Jul 19, 2010

Abstract

Close
Virtual, mobile, and mixed reality systems have diverse uses for data visualization and remote collaboration in industrial settings, especially factories. We report our experiences in designing complex mixed-reality collaboration, control, and display systems for a real-world factory, for delivering real-time factory information to multiple users. In collaboration with (blank for review), a chocolate maker in San Francisco, our research group is building a virtual “mirror” world of a real-world chocolate factory and its processes. Real-world sensor data (such as temperature and machine state) is imported into the 3D environment from hundreds of sensors on the factory floor. Multi-camera imagery from the factory is also available in the multi-user 3D factory environment. The resulting "virtual factory" is designed for simulation, visualization, and collaboration, using a set of interlinked, real-time 3D and 2D layers of information about the factory and its processes. We are also looking at appropriate industrial uses for mobile devices such as cell phones and tablet computers, and how they intersect with virtual worlds and mixed realities. For example, an experimental iPhone web app provides mobile laboratory monitoring and control. The app allows a real-time view into the lab via steerable camera and remote control of lab machines. The mobile system is integrated with the database underlying the virtual factory world. These systems were deployed at the real-world factory and lab in 2009, and are now in beta development. Through this mashup of mobile, social, mixed and virtual technologies, we hope to create industrial systems for enhanced collaboration between physically remote people and places – for example, factories in China with managers in Japan or the US.
Publication Details
  • IEEE Virtual Reality 2010 conference
  • Mar 19, 2010

Abstract

Close
This project investigates practical uses of virtual, mobile, and mixed reality systems in industrial settings, in particular control and collaboration applications for factories. In collaboration with TCHO, a chocolate maker start-up in San Francisco, we have built virtual mirror-world representations of a real-world chocolate factory and are importing its data and modeling its processes. The system integrates mobile devices such as cell phones and tablet computers. The resulting "virtual factory" is a cross-reality environment designed for simulation, visualization, and collaboration, using a set of interlinked, real-time 3D and 2D layers of information about the factory and its processes.
2009
Publication Details
  • Presentation at SIGGRAPH 2009, New Orleans, LA. ACM.
  • Aug 3, 2009

Abstract

Close
FXPAL, a research lab in Silicon Valley, and TCHO, a chocolate manufacturer in San Francisco, have been collaborating on exploring emerging technologies for industry. The two companies seek ways to bring people closer to the products they consume, clarifying end-to-end production processes with technologies like sensor networks for fine-grained monitoring and control, mobile process control, and real/virtual mashups using virtual and augmented realities. This work lies within and extends the area of research called mixed- or cross-reality

Mirror World Chocolate Factory

Publication Details
  • IEEE Pervasive Computing July-August 2009 (Journal, Works in Progress section)
  • Jul 18, 2009

Abstract

Close
FXPAL, a research lab in Silicon Valley, and TCHO, a chocolate manufacturer in San Francisco, have been collaborating on exploring emerging technologies for industry. The two companies seek ways to bring people closer to the products they consume, clarifying end-to-end production processes with technologies like sensor networks for fine-grained monitoring and control, mobile process control, and real/virtual mashups using virtual and augmented realities.