HoloGraphics: combining holograms with interactive computer graphics

bimber-photoOliver Bimber
bimber@ieee.org

About the author
Oliver Bimber is a Junior Professor for Augmented Reality at the Bauhaus University Weimar in Germany. He holds a PhD in Computer Science from Technical University of Darmstadt. Contact him at bimber@ieee.org. More information is available at http://www.HoloGraphics.de

Just like computer graphics, holograms are being applied as tools to solve individual research, engineering, and presentation problems within several domains. Up until today, however, these tools have been applied separately. The overall goal of our project is to combine both technologies to create a powerful tool for science, industry and education. We are currently investigating the possibility of integrating computer generated graphics and holograms.

Our goal is to combine the advantages of conventional holograms (i.e. extremely high visual quality and realism, support for all depth queues and for multiple observers at no computational cost, space efficiency, etc.) with the advantages of today’s computer graphics capabilities (i.e. interactivity, real-time rendering, simulation and animation, stereoscopic and autostereoscopic presentation, etc.).

Several engineering and computer science topics will be addressed throughout the project: The potentials of different hologram types with respect to the project’s goal have to be investigated. New three-dimensional displays that combine computer graphics and holography will be engineered. New real-time rendering algorithms, registration methods, and human–computer interaction techniques that are adequate for the proposed metaphor will be developed.

The outcome will be a three-dimensional display concept whose application is envisioned in areas such as scientific visualization (e.g., paleontology, pathology, density, medicine, biomedicine, orthopedics or archeology), industrial simulation (e.g., design, manufacturing and quality assurance), and education (e.g., medical training or public museums).

Here are some of our initial results.

Using digital light to reconstruct the holographic image

The two basic hologram types—transmission and reflection—are both reconstructed by illuminating them with spatially coherent light (i.e. using a point-source of light). These two types have generated a number of variations. Although some holograms can be reconstructed only with laser light, others can be viewed under white light.

Rainbow holograms, one of the most popular types of white-light transmission hologram, diffract each wavelength of the light through a different angle. This lets viewers observe the recorded scene from different horizontal viewing positions but also makes the scene appear in different colors when observed from different vertical points of view. In contrast to rainbow holograms, white-light reflection holograms can provide full parallax and display the recorded scene in a consistent color (monochrome or multi-color) for different viewing positions.

Conventional video projectors represent point sources that are well suited for viewing white-light reflection or transmission holograms. Today’s high-intensity discharge lamps of projectors can produce a very bright light. The main advantage for using video projectors is that the reference wave used to reconstruct the hologram can be digitized. Thus it is possible to control the amplitude and wavelength of each discrete portion of the wavefront over time.

Figure 1: The projected reference waves and the resulting holographic images.

Figure 1: The projected reference waves and the resulting holographic images.

Figure 1 shows the projected reference wave in different states, and the resulting holographic image of a monochrome white-light reflection hologram. A uniform reference wave reconstructs the entire hologram uniformly. Selectively emitting light in different directions allows us to create an incomplete reference wave that reconstructs the hologram only partially. Local amplitude variations in the reference wave result in proportional amplitude variations in the reconstructed object wave. Variations in wavelength do not lead to useful effects in most cases due to the wavelength dependency of holograms. But this is still a matter for further investigations.

Partially reconstructing object waves

It is possible to reconstruct the object wave of a hologram only partially, leaving gaps where graphical elements can be inserted.

Both reflection holograms (without an opaque backing layer) and transmission holograms remain transparent if not illuminated. Thus, they can serve as optical combiners—leading to very compact displays.

Real-time computer graphics can be integrated into the hologram from one side, while illuminating it partially from the other side [1]. Thereby, rendering and illumination are view-dependent and have to be synchronized.

If autostereoscopic displays are used to render 3D graphics registered to the hologram, both holographic and graphical content appear three-dimensional within the same space. If depth information of both is known, correct occlusion effects between hologram and graphics can be generated.

Figure 2: Rainbow  hologram  of  a  dinosaur  skull  combined  with  graphical representations of soft tissue and bones.

Figure 2: Rainbow hologram of a dinosaur skull combined with graphical representations of soft tissue and bones.

Figure 2 shows a rainbow hologram of a dinosaur skull combined with graphical representations of soft tissue and bones. If the holographic plate is illuminated with a uniform light, the entire hologram is reconstructed. If the plate is illuminated only at the portions not occluded by graphical elements, the synthetic objects can be integrated by displaying them on the screen behind the plate.

Light interaction

The reconstructed object wave’s amplitude is proportional to the reference wave’s intensity. In addition to using an incomplete reference wave for reconstructing a fraction of the hologram, intensity variations of the projected light permit local modification of the recorded object wave’s amplitude.

Practically, this means that to create the illumination image which is sent out by the projector, graphical shading and shadowing techniques are used to reconstruct the hologram instead of illuminating it with a uniform intensity. To do this, the real shading effects on the captured scenery caused by the real light sources used for illumination during hologram recording, as well as the physical lighting effects caused by the video projector on the holographic plate, must both be neutralized. Next, the influence of a synthetic illumination must be simulated [1].

Using conventional graphics hardware, it becomes possible not only to create consistent shading effects, but also to cast synthetic shadows correctly from all holographic and graphical elements onto all other elements.

Figure 3: A  rainbow  hologram  with  3D  graphical  elements  and  synthetic shading and shadow effects.

Figure 3: A rainbow hologram with 3D graphical elements and synthetic shading and shadow effects.

The figures show the same rainbow hologram as above with 3D graphical elements and synthetic shading effects. Shadows are cast correctly from the hologram onto the graphics and vice versa. A virtual point-source of light was first located at the top-left corner, and then moved to the top-right corner, in front of the display.

Proof-of-concept prototypes

The desktop prototypes shown in figure 4 consist entirely of off-the-shelf components, including either an autostereoscopic lenticular-lens sheet display with an integrated head-finder for wireless user tracking, or a conventional CRT screen with active stereo glasses, wireless infrared tracking, and a touch screen for interaction.

Both prototypes use digital light projectors. A single PC with a dual-output graphics card renders the graphical content on the screen and illuminates the holographic plate on the video projector.

Figure 4: An autostereoscopic (left) and a stereoscopic (right) proof-of-concept prototype.

Figure 4: An autostereoscopic (left) and a stereoscopic (right) proof-of-concept prototype.

In both cases, the screen additionally holds further front layers—glass protection, holographic emulsion, and optional mirror beam splitter (used for transmission holograms only).

Interaction with the graphical content is supported with a mouse or a transparent touch-screen mounted in front of the holographic plate.

Experiments with a digital multiplex hologram

Digital holography uses holographic printers to expose the photosensitive emulsion with computer generated or captured images.

This results in conventional holograms with digital content rather than real scenery. Pre-processed 2D and 3D graphics or digital photographs and movies can be printed. This allows, for instance, the holographic recording of completely synthetic objects, real outdoor scenes, and objects in motion—which is difficult and sometimes impossible to achieve with optical holography.

Like optical holograms, digital holograms can be multiplexed. This allows us to divide the viewing space and to assign individual portions to different contents.

The content for digital holograms can easily be created by non-experts, and the printing process is inexpensive. Usually a 3D graphical scene, a series of digital photographs or a short movie of a real object is sufficient for producing digital holograms. However, these digital holograms lack in the quality (resolution, color appearance, sharpness, etc.) of conventional optical holograms.

Figure 5: A multiplexed digital reflection hologram of a car headlight with integrated CAD data.

Figure 5: A multiplexed digital reflection hologram of a car headlight with integrated CAD data.

Figure 5 shows a digital color white-light reflection hologram of a car headlight. It was generated by taking 360 perspective photographs from different angles (in 0.5 steps to cover a 110 total viewing zone plus two 35 clipping areas). The perspective photographs were multiplexed into different sub-zones (40 = 80 images for the front view + 2 × 35 = 140 images for the side and rear views +  2 × 12.5 = 50 images to fill the partially visible clipping area outside the 110 total viewing zone + 2 × 22.5 = 90 images to fill the invisible clipping area outside the 110 total viewing zone).

Consequently, three different partial views (front, rear, and side) can be observed by moving within the total viewing zone of 110. After registering the holographic plane and calibrating the projector, interactive graphical elements, such as wire-frame or shaded CAD data can be integrated into the hologram.

Holographic windows

The ability to control the reconstruction of a hologram’s object wave allows integrating them seamlessly into common desktop-window environments.

If the holographic emulsion that is mounted in front of a screen is not illuminated, it remains transparent. In this case the entire screen content is visible and an interaction with software applications on the desktop is possible in a familiar way.

The holographic content (visible or not) is always located at a fixed spatial position within the screen/desktop reference frame. An application that renders the graphical content does not necessarily need to be displayed in full-screen mode (as in the examples above), but can run in a ‘windows’ mode—covering an arbitrary area on the desktop behind the emulsion.

If the position and the dimensions of the graphics window are known, the projector-based illumination can be synchronized to bind the reference wave to the portion of the emulsion that is located directly on top of the underlying window. Thereby, all the techniques that are described above (partial reconstruction and intensity variations) are constrained to the window’s boundaries. The remaining portion of the desktop is not influenced by the illumination, the graphical or the holographic content.

In addition, the graphical content can be rendered in such a way that it remains registered with the holographic content—even if the graphical window is moved or resized.

This simple, but effective technique allows a seamless integration of holograms into common desktop environments. It allows us to temporarily minimize the “holographic window” or to align it over the main focus while working other applications.

Figure 6: A holographic window in different states on a desktop together with other applications.

Figure 6: A holographic window in different states on a desktop together with other applications.

Figure 6 shows a holographic window in different states on a desktop together with other applications. It displays an optical (monochrome) white-light reflection hologram of a dinosaur skull with integrated graphical 3D soft tissues. A stereoscopic screen was used in this case, because autostereoscopic displays (such as lenticular screens or barrier displays) do not yet allow an undisturbed view on a non-interlaced 2D content.

Outlook

Holograms can store a massive amount of information on a thin holographic emulsion. This technology can record and reconstruct a 3D scene with almost no loss in quality. Moore’s law—which asserts that computing power doubles every 18 months—must be applied many times for graphical or electro-holographic rendering techniques and displays in order to reach this quality at interactive frame rates. A combination of interactive computer graphics and high-quality holograms represents an alternative that can be realized today with off-the-shelf consumer hardware. We believe that this concept can be beneficial for many applications.

Archaeologists, for example, already use holograms to archive and investigate ancient artifacts. Scientists can use hologram copies to perform their research without having access to the original artifacts or settling for inaccurate replicas. They can combine these holograms with interactive computer graphics to integrate real-time simulation data or perform experiments that require direct user interaction, such as packing reconstructed soft tissue into a fossilized dinosaur skull hologram. In addition, specialized interaction devices can simulate haptic feedback of holographic and graphical content while scientists are performing these interactive tasks. An entire collection of artifacts will fit into a single album of holographic recordings, while a light-box-like display such as that used for viewing x-rays can be used for visualization and interaction.

The same applies to the biomedical domain that already uses digital volumetric holograms produced from CT or MRI data of inner organs.

In the automotive industry, for instance, complex computer models of cars and components often lack realism or interactivity. Instead of attempting to achieve high visual quality and interactive frame rates for the entire model, designers could decompose the model into sets of interactive and static elements. The system could record physical counterparts of static elements in a hologram with maximum realism, and release computational resources to render the interactive elements with a higher quality and increased frame rate. Multiplexing the holographic content also lets users observe and interact with the entire model from multiple perspectives. Beside display holograms, holographic interferograms used for non-destructive measurement and testing are yet another example of industrial applications. Analogue interferograms that indicate motion, vibration, or deformations of objects can be combined with digital simulation data.

Augmenting holograms in museums with animated multimedia content lets exhibitors communicate information about the artifact with more excitement and effectiveness than text labels offer. Such displays can also respond to user interaction. Because wall-mounted variations require little space, museums can display a larger number of artifacts.

Figure 7: Illustrations of envisioned future applications: Museum displays, and scientific visualization and simulation.

Figure 7: Illustrations of envisioned future applications: Museum displays, and scientific visualization and simulation.

Figure 7 shows two illustrations of envisioned future applications: A wall-mounted display in a museum environment, with a ceiling-mounted video projector replacing conventional spotlights, and a desktop display that can be used in a light-box fashion. A special input device allowing interaction, including haptic feedback of holographic and graphical content.

The technical and scientific progress that is planned to be made during this project can be organized into six linked goals:

Holograms
Investigation of different hologram types and individual solutions that combine them with computer graphics. Optimization of optical properties during the recording process to achieve the best possible effects.
Displays
Experiments with different optical setups that support the integration of computer graphics into the different hologram types. Experiments with different stereoscopic and autostereoscopic techniques. Investigation of different display form factors that serve a variety of applications.
Calibration and registration
Development of fully- or semi-automated methods that calibrate the display optics, extract auxiliary information (such as depth) recorded in the hologram and register holographic and graphical content.
Rendering
Development of effective rendering and illumination algorithms that support different hologram types, special effects, and a realistic and consistent presentation of holographic and graphical content at interactive frame rates.
Interaction
Investigation of the potentials and limitations of existing interaction techniques and devices in combination with interactive holograms. Development of new interaction forms that are suited for the different display approaches and potential application areas.
Demonstration and evaluation
Implementation of demonstrators that address different application areas. The effectiveness of the proposed concept is evaluated by presenting the demonstrators to domain experts.

Acknowledgements

This project is supported by the Deutsche Forschungsgemeinschaft (DFG). The experiments shown in figure 5 were supported by DaimlerChrysler AG Research Technology. The content shown in figures 2, 3 and 6 was provided by Ohio University’s department of Biomedical Sciences. The HoloGraphics project is supported by the Deutsche Forschungsgemeinschaft (DFG).

References

[1]  O. Bimber January 2004 Combining holograms with interactive computer graphics IEEE Computer 85–91

0 Response to “HoloGraphics: combining holograms with interactive computer graphics”


  • No Comments

Leave a Reply

*