New super resolution method reveals finer details without constantly zooming in

Since the early 1930s, electron microscopy has provided an unusual access to the foreign world of the extraordinarily small, publicly intricate details that would otherwise be indistinguishable from conventional light microscopy. But in order to achieve high resolution over a large specimen area, the energy of the electron beams must creep up, which is costly and harmful to the specimen under observation.

Researchers at Texas A&M University may have found a new method to improve the quality of low-resolution electron micrographs without compromising the integrity of exam samples. By training deep neural networks, a kind of artificial intelligence algorithm, on pairs of images from the same sample, but at different physical resolutions, they have found that details in lower resolution images can be further improved.

“Normally, a high-energy electron beam is passed through the sample at locations where larger image resolution is desired. But with our image processing techniques, we can super-resolution an entire image using just a few smaller, high-resolution images,” said Dr. Yu Ding, Mike and Sugar Barnes Professor in the Wm Michael Barnes ’64 Department of Industrial and Systems Engineering. “This method is less destructive because most parts of the specimen do not have to be scanned with high-energy electron beams.”

The researchers published their technique for image processing in Institute of Electrical and Electronics Engineering Transaction Processing Institute in June.

Unlike light microscopy where photons, like small packets of light, are used to illuminate an object, electron microscopy uses a beam of electrons. The electrons reflected from or passing through the object are then collected to form an image called the electron micrograph.

Thus, the energy of the electron beams plays a crucial role in determining the resolution of images. That is, the higher the energy electrons, the better the resolution. However, the risk of damaging the specimen also increases, similar to how ultraviolet rays, which are the more energetic siblings of visible light, can damage sensitive materials such as the skin.

“There’s always that dilemma for scientists,” Ding said. “To maintain the integrity of the specimen, high-energy electron beams are used sparingly. But if one does not use energy beams, high resolution as the ability to look at finer scales is limited.”

But there are ways to get high resolution or super resolution with low resolution images. One method involves using multiple low-resolution images of essentially the same region. Another method learns common patterns between small image patches and uses unrelated, high-resolution images to enhance existing low-resolution images.

These methods use almost exclusively natural light images instead of electron micrographs. Because of this, they have problems for super-resolution electron micrographs because the underlying physics for light and electron microscopy is different, Ding explained.

The researchers turned to pairs with low and high resolution electron microscopic images for a given sample. Although these types of pairs are not very common in public image databases, they are relatively common in materials science research and medical imaging.

For their experiments, Ding and his team first took a low-resolution image of a specimen and then subjected roughly 25% of the area under observation to high-energy electron beams to obtain a high-resolution image. The researchers noted that the information in the high-resolution and low-resolution pair is very tightly correlated. They said this property could be used, although the available dataset may be small.

For their analyzes, Ding and his team used 22 pairs of images of materials infused with nanoparticles. They divided the high-resolution image and the equivalent area in the low-resolution image into three by three subimages. Next, each subimage pair was used to “self-train” deep neural networks. After training, her algorithm became familiar with recognizing image functions, such as edges.

When they tested the trained deep neural network at a new location on the low-resolution image, where there was no high-resolution counterpart, they found that their algorithm could improve features that were as low as 50% were distinguishable.

Although its image processing technique shows a lot of promise, Ding notes that it still requires a lot of computing power. In the near future, his team will focus their efforts on developing algorithms that are much faster and can be supported by inferior computer hardware.

“Our shared image processing technology opens up details in low-resolution images that were previously unrecognized,” Ding said. “We are all familiar with the function of the magic wand on our smartphones. It makes the image clearer. What we intend to do in the long run is to provide the research community with a similarly handy tool for improving electron micrographs.”