Transfer Learning For Cell Image Reconstruction Andrei Raureanu University of Twente P.O. Box 217, 7500AE Enschede The Netherlands [email protected] ABSTRACT results in some scan patches being blurry due to incorrect Whole slide scanners are a useful tool for doing cell anal- focus of the lens. This also happens even if the microscope ysis in an efficient manner. A big problem with this is has an automated focus system. The blurry areas in the that scanning is not always perfect, which results in arti- scan then need to be inspected manually which makes it facts such as blur in some parts of the scan. This causes tedious and time consuming. further problems for the specialists using the scanner, as Various algorithms and systems have been proposed to they have to manually inspect the blurry areas in ques- deal with this issue [9, 15, 12]. There are deep learning tion and give an objective conclusion. To solve this issue, solutions that detect blurry areas and segment them [11] two Convolutional Autoencoders models are designed and which is relevant as those areas are the most in need of de- implemented to reconstruct cell slide images. The per- blur. This allows the microscope to re-scan just the blurry formance of the models to remove blur from sections of area specifically without doing a full scan. Unfortunately cell slides will then be investigated. The robustness of even rescanning a small section requires heavy time-loads. the Autoencoders is also tested on cell images generated In recent years, Autoencoders have gained a lot of pop- artificially that have had Gaussian blur applied to them. ularity. This is due to their ability to reconstruct input Both trained models successfully deblur cell images with containing artifacts into its clear counterpart. Because of minor performance decreases when the blur is caused by this, the use of Autoencoders looks to be very suitable for the camera lens focusing below the focal plane. The re- the scope of this project. Another advantage of Autoen- constructions of synthetic cells is also achievable with only coders is not needing labeled data for training which also a 15% performance decrease when deblurring cell images makes it ideal for unsupervised learning. It is also im- with high amounts of Gaussian blur applied to them. portant to note that sometimes getting big quantities of labeled data for a specific model training can be difficult. Keywords In this research, two convolutional Autoencoder are trained denoising, Autoencoder, neural network, cell imaging, ma- using two different methods to reconstruct blurry cell im- chine learning, image reconstruction ages into its non-blurry counterpart. The focus will be made on out-of-focus blur anomalies as these are the most 1. INTRODUCTION common problem in cell imaging. In the biomedical industry, whole slide scanners are one of the many tools to analyse cells for research purposes or 2. PROBLEM STATEMENT disease diagnosis. The underlying process of scanners, cell Research has been done on ways to detect areas with imaging, consists of staining the cell dish with a fluores- blurry anomalies in cell images for re-scanning purposes [17]. cent substance, taking pictures of different areas using a Some examples even use deep learning for this [11]. De- microscope with high magnification (20x - 40x) and then spite that, there have been very few attempts at "fixing" patching all of them together to create one high-resolution the out-of-focus image directly. This paper presents two scan. The scan image can then be further used for analysis systems that are trained to reconstruct out-of-focus cell of the cells such as classification or counting. images, thus de-blurring them. The two systems are also able to reconstruct images captured by other scanners However, this technique cannot deliver high-quality im- other than the one they were trained on. ages all the time. It is estimated that 5% of the scans present artifacts [14]. One of the most common problems 2.1 Research Questions encountered in cell imaging is out-of-focus blur [5]. This is In retrospect of the problem statement, the following re- caused by the fact that cells are not on the same level of the search questions are presented: Z-axis on the chamber slide, meaning that individual cells can be higher or lower from each other on the dish. This • RQ1: To what extent can an Autoencoder recon- struct out-of-focus cell images? Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies • RQ2: How robust is the Autoencoder for out-of- are not made or distributed for profit or commercial advantage and that focus reconstruction when presented with cell images copies bear this notice and the full citation on the first page. To copy oth- erwise, or republish, to post on servers or to redistribute to lists, requires recorded with a different microscope? prior specific permission and/or a fee. th nd • RQ2.1: To what extent can it reconstruct out-of- 35 Twente Student Conference on IT July 2 , 2021, Enschede, The Netherlands. focus cell images recorded with a different micro- Copyright 2021, University of Twente, Faculty of Electrical Engineer- scope? ing, Mathematics and Computer Science. 1 Figure 1. An example of a pair of two cell images resulted from Hoechst staining. a) shows an in-focus cell image while b) shows the same cell slide but this time out-of-focus due to incorrect microscope calibration. 3. RELATED WORK the blur from an image by sharpening it. For this they use There has been substantial research done in the detection convolutional layers inside the Autoencoder to downsam- of out-of-focus images in whole slide scanning. Senaras et ple the training data, extract the relevant features from it al. [11] presented a deep learning solution to detect and and then convert the image to a sharper version of itself. segment blurry areas in digital whole slide images under From what has been gathered so far, there is a solid foun- the form of a convolutional neural network. In another dation of artifact recognition and classification along with paper, Yang et al. [17] argued that relative differences be- previous use cases of Autoencoders in image related tasks, tween two or more images in general lead to inconsisten- but not enough work on using Autoencoders on cell im- cies, which is why they created a deep neural network to ages specifically for deblurring. In this paper two different predict an absolute measure of quality based on level of Autoencoders are proposed for dealing with this problem. blur. What is different from Senaras' [11] model is that their training data was synthetically defocused to 11 ab- solute defocus levels. 4. METHODS For this research, two Autoencoder architectures have been Besides deep learning solutions, there have also been algo- designed and implemented. They will be compared be- rithms and systems proposed for detection. One such ex- tween each other using the metrics mentioned to see which ample of these is the system designed by Zerbe et al. [18]. approach works best. The first Autoencoder model is They make use of the image analysis software ImageJ and trained using the whole image from the training set, mean- Tenenbaum gradient to assess and classify image patch ing no segmentation is used. The second Autoencoder sharpness based on the degree of blur present. The full however, is trained on a data set derived from the original microscope image is split into image patches. Then, us- Human U2OS cell data set, each image being segmented ing batch processing through the use of multiple comput- into 36 patches that correspond with an area of the origi- ing nodes, each image patch is classified. Finally, all the nal image. Each image patch is then used as training data patches are aggregated in a sharpness map. for the second Autoencoder. In a similar fashion, Shakhawat et al. [12] propose a method Moving forward into the paper, the first Autoencoder shall to find the origin of the artifact, meaning whether the be referred to as "whole image Autoencoder" and the sec- distortion happened during the slide preparation stage or ond Autoencoder design as "split image Autoencoder". scanning, but with the help of a support vector machine The diagrams for both Autoencoders can be found in Ap- to grade the quality of the region containing artifacts from pendix C1 and C2 respectively. the whole cell image scan. There are records of Autoencoders being used for image 4.1 Whole image Autoencoder processing. Gupta et al. [6] perform motion blur removal The whole image Autoencoder starts with an Input layer using Coupled Autoencoders. They use two Autoencoders, of 128x128x1, which corresponds to the image shape of the the first one being the source and the second being the tar- training set. Afterwards, two pairs of Conv2D and Max- get. The first one makes use of a blurry image, which is Pooling2D layers are added, with a BatchNormalization formed out of the clear image with a blur kernel applied to layer inbetween them. The filter size of the first Conv2D it, and the second Autoencoder uses the clear image sam- layer is 32 with kernel size of (3, 3) while the second ple. The coupling then learns the mapping of the blurry Conv2D layer has a filter size of 64 also with a kernel image to the clear representation. This is done by hav- size of (3, 3). Both Conv2D layers have the same ReLu ing the first Autoencoder learn the latent representation activation function with padding "same".