admin

AI triumphs and revolutions fill the daily news; we are told that AI can now detect cancer, find lifesaving treatments, drive cars, and much more. So why do only a small proportion of organisations successfully implement AI assisted solutions [1]?

Behind the flashy headlines lies an uncomfortable truth: to build AI models that can generate useful insights, one needs data that has been painstakingly labelled by humans. For example, individuals need to annotate images before they can teach an AI model what is visible in those images (e.g., how many cars are in the image and where are the cars located?) or have access to a big database of clinically validated drugs before teaching an AI how to predict new assets. This strategy worked well for applications such as autonomous driving where labelled data is cheap and easy to come by (e.g., dashboard camera images) – after all, non-experts can correctly identify common objects in images. However, when dealing with expert domains such as the Life Sciences sector, labelling data quickly becomes difficult. For example, labelling a single pathology image to teach an AI to diagnose cancer takes an expert pathologist an average of 2 hours – a cost of $280/image [2][3]. The first AI powered FDA approved prostate cancer detection tool required 12,160 of such labelled images [4]. Hence, labelling enough data to bring an AI tool into the clinic costs approximately $6M. Unfortunately, labelling a single dataset is not enough due to Model/Data drift [5], i.e., the decrease of AI model performance due to changes in data over time. For example, different imaging devices for pathology images could yield data that the original model may not be able to analyse correctly anymore. Hence models must be retrained from time to time, increasing costs dramatically.

Cases in which collecting and labelling enough data is not only expensive but almost impossible are even more challenging. Consider building an AI model that detects defects in assembly lines. Industry experts estimate that there might be as few as 40 images containing defects in a dataset of millions of images, so that no company may ever collect enough data to deploy AI [6]. Amongst the most difficult cases are those in which data collection and labelling requires complex and lengthy specialised human tasks. In gene-editing experiments, for example, it may take a month to generate a single labelled datapoint [7], e.g., measure the effect of gene deletion on a disease model such as human derived stem cells. When trying to predict the clinical outcomes of drugs, we may never have access to enough data for AI: only a few drugs are eventually tested in humans after several years of development time [8], and unfortunately most of these treatments fail [9].

As of today, we have only scratched the surface of what is possible with AI. Despite our collective mission to optimise AI on Big Data, our models often fail in the real world when using small datasets. Additionally, Big Data struggles with the “Big Data Paradox”, i.e., inherent dataset biases are amplified with bigger datasets [11]. Leading industry experts estimate that for every Big Data application there are thousands of Small Data applications that currently remain unsolved (Figure 1) [12]. To truly solve the big challenges ahead, AI leaders estimate that we need to leverage small datasets and enable tens of thousands of lightweight Small Data AI models for niche applications [12].

Figure 1 –To date, AI has been successful in Big Data applications. However, Big Data is just the tip of the iceberg and there is a massive unmet need of AI in Small Data applications

How may AI work for Small Data? Carefully examining nature may help: toddlers do not need to see tens of thousands of animals and be told by their parents which ones are cats. Often a few examples suffice. Note that that true Small Data Problems are mostly impossible to solve: no toddler can learn to identify cats from only one labelled example of a cat. However, by the time their parents points at a cat and says “cat” the toddler has seen a lot of the world. They already have a general understanding of objects and shapes in the world that they can leverage to learn what a cat is. In other words: prior to encountering the Small Data problem of naming cats the toddler has already processed and potentially categorised massive amounts of raw unlabelled data about the world around them. 

Small Data AI techniques emulate nature. Organisations typically have abundant unlabelled data but struggle to generate labelled data. For example, taking an x-ray of a patient is faster than having a radiologist label it in detail, and it is often cheap and easy to access raw, anonymised, unstructured imaging datasets from hospitals but expensive to hire medical experts to label them. Similarly, generating a library of potential drug compounds to test against a given disease is cheap, running trials to test them is not. Owing to this, Small Data AI algorithms typically enable learning from big unlabelled or partially labelled datasets (Unsupervised/Semi-supervised Learning), or use AI models pretrained on big, labelled datasets and then refined on a related Small Data problem (Transfer Learning, Synthetic Data). See Figure 2 & Table 1 for an overview of the different techniques with examples.

Figure 2 – Venn Diagram comparing different Small Data AI techniques in the context of supervised/unsupervised learning.

Technique	Description	Strengths / Weaknesses
Synthetic Data	Synthetic labelled data is generated with generative models that were trained on a related dataset and subsequently used for supervised learning. Example: Using a generative model to learn how glass defects look like and apply this to small datasets of glass without a defect to generate data that contains defects	Strengths: The only technique that may work if one has neither labelled nor unlabelled data on a given Small Data problem Weaknesses: Notoriously difficult to train so that each new application requires a lot of work
Auto-Encoder	Auto-Encoders are trained on unlabelled data to learn a low dimensional representation of the data.  Example: Reducing complex genetic sequences to a few numbers to separate different kinds of sequences.	Strengths: No labelled data required, Easy to use Weaknesses: Unsupervised models rarely learn human relevant features.
Self-Supervised Learning	Similar to Auto-Encoders but the training is augmented by exploiting the underlying structure of the unlabelled data. Example: Masking parts of microscopy images and make an AI model learn to reconstruct the full images. This way the network learns general features of the data and can be specialised on smaller datasets afterwards (Semi-Supervised)	Strengths: Very powerful & popular technique especially if combined with labelled data (semi-supervised). Weaknesses: Finding applicable perturbations might require some trial and error for different datatypes
Transfer & Meta Learning	A 2-step process in which an AI model is first trained on a big, labelled dataset that is like the small dataset for which an AI is required. The trained AI model is then fine-tuned on the small, labelled dataset. Adding Meta Learning to the first training ensures that the resulting model can learn new tasks very efficiently.  Note that the first training step could also be done using self-supervised learning, making this a good semi-supervised technique as well. Example: Train an image analysis AI on a big dataset containing many labelled cell microscopy images. Then fine tune the model on a small dataset from one specific cell type.	Strengths: Very powerful technique that can be used in many cases. Weaknesses: Training the model twice poses some practical challenges.
Active Learning	In active learning one starts with an AI model that can predict which datapoints need to be labelled first. After the first labelling step, the process is repeated so that humans only need to label the minimal number of datapoints for the AI to achieve the best performance. Example: After testing a few drugs in the laboratory an AI model is built to predict outcomes for other drugs together with a prediction confidence score. Subsequently, the drugs that exhibit low confidence are tested in the laboratory to boost performance.	Strengths: Good at finding edge cases that require labelling & does not require unlabelled data Weaknesses: May not reduce the amount of data required by much

Table 1 – An overview of the most popular supervised and unsupervised Deep Learning based Small Data AI techniques.

Life Sciences industries are amongst the most affected by the Small Data problem. Unlabelled data, i.e., microscopy images or potential target binding compounds, are relatively accessible, but labelled data requires expert labelling work that may involve slow computations (e.g. computer programs that take days to compute an analysis), lengthy manual labelling by eye (e.g., of pathology images), and expensive laboratory experiments (e.g. compound binding assays). Life Sciences data is heterogeneous, messy, and small making the industry ripe for disruption with lightweight problem focussed Small Data AI models [13]. DeepMirror is at the forefront of this disruption by providing Small Data AI as a service to organisations in the Life Sciences. To achieve this, we use a mixture of Transfer Learning and Self/Semi-Supervised learning paired with specialist AI Models for different Life Science data types (e.g., medical images, small molecules, and DNA/RNA sequences). Our technology enables tumour analysis in pathology images of patients using 100x less labelled data than conventionally required [14], and can predict new & better drugs against malaria from just 300 previously tested drugs [15]. Have a look at our blog for more information [16], and reach out to us if you are interested in what we do. Let’s work together to make breakthrough discoveries with Small Data AI!

Contact: Max Jakobs (max@deepmirror.ai)

Edited by: Andrea Dimitracopoulos, Eva Pillai, Craig Russell, Ala Alenazi, Ryan Greenhalgh, Amir Shirian, Sasha Eremina, Brieuc Lehmann

References

[1]	[Online]. Available: https://www.mckinsey.com/business-functions/mckinsey-analytics/our-insights/global-survey-the-state-of-ai-in-2020.
[2]	[Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6669998/.
[3]	[Online]. Available: https://www.salary.com/research/salary/benchmark/physician-pathology-hourly-wages.
[4]	[Online]. Available: https://arxiv.org/pdf/1805.06983.pdf.
[5]	[Online]. Available: https://www.forbes.com/sites/forbestechcouncil/2021/09/23/model-drift-in-data-analytics-what-is-it-so-what-now-what/?sh=6ffef4cb4862.
[6]	[Online]. Available: https://www.protocol.com/enterprise/landing-mariner-ai-manufacturing-defect.
[7]	[Online]. Available: https://www.synthego.com/crispr-benchmark.
[8]	[Online]. Available: https://www.nature.com/articles/nrd3078.
[9]	[Online]. Available: https://theconversation.com/90-of-drugs-fail-clinical-trials-heres-one-way-researchers-can-select-better-drug-candidates-174152.
[10]	[Online]. Available: https://www.forbes.com/sites/brentdykes/2015/10/14/analyzing-big-data-8-tips-for-finding-the-signals-within-the-noise/?sh=5969fd7e16f5.
[11]	[Online]. Available: https://www.nature.com/articles/s41586-021-04198-4
[12]	[Online]. Available: https://www.protocol.com/newsletters/protocol-enterprise/manufacturing-ai-salesforce-shay-banon?rebelltitem=1#rebelltitem1.
[13]	[Online]. Available: https://www.mercuryds.com/blog/5-data-challenges-healthcare-lifesciences.
[14]	[Online]. Available: https://deepmirror.ai/2022-02-14-digitalpathology/.
[15]	[Online]. Available: https://deepmirror.ai/2022-03-18-open-source-drug-discovery/.
[16]	[Online]. Available: https://deepmirror.ai/blog/.

Written by Ashna Ahmad

The process of analysing pathology images, also called slides, to diagnose cancer and other diseases is still largely driven by expert human pathologists. With one in two people developing cancer at some point in their lives,  it is becoming increasingly important to diagnose and treat cancers early – but pathologists are in limited supply and take several hours to analyse a single slide and find hallmarks of diseases in patient samples (Figure 1). Exclusively employing human pathologists unnecessarily lengthens the time to diagnosis and leads to increased costs for public healthcare systems already strained by global health emergencies such as the COVID19 pandemic.

Figure 1: A whole-slide-image (WSI) of a prostatectomy specimen. Pathologists need to manually scan through the whole image to detect whether cancerous tissue is present. Cancerous tissue might be present in small subregions (also called “patches”) as the one highlighted in yellow. Scanning through the whole image might take many hours of exhausting precision work. If the region is missed, the patient could be misdiagnosed.

For this reason, digital pathology – the technology used to analyse information from a digital slide – presents us with incredible opportunities to make diagnostic processes much simpler and faster. Digital whole-slide imaging (WSI) provides a reliable platform for many basic analysis tasks, as well as image-sharing between different teams, and automated image analysis methods using WSI are growing in sophistication and capability. In 2021 Paige.ai developed the first FDA-approved digital pathology tool for prostate cancer detection [1], but new breakthroughs are needed before digital tools become sufficiently accurate and low-cost to effectively assist pathologists in their diagnoses, which are currently carried out through manual light microscopic evaluations as the industry standard.

One of the main costs associated with the development of digital pathology tools is the sheer amount of data curation required. Creating an AI model for automated image analysis may require years of model training and optimisation, and huge training sets made of tens of thousands of individual WSIs. The model employed by Paige.ai needed more than 12,000 WSIs to be powerful enough to be used for diagnosis [2]. Populations and diseases vary over time, and AI models need to be regularly and rapidly updated. This requires even more annotated WSIs. If we want to use digital pathology as a method to cut down the time and expenses involved in WSI analysis, it is crucial to develop models which can work with small labelled datasets.

Meet DeepMirror Spark. For Digital Pathology!

This is where DeepMirror Spark, our Breakthrough Discovery Platform, comes in. Our semi-supervised technology can build high-accuracy models with 10-100x less labelled data than conventional training methods. We previously showed how the platform performed for instance segmentation in microscopy image data (see here). Strikingly, our technology is also effective for the most important WSI applications: semantic segmentation of sub-cellular structures in digital slide images (i.e. separating regions into different tissue types), and classification of image regions (i.e. answering whether a particular region contains cancerous tissue or what clinical outcome is associated with a given image). Accurate segmentation and classification are instrumental in quickly identifying potentially unhealthy tissue and separating patient images into groups for detailed analysis. Making these processes as fast and effective as possible could have an enormous impact on the diagnostic process.

We tested DeepMirror Spark on both classification and semantic segmentation of WSI patches. Figure 2 shows results from building an image classification model for the PatchCamelyon dataset, a collection of 200,000 images containing either some tumour tissue or only non-tumour tissue. Our model produces consistently more accurate classifications compared to conventional training, and performance plateaus at a dataset size of 1,000 images, 200x smaller than the original dataset, at an Area Under Curve (AUC) score of >0.90. Achieving highly accurate classification results with 1,000 training images, rather than 10x or even 100x that amount, would enable the rapid model development needed for large-scale AI deployment of classification models in digital pathology. We are carrying out pilot studies with partner organisations to bring this breakthrough advances to the clinic.

Figure 2: Side-by-side comparison of DeepMirror Spark and conventional training for image classification. We trained a custom classifier neural net on the PatchCamelyon dataset (https://patchcamelyon.grand-challenge.org) which contains 92×92 pixel images that depict either tissue contain tumour or no-tumour. In total the dataset contained ~200,000 images. (A) Example classifications for conventional training and DeepMirror Spark compared with Ground Truth (GT) using only 1,000 images. (B) Cross validated area under the curve (AUC) scores for varying amounts of labelled data. DeepMirror Spark outperforms conventional training at each dataset split.

Using DeepMirror Spark for semantic segmentation produced even more ground-breaking results. Using a modified version of the popular DeepLabv3+ network we again performed an ablation study in which we systematically increased the number of samples used for training with DeepMirror Spark and a conventional training algorithm (Figure 3). As a benchmark we used the PESO dataset which contains patient prostate epithelium samples in which cancerous tissue has been segmented. Networks trained with DeepMirror Spark reached peak performance (a median IoU of ~0.93) with just 200 (!) samples. Using conventional training, scores were highly variable between cross-validation steps and generally below 0.6 IoU, even when using 2,000 (10x more) samples.

Figure 3: Side by side comparison between DeepMirror Spark and conventional training for the semantic segmentation of tumours in prostate epithelium (PESO dataset). We trained a customised DeepLabv3+ semantic segmentation model both with DeepMirror Spark and conventional training. To do so we sampled 10,000 patches from the full dataset and trained the network with an increasing number of samples using either DeepMirror Spark or conventional training. (A) Example segmentations for 256×256 pixel patches and the corresponding Jaccard Scores for models trained with 200 patches. (B) Cross-validated Jaccard Score (Intersection over Union)) as a function of the number of used patches (256×256 pixels). As seen, conventional training performance is highly variable while DeepMirror’s training reaches peak performance with as few as 200 patches.

These results demonstrate the breakthrough capabilities of our discovery platform to build the future of digital pathology. DeepMirror Spark is an exciting step towards a cost-effective AI solution capable of assisting medical professionals in the long, labour-intensive analysis workflows of current pathology.

References

[1] https://www.fiercebiotech.com/medtech/fda-clears-paige-s-ai-as-first-program-to-spot-prostate-cancer-amid-tissue-slides
[2] https://arxiv.org/pdf/1805.06983.pdf

Written by Ryan

AI models get smarter, more accurate, and therefore more useful over the course of their training on large datasets that have been painstakingly curated, often over a period of years. 

But in real-world applications, datasets start small. To design a new drug, for instance, researchers start by testing a compound and need to use the power of AI to predict the best possible permutation. But the actual datasets are too small for conventional AI, leaving the models without the raw material on which to train.

At DeepMirror we’ve pioneered new AI training algorithms that learn to predict from small datasets. Our new training algorithm, called DeepMirror Spark, reduces the amount of data required to train AI for computer vision by 10-100x (check out how this works for biomedical images here (https://deepmirror.ai/2021-12-09-spark-instance-segmentation/).

The jump from training an AI model to delivering the end result is not simple. In fact, most machine learning models never make it into production [1]. Yes, that’s right. All those hours clocked by talented engineers don’t get used by customers in the real world. When I first heard this statistic, I was shocked – these models are the most valuable part of research, so how could this be the case? In this post, I’ll address some of the issues we often face when transitioning from machine learning research to production. One of the benefits of working at a start-up is that we can establish good processes from the outset to develop flawless internal machine learning and deployment workflows. This is what I’d like to share.

For each new project. we need to manage multiple trained models, grow our annotated datasets, compare the performance of different models, benchmark them against different datasets, and deploy the winners for our users. Thus, to optimise our research to production workflow, we face these five engineering challenges:

1. Log, monitor, and share experiments
2. Data versioning and storage
3. Reproduce and replicate 
4. Autoscaling GPU compute
5. Serving and deploying

It would take a lot of time and money to design a system to manage each of these machine learning operation (MLOps) components from scratch. After researching several MLOps providers, we found that ClearML significantly outperformed other offerings. ClearML is a Python-based MLOps platform that provides the tools to solve our research to production challenges outlined above. We were mainly interested in having the infrastructure automatically monitor and log the training of models and a simple deployment step. Figure 1 summarises our machine learning lifecycle and systems design. Here, there are two main processes happening:

1. The machine learning team can create and design different models, train the models with scalable compute, and benchmark the models across the team.  
2. The client can interact with the backend API by submitting data and using models to make predictions on their data. It is also possible to train new models, which is handled by the same framework used by the machine learning team.

Figure 1: Diagram of DeepMirror ML-Ops research to production workflow and systems design.

Let’s take a look at how we used ClearML to solve the five challenges mentioned above.

Log, Monitor, and Share Experiments

At DeepMirror, we want to develop new technologies to rapidly train our AI, and deploy the best performing models to clients. To select the best performing models, we need to quickly compare the performance of different models and benchmark the models against other datasets. This requires us to log, monitor, and share results from various experiments and across research teams. Without logging and monitoring, experiments can lie redundant on local machines, or researchers will have to send files back and forth slowly: accumulating technical debt. 

ClearML made it is easy to log and monitor experiments. All we needed to do to set up automatic logging was to initialize a ClearML task at the start of our training script:

Now all the terminal outputs and Tensorboard outputs were saved remotely for other engineers to view on the ClearML UI.

Data versioning and Storage

For internal ML research, we need engineers to be able to efficiently access datasets to create new models. During this process, it can be difficult to manage access to collections of datasets between engineers, giving rise to data versioning and storage challenges. 

In addition to sharing the logs and monitoring experiments across the ML team, it is also important to be able to replicate datasets when training on different machines. ClearML makes it easy to upload and replicate datasets with a few lines of code. To upload a new dataset, our engineers can push it to our cloud storage by running the following code:

It’s also easy to replicate the dataset within a task:

This makes it easy for different ML engineers working on different machines to programmatically retrieve datasets locally.

Reproduce and Replicate

During the development process we need to be able to replicate older versions of our code. For example, a researcher might want to retrain an old model with a new hyperparameter, or on a different dataset. Usually, this would require the engineer to replicate the dependency environment and datasets. Ensuring that the correct environment is set up can often be a time sink for the engineer.

The ClearML UI makes it easy to replicate and reproduce different experiments (tasks). A task can be cloned and then retrained or have specific hyperparameters changed. This makes it easy to run experiments locally, without even needing Python installed – simply through the browser.

Autoscaling GPU compute

Additionally, we need to be able to train multiple models to benchmark model performance. To get fast turnaround, it is critical to have access to a large GPU compute resource. Running many GPU instances on a cloud provider can be expensive, so ­‑ ideally, we want to have just the right number of resources on demand. To meet this requirement, we need to have an autoscaling GPU compute. 

One of the features that we really love about ClearML is the ClearML Agent and autoscaling service. The ClearML Agent can be installed on our local GPU machines, and when a new job is sent to the queue (either by a ML engineer or client over the browser) the Agent will receive and execute the task locally (illustrated in Figure 1). At first, we used a few on-premise machines to train models, but quickly ran out of compute resources during surges of demand. ClearML saved us again – this time with their Autoscaler. We used the AWS autoscaling service from ClearML to monitor the number of jobs in the task queue and spin up EC2 GPU instances with ClearML Agents installed to execute all the tasks in the queue. The autoscaling feature significantly accelerated our ML model development and deployment. 

Serve and Deploy

A key component of our offering is to provide clients the ability to train their own custom models. To get this done, we need a system that allows crosstalk between ML engineers and clients with shared resources. Once our team has developed a working model or a client has trained a model, we need to be able to serve and deploy the model back to the client.

Serving models in this way enables clients to make predictions on their data. To do this, we simply use ClearML on our backend to cache a local copy of the model from a specific task in just a few lines of code. 

In summary, ClearML has enabled us to smash through the research-to-production barrier. The platform makes it easy for our research team to quickly get models into production while monitoring their performance, without worrying about GPU resources. The team at ClearML and their community have even been great at providing support on the community Slack. Using ClearML as an all-in-one ML-Ops solution has saved us time and money and will continue doing so in the future as we can focus on developing our core technology. 

Reference

1. https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/

Written by Max

Almost all biomedical image analyses start with the acquisition of a few images, be it a clinical trial in which MRI scans are acquired from patients or a laboratory trying out new treatments on cultured cells. To extract clinical & scientific insights from these images, researchers and clinicians could benefit from recent advances in Artificial Intelligence (AI), for example to automatically detect specific regions of interest in these images, or to automatically classify images into multiple categories. While these tasks can be carried out reliably by human experts, this takes a lot of time and money. AI can help but needs to be trained with giant datasets that have been painstakingly curated by hand. Human-driven data curation (also called annotation) often take months or years and poses a barrier to the application of AI to new problems.

But what if one could reduce the number of annotations required to train AI? With reduced annotations, researchers & clinicians would be able to get AI up and running for new applications in no-time, accelerating breakthroughs in the biomedical sector. This would have tremendous real-world applications since while the amount of raw data is often vast, the number of validated data is not. At DeepMirror, we set out to tackle this challenge by considering how humans learn. To learn to recognize cats, for example, humans do not need a giant dataset of annotated cat images. Often a few are enough and by cross referencing with other images a human can accurately locate cats in their vicinity. We wondered if we could replicate this with an AI technique called semi-supervised learning. In semi-supervised learning, AI is trained with a few known & validated datapoints, such as for example a few images in which cats have been annotated, and a large dataset of non-annotated examples. After almost 2 years of iterating on semi-supervised learning algorithms, we built a modern framework that focuses on real world applicability, i.e., is easily adjustable for any kind of AI task.

Meet DeepMirror Spark, a semi supervised AI training platform that uses specialized data augmentations (i.e. distorting data so that it “looks” like new data), adversarial learning (using two competing AI with the objective of improving AI performance), and other techniques, to train AI on small datasets!

Our early clients often struggled with instance segmentation, which is the accurate detection of individual object shapes from images. Instance segmentation can for example be used to detect the size of cancers in radiology images and the abundance of biomarkers in histopathological images. To test and benchmark our DeepMirror Spark platform, we applied it to a customized biomedical instance segmentation architecture based on the popular UNet (Figure 1).

AI training performance

This UNet-based implementation of DeepMirror Spark alone can be used for a myriad of different image analysis tasks. In Figure 2 & 3, for example, we trained a UNet to detect Islets of Langerhans in human pancreas images with just 46 annotated images, and brain tumors in MRI scans with only 75 images. In both cases conventional (i.e., non semi-supervised) training led to lower training performance and worse segmentation.

Ablation Studies on radiology & cell biology image data

To test DeepMirror Spark’s performance as a function of annotated images we performed ablation studies, i.e. we took several publicly available and internally generated image datasets and varied the number of labelled images. In doing this experiment we were able to quantify the improvement that DeepMirror Spark has over conventional training, for small datasets (Figure 4-6). 

We observed that DeepMirror Spark enabled us to reach the maximum Average Precision (AP) score with only 20-300 images depending on complexity of the data. Conventional training of the same network failed to reach the same AP scores with any number of images from a small dataset, implying that one can generate functional AI much faster using DeepMirror Spark.

What’s next

We are now working on extending the DeepMirror Spark platform to perform other tasks beyond instance segmentation. In the next few weeks, we will add image classification, semantic segmentation and more. Additionally, we are working on integrating other datatypes such as genomic sequences and molecular structures for gene editing experiments and drug discovery! Stay tuned for further blogposts on these topics 🙂

If you want to add AI to your projects in the fastest way possible, get in touch with us at enquiries@deepmirror.ai to do a pilot study on your dataset!

Figures

Figure 1: Schematic – DeepMirror Spark for UNet based instance segmentation. To use DeepMirror Spark for instance segmentation we trained a custom UNet with the platform algorithm using both labelled and unlabeled images of biological cells. The trained UNet can then be used for instance segmentation on other images to generate annotations or perform analysis.

Figure 2: Detecting Islets of Langerhans with DeepMirror Spark. We trained a UNet to detect islets of Langerhans (or pancreatic islets) in human pancreatic slices with both DeepMirror Spark and conventional non-semi-supervised training. Cells inside pancreatic islets produce insulin and measuring their abundance is important in diabetes research & diagnosis.  While the network that was trained with Spark was able to pick up Islets, the one that was trained conventionally did not. The Root Mean Squared Error (RMSE) of the validation dataset during training shows this difference. The networks were trained with 46 labelled images and 1087 unlabeled ones. Image source: Novo-Nordisk histologic image analysis challenge (https://www.innovitaresearch.com/2020/04/28/novo-nordisk-challenge-histologic-image-analysis-of-pancreatic-tissue/).

Figure 3: Detecting Brain Tumors with DeepMirror Spark. We trained a UNet to detect brain tumors in MRI scans with both DeepMirror Spark and conventional non-semi-supervised training. Diagnosing tumors rapidly without constant expert supervision would free up radiologists’ time to focus on the important cases. While both networks were able to pick up the tumor in the example, the conventionally trained one also mis-classified other parts of the image as tumors. The Root Mean Squared Error (RMSE) of the validation dataset during training shows this difference. The networks were trained with 75 labelled images and 750 unlabeled ones. Dataset obtained from: (https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0157112)

Figure 4: Ablation study of neuron segmentation. Here we iteratively increased the number of labeled images used for training to a maximum of 80 images. The left hand side shows the raw image and a manual labeling. The Average Precision (AP) score for semi-supervised training with DeepMirror Spark outperformed conventional training at all labelled dataset sizes and reached a plateau after ~20 images for DeepMirror Spark. All quantifications were done on a separate test dataset. The Dataset was generated and annotated by Eva Kreysing from the Franze laboratory at the University of Cambridge.

Figure 5: Ablation study of brain tumour segmentation. Here we iteratively increased the number of labeled images used for training to a maximum of 640 images. The left hand side shows the raw image and a manual labeling. The Average Precision (AP) score for semi-supervised training with DeepMirror Spark outperformed conventional training at all labelled dataset sizes and reached a plateau after ~300 images. All quantifications were done on a separate test dataset. Dataset taken from: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0157112.

Figure 6: Ablation study of generic cell segmentation (Human Protein Atlas dataset). Here we iteratively increased the number of labeled images used for training to a maximum of 320 images. The left hand side shows the raw image and a manual labeling. The Average Precision (AP) score for semi-supervised training with DeepMirror Spark outperformed conventional training at all labelled dataset sizes and reached a plateau after ~10 images. All quantifications were done on a separate test dataset. Data taken from https://www.proteinatlas.org and annotated by hand.