Multiplexed imaging methods can measure the expression of dozens of proteins while preserving spatial information. While these methods open an exciting new window into the biology of human tissues, interpreting the images they generate with single cell resolution remains a significant challenge. Current approaches to this problem in tissues rely on identifying cell nuclei, which results in inaccurate estimates of cellular phenotype and morphology. In this work, we overcome this limitation by combining multiplexed imaging’s ability to image nuclear and membrane markers with large-scale data annotation and deep learning. We describe the construction of TissueNet, an image dataset containing more than one million paired whole-cell and nuclear annotations across eight tissue types and five imaging platforms. We also present Mesmer, a single model trained on this dataset that can perform nuclear and whole cell segmentation with human-level accuracy – as judged by expert human annotators and a panel of pathologists – across tissue types and imaging platforms. We show that Mesmer accurately measures cell morphology in tissues, opening up a new observable for quantifying cellular phenotypes in vivo. We make this model available to users of all backgrounds with both cloud-native software and on-premise software. Last, we also describe ongoing work to develop a similar resource and models for dynamic live-cell imaging data.
David Van Valen PhD is faculty in the Division of Biology and Bioengineering at the California Institute of Technology. His research group's long-term interest is to develop a quantitative understanding of how living systems process, store, and transfer information, and to unravel how this information processing is perturbed in human disease states. To that end, his group leverages and pioneers the latest advances in imaging, genomics, and machine learning to produce quantitative measurements with single-cell resolution as well as predictive models of living systems. Prior to joining the faculty, he studied mathematics (BS 2003) and physics (BS 2003) at the Massachusetts Institute of Technology, applied physics (PhD 2011) at the California Institute of Technology, and medicine at the David Geffen School of Medicine at UCLA (MD 2013).