Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

  1. K. C. Zhou, M. Harfouche, C. L. Cooke, J. Park, P. C. Konda, L. Kreiss, K. Kim, J. Jonsson, T. Doman, P. Reamey, V. Saliu, C. B. Cook, M. Zheng, J. P. Bechtel, A. Begue, M. McCarroll, J. Bagwell, G. Horstmeyer, M. Bagnat and R. Horstmeyer, “Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second,” Accepted to Nature Photonics (2023). CodeDataPre-PrintDOIProject Page

Abstract


Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over a 135-cm<sup>2</sup> area, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms: ants, fruit flies, and zebrafish larvae.

Overview of 3D-RAPID:

3D-RAPID uses a parallelized architecture consisting of 54 synchronized cameras, with a total spatiotemporal throughput exceeding 5 gigapixels/sec. Using a self-supervised, neural network-based 3D reconstruction and stitching algorithm, we are able to generate 3D videos of freely behaving organisms, such as zebrafish, ants, and fruit flies.

Freely swimming zebrafish larvae:

The left panel shows the full-FOV, stitched composite and the right panels show the zoomed-in, individually tracked fish. The video alternates between showing the photometric (RGB) frames and the 3D height map frames.

Field of view: 135 cm,
frame size: 36.6 MP,
frame rate: 60 fps

Freely exploring adult fruit flies:

The left and right panels the stitched photometric (RGB) composites and the reconstructed 3D height maps, respectively.

Field of view: 135 cm,
frame size: 36.6 MP,
frame rate: 60 fps

Zebrafish larvae feeding on brine shrimp:

The left and right panels the stitched photometric (RGB) composites and the reconstructed 3D height maps, respectively.

Field of view: 135 cm,
frame size: 36.6 MP,
frame rate: 60 fps

// First try loading jQuery from Google's CDN