Gigapixel behavioral and neural activity imaging with MCAM

[Paper] [Code][Dataset]

Authors Eric E Thomson1, Mark Harfouche3, Pavan C Konda2, Catherine Seitz1, Kanghyun Kim2, Colin Cooke2, Shiqi Xu2, Whitney S Jacobs1, Robin Blazing1, Yang Chen1, Sunanda Sharma3, Timothy W Dunn2, Jaehee Park3, Roarke Horstmeyer2, 3* & Eva A Naumann1*
Affiliations:
  1. Duke School of Medicine Department of Neurobiology, Durham, NC 27710
  2. Biomedical Engineering, Duke University, Durham, NC 27710
  3. Ramona Optics Inc., Durham, NC 27701
*Corresponding Authors/These authors jointly supervised this work: eva.naumann@duke.edu roarke.w.horstmeyer@duke.edu

Abstract

Multi-Camera array Microscope or MCAM refers to an assembly of multiple microscopes, each equipped with image sensors, tightly packed together in an array like fashion. A usual assembly consists of upto 96 cameras which together capture and computationally generate gigapixel-scale images for a given sample over a hundered of cm2 of Field of View(FOV). The MCAM is also capable of recording  movies for dynamic samples at a near cellular resolution and sensitivity of 5 um for the same FOV.
MCAM-96_Exploded_view

Overview of a Multi-Camera Array Microscope(MCAM)

 Here we employ this system to observe behaviour and capture high resolution anatomical feature details for various freely moving model organisms over a large FOV. Thus, our system enables high-resolution recording from multiple spatial scales simultaneously which allows its usage in wide range of biological imaging applications, such as, studying cellular structures and capturing large-group behavioural dynamics .

We also demonstrate the ability to acquire dual channel fluorescence video of multiple freely moving zebra fish and record their neural activity using ratiometeric calcium imaging, which supplements the existing behavioural tracking analysis by providing more quantitative measurements for these moving specimen.

Example frame of freely swimming zebrafish larvae

Raw Video Snapshot

Gigapixel Imaging System:  A.Four modular units in a 2×2 fashion, each consisting of 24 sensors, arranged together to demonstrate scalibility. B. Raw Video Acquisition

Design

Following are the design specifications of the acquisition system:

  1. The camera assembly consists of upto  in 8 x 12 rectanglar array fashion
  2. FOV of imaging sample is 384 cm 2
  3. Full pitch optical resolution is 18 um 
  4. Scalable

Generating image composite and performing analysis for MCAM

This is the heading

Observing Zebrafish behaviour at multiple spatial scales and comparing it with a single-lens system imaging same FOV(16x24 cm)

Methods

  1. Simultaneous Image acquisition from the camera array.
  2.  Data acquisition post processing for efficient storage
  3. Standard image processing to correct for vignetting and brightness variations between the cameras.
  4. Digitally stitch acquired image into a single composite frame.
  5. Perform image analysis task such as organism trajectory assesment and multi-organism location measurement. 
  6. Use machine learning and other computational algorithms to perform further analysis on the acquired dataset.
 

Results

A. High-resolution, large-FOV imaging of  large groups of larval zebrafish

Key points: 
  1.  Recorded behaviour of ~100 wild-type 8 days post-fertilization zebrafish placed into a 22 x 11 3D printed arena.
  2. Resolved various morphological features of individual fish such as: pectoral fish position and distribution of melanophores.
  3. A CNN based object detector adopted to detect and crop individual organisms for further analysis  

A. CNN Object Detection

B. Zebrafish headshots

C. Zebrafish identification

D. Tail Tracking

Fig_2_E_Eye_position_tracking

E. Eye position tracking

F. Stereoscopic Imaging and depth

G. 3D. tracking of zebrafish, Z-displacement estimated by stereoscopic depth tracking

A) Nine Zebrafish headshots as input to the Siamese neural network for zebrafish identification. B) 2D Visualization demonstrating network capabilitiy to differentiate among the zebrafishes. D)Tail tracking visualization for three zebrafish E.) Histograms of Eye angles on 130 ZF across 20 frames F. Schematic of stereoscoic depth tracking.G.) 3D trajectory of 2 example larvae recorded gigapixel video of 130 WT zebrafish. 

  1. Vast reduction in memory and dataset handling requirements was achieved by performing analysis only on the cropped images.
  2. Exploited high resolution imaging capability and differences in melanophore patterns to recognise individual zebrafish using morphological differences among the zebrafishes.
  3. Also identified behavourial details of each zebrafish by performing tail tracking and capturing eye position measurements.

B. Simultaneous behaviour and fluorescence imaging

 
Key Points: 
  1. Integrated fluorescence imaging capabilities to MCAM unit by adding  two sets of custom excitation sources for a 24 unit MCAM.
  2. Also mounted a bank of 24 emission filters via 3D printed filter array holder over each camera lens.
  3. The above addition enabled capturing epi-illuminated fluorescence images from each MCAM sensor.

A. Fluoroscence Imaging setup

B. Drosophilla melanogaster example fluorescent MCAM image at high magnification

C. Calcium Imaging

D. Fluorescence Imaging

E. Neural activity monitoring during swimming for six example zebrafish captured in D.

A. Fluorescence Imaging setup schematic, B. Drosophilla melanogaster example fluorescent MCAM image at high magnification C.  Average fluorescent Imaging of agarose embedded zebrafish with, coloured sections represent ROI plotted on the right. Avg. ∆F/F traces demonstrate MCAM capabilities to resolve differences in neural activity,   D. 14 Freely moving Zebrafish in a 24 camera MCAM system, tracked over a 120 s video, E. Fluorescent activities of six example zebrafish captured in D.

  1. Imaged freely moving drosophila larvae, epi-fluorescent setup enabled observation of proprioceptive neurons(responsible for movements in an individual) in the region of interests.
  2. Monitored neural activity by acquiring fluorescence images from zebrafish expressing GCaMP6( a genetically encoded fluorescent calcium indicator).
  3. Combined fluorescence and behavioral tracking, using a 24-camera MCAM, observed increase and decrease in neural activity marking onset and end of swimming bouts(seen in sporadically moving zebrafish).

A. Dual Channel Setup

B. Dual Color Zebrafish

C. Ratiometric Imaging

A. Dual Channel Imaging configuration. Each area imaged by two MCAM sensors simultaneously, recording red and green fluorescence signals. B. Example dual-channel fluorescence image for a transgenic zebrafish expressing two different fluorescent signals indicators. C. Ratiometric analysis of neural activity for the sample example zebrafish. Movement is marked by increase in fluorescence signal.

  1. Installed alternating red and green fluorescent filters and utilized MCAM overlap between adjacent FOVs to perform ratiometric imaging.
  2. Captured dual channel fluorescence red signal doesn’t change and helps in compensating movement artifacts while recording GCaMP6 fluorescence

Example Fluorescence videos

Fluorescence activity of  different Zebrafish at multiple spatial scales

// First try loading jQuery from Google's CDN