Differentiable Rendering for 3D Fluorescence MicroscopyDifferentiable rendering is a growing field that is at the heart of many
recent advances in solving inverse graphics problems, such as the
reconstruction of 3D scenes from 2D images. By making the rendering process
differentiable, one can compute gradients of the output image with respect to
the different scene parameters efficiently using automatic differentiation.
Interested in the potential of such methods for the analysis of fluorescence
microscopy images, we introduce deltaMic, a microscopy renderer that can
generate a 3D fluorescence microscopy image from a 3D scene in a fully
differentiable manner. By convolving the meshes in the scene with the point
spread function (PSF) of the microscope, that characterizes the response of its
imaging system to a point source, we emulate the 3D image creation process of
fluorescence microscopy. This is achieved by computing the Fourier transform
(FT) of the mesh and performing the convolution in the Fourier domain. Naive
implementation of such mesh FT is however slow, inefficient, and sensitive to
numerical precision. We solve these difficulties by providing a memory and
computationally efficient fully differentiable GPU implementation of the 3D
mesh FT. We demonstrate the potential of our method by reconstructing complex
shapes from artificial microscopy images. Eventually, we apply our renderer to
real confocal fluorescence microscopy images of embryos to accurately
reconstruct the multicellular shapes of these cell aggregates.
arxiv.org