Experimental assessment of a 3D plenoptic endoscopic imaging system

Spatial reconstruction of 3D-printed objects. (a-c) Microlens image of a plane and inhomogeneous objects and (d-f) its reconstructed depth maps.

A realistic three-dimensional (3D) reconstruction of a surgical site has become a highly needed imaging method in minimally invasive surgery. To date, common clinical and research efforts have employed 3D reconstruction methods based on stereoscopy, time-of-flight, or structured illumination to extract the concavity and convexity of the target specimen. Nevertheless, all of these techniques suffer from mediocre performance for obtaining high depth resolution unless some auxiliary measures are employed, which increase the system complications inevitably.

Recently, an extension of stereoscopy technique called plenoptic imaging has been investigated to deduce 3D shape, which has been widely applied in industry for its multi-focus function and the simplified correspondence searching performance. However, its application in medicine has not been fully explored, as the current depth precision is limited to 1 mm, and no endoscopic plenoptic camera exists.

To adopt the plenoptic technique in 3D endoscopic vision, the photonics and mechanical engineering groups led by Dr. Jin Ung Kang (Photonics and Optoelectronic Lab, Johns Hopkins University) and Dr. Axel Krieger (Sheikh Zayed Institute for Pediatric Surgical Innovation, Children's National Health System) have proposed a plenoptic endoscope with a customized relay optical system to be used in minimally invasive setting. This work was published in Chinese Optics Letters, Volume 15, No. 5, 2017 (Hanh N. D. Le, et al., Experimental assessment of a 3-D plenoptic endoscopic imaging system).

The plenoptic imaging technique involves a microlens array integrated onto an imaging sensor, such that each point of the object can be viewed and imaged at different angles via adjacent microlenses, which is analogous to the stereoscopy approach. Besides, the presented setup compensates for the aperture mismatch between the endoscope and the microlens array fabricated on the plenoptic camera. The achieved depth accuracy error of about 1 mm and precision error of about 2 mm are recorded within a 25 mm × 25 mm field of view and the system operates at 11 frames per second.

Future development on the design will focus on facilitating multiple modalities to achieve an even higher level of surgical vision in minimally invasive surgery.

三维全光内窥镜成像，让手术视野更清晰

图片说明：3D打印物体的空间重建成像：（a-c）平面和不规则物体的微透镜成像，及其对应的（d-f）三维重建的深度图。

到目前为止，一般的临床研究工作多使用基于体视镜、飞行时间或者结构照明的三维重建方法来提取目标样本的深度轮廓。然而，所有这些技术都受困于其在获取深度信息方面所存在的短板。除非采用一些辅助的手段，否则利用上述方法将很难得到高分辨的深度信息。但是辅助手段的引入又会不可避免地增加系统的复杂性。

近来，全光成像作为体视镜技术的一种拓展得到了广泛研究。全光成像技术采用了微透镜或多相机阵列，使得其具备了多重焦点功能和简化的对应搜索表现，在记录光的强度信息的同时，也能够感知光的色彩和方向信息，由此在工业上得到了广泛应用。然而，全光成像技术在医疗领域的应用潜力却尚未得到充分的挖掘，这是由于其目前的深度精度只能达到约1 mm，另外，内窥镜全光相机也不存在。

为了将全光成像技术应用到三维内窥镜观察当中，约翰霍普金斯大学的Jin Ung Kang博士和谢赫扎耶德小儿外科创新研究所的Axel Krieger博士领导的研究团队提出了一种用于微创情形的带有中继光学系统的全光内窥镜方案，并对体系的三维成像性能进行了实验评估。相关结果发表在Chinese Optics Letters 2017年第5期上(Hanh N. D. Le, et al., Experimental assessment of a 3-D plenoptic endoscopic imaging system）。

这一全光成像技术将一个微透镜阵列集成在成像传感器上，相邻的微透镜可以不同的角度对物体上的每一点进行观察和成像，其原理类似于立体镜方法，不过避免了立体镜方法中需要设置两个观察视角的繁琐程序。另外，这一布置对存在于内窥镜和全光相机上的微透镜阵列间的孔径失配作了补偿。当系统工作在11 fps的状态时，在25 mm × 25 mm的视场范围内可获得的深度的两种精度误差（accuracy error，由在0度和45度角的拟合面和参考面的平均距离和标准差表征；precision error，由数据点与拟合面之间的偏差来定义）分别约为1 mm和2 mm。

该研究团队下一步的工作旨在改进多个模块之间的配合，以便进一步提高这一全光内窥镜系统在微创手术中的观察视野的精度。