视觉定位论文35

Paper #35

题目：

Hierarchical Visual Localization for Visually Impaired People Using Multimodal Images

基于多模态图像的视障人士分层视觉定位

出书来历：Expert Systems with APPlications（ELSEVIER）， 2021

摘要：

Localization is one of the crucial issues in assistive technology for visually impaired people. In this paper, we propose a novel hierarchical visual localization pipeline based on the wearable assistive navigation device for visually impaired people. The proposed pipeline involves the deep descriptor network, 2D-3D geometric verification and online sequence matching. Images in different modalities (RGB, Infrared and Depth) are fed into Dual Desc network to generate robust attentive global descriptors and local features. The global descriptors are leveraged to retrieve the coarse candidates of query images. The 2D local features, as well as 3D sparse point cloud, are used in geometric verification to select the optimal results from the retrieved candidates. Finally, sequence matching robustifies the localization results by synthesizing the verified results of successive frames. The proposed unified descriptor network Dual Desc surpasses the state-of-the-art NetVLAD and its variant on the task of image description. Validated on the real-world dataset captured by the wearable assistive device, the proposed visual localization utilizes multimodal images to overcome the disadvantages of RGB images and robustifies the localization performance by deep descriptor network and hierarchical pipeline. In the challenging scenarios of the Yuquan dataset, the proposed method achieves the F1 score of 0.77 and the mean localization error of 2.75, which is satisfactory in practical use.

• 引言

定位是视障人帮助技术的关键题目之一。

• 针对题目


• 提出的方式

本文提出了一种新的基于可穿着帮助导航装备的分级视觉定位管道。该管道包括深层描写子收集、2D-3D多少考证和在线序列婚配。
分歧模态(RGB、红外和深度)的图像被输入到双描写收集Dual Desc中，天生鲁棒的留意全局描写符和部分特征。操纵全局描写符检索查询图像的粗候选图像。操纵二维部分特征和三维稀疏点云停止多少考证，从检索的候选成果当挑选最优成果。最初，序列婚配经过综合持续帧的考证成果对定位成果停止鲁棒化。提出的同一描写符收集Dual Desc在图像描写使命上超越了最早辈的NetVLAD及其变体。

• 尝试成果

在可穿着帮助装备收集的实在数据集上停止了考证，提出的视觉定位方式操纵多模态图像克服了RGB图像的弱点，并经过深度描写子收集和分层管道增强了定位性能。

单元：

Zhejiang University；University of Huddersfield

进献：

1. The unified descriptor network Dual Desc extracts both attentive global and 2D-3D local descriptors from RGB-D-IR images, which leverages the different advantages of the image modalities to generate the robust image description for visual localization with visual variations. 同一描写子收集Dual Desc从RGB-D-IR图像中提取留意全局描写子和2D-3D部分描写子，操纵图像模态的分歧上风天生鲁棒图像描写，用于视觉变化的视觉定位。
2. The proposed hierarchical visual localization pipeline involving image retrieval, geometric verification, and sequence matching makes full use of scene semantics and scene geometry of images and is validated to be superior in the scenario of assistive localization. 所提出的分层视觉定位pipeline 包括图像检索、多少考证和序列婚配，充实操纵了图像的场景语义和场景多少，在帮助定位场景中具有较好的结果。

论文内容：

摘要图：

MinkLoc3D计较查询点云的全局描写符。定位是经过在数据库中搜索具有比来描写符的地理标志点云来履行的。

整体结构

分支NetAVLAD生玉成局描写符用于位置检索，分支Dense Desc天生2D部分特征和3D点云用于多少考证。

分层视觉定位由位置检索、多少考证和序列婚配三个阶段组成

尝试

操纵该pipeline对Yuquan多模态(a)子集1和(b)子集2停止视觉定位的一些成果。查询的图像索引(左)、基于resnet的成果(中)和基于vgg的成果(右)在图像上方标志，正确/毛病的定位成果用绿色/红色暗示。

作者注：将视觉定位利用在新范畴也是很是好的idea