Mitigation Strategies to Improve Reproducibility of Poverty Estimations From Remote Sensing Images Using Deep Learning
Mitigation Strategies to Improve Reproducibility of Poverty Estimations From Remote Sensing Images Using Deep Learning
复制标题
DOI：
10.1029/2022ea002379
复制
发表时间：
2022-08-01
影响因子：
3.1
通讯作者：
Mouillot, D.
中科院分区：
文献类型：
--
作者：
Machicao, J.;Ben Abbes, A.;Mouillot, D.
在过去的十年里，随着坚持良好研究实践的努力增加，计算机科学实验中的可重复性和可复制性(R&R)的挑战已经成为人们关注的焦点。然而，由于使用的技术复杂，使用深度学习(DL)的实验仍然很难重现。从遥感图像估算贫困指标(例如，财富指数水平)等挑战需要使用不同地理位置的大量数据，如果不使用数字图书馆技术，这些挑战是不可能实现的。为了检验数字视觉实验的重复性，我们对三个数字视觉实验的重复性进行了回顾，这些实验分析了卫星图像和街道图像中的视觉指标。对于每个实验，我们确定了在使用的数据集、方法和工作流程中发现的挑战。作为这一评估的结果，我们提出了一份包含相关公平原则的清单，以筛选实验的重复性。根据从这项研究中吸取的经验教训，我们建议采取一系列行动，旨在提高此类实验的重复性，减少浪费努力的可能性。我们认为，目标受众是广泛的，从寻求复制实验的研究人员，报告实验的作者，或寻求评估他人工作的评论家。
The challenges of Reproducibility and Replicability (R & R) in computer science experiments have become a focus of attention in the last decade, as efforts to adhere to good research practices have increased. However, experiments using Deep Learning (DL) remain difficult to reproduce due to the complexity of the techniques used. Challenges such as estimating poverty indicators (e.g., wealth index levels) from remote sensing imagery, requiring the use of huge volumes of data across different geographic locations, would be impossible without the use of DL technology. To test the reproducibility of DL experiments, we report a review of the reproducibility of three DL experiments which analyze visual indicators from satellite and street imagery. For each experiment, we identify the challenges found in the data sets, methods and workflows used. As a result of this assessment we propose a checklist incorporating relevant FAIR principles to screen an experiment for its reproducibility. Based on the lessons learned from this study, we recommend a set of actions aimed to improve the reproducibility of such experiments and reduce the likelihood of wasted effort. We believe that the target audience is broad, from researchers seeking to reproduce an experiment, authors reporting an experiment, or reviewers seeking to assess the work of others.