Developing an Automated Reliable Methodology for the Detection and Correction of Error within a Sanitary and Storm Water Data Collection and Processing System

流域径流监测数据被市政当局和工程公司广泛用作设计、调整、监测、控制和优化排水和排水系统的依据。因此，这些时间序列的准确性和完整性在水资源管理和工程中起着重要作用，因此，确保流量传感器和雨量计产生的数据准确和可靠至关重要。水文事件和相关数据集具有复杂的性质，各种误差源以及传感器故障或技术问题可能导致记录的数据中存在空白和不规则，从而限制生成数据集的使用。因此，有必要采用一种可靠、高效的算法来识别数据异常和纠正错误数据集。提出的方法旨在生成一种自动算法，通过使用统计和数学分析来识别和标记记录的数据序列中的错误。之后，将通过适用的水力和水文方程和/或统计估计来重建和替换有缺陷的数据。合作伙伴公司Civica基础设施公司开发了一个功能强大的基于网络的、专门构建的数据存储平台，名为DataCurrent，为收集、处理和分析时间序列分水岭数据提供了高效和可靠的方式。尽管DataCurrent的稳健性很强，但经常需要手动检查数据以识别并更正或删除不正确的数据。这是一个耗时且昂贵的过程。该项目的结果除了社会和环境方面的好处外，还将通过DataCurrent软件内的流量监测数据质量保证和控制(QA/QC)过程的自动化带来显著的好处，减少与手动数据更正程序相关的时间和成本，并增强收集的数据的质量和可靠性。

Watershed flow monitoring data are widely used by municipalities and engineering firms as the bases fordesigning, sizing, monitoring, controlling, and optimizing drainage and sewer systems. Hence, the correctnessand completeness of such time-series plays a significant role in water resources management and engineeringand as such it is crucial to ensure data generated from flow sensors and rain gauges are accurate and reliable.Hydrological events and associated datasets are of a complex nature and various sources of error along withsensors malfunctioning or technical issues may result in gaps and irregularities within logged data which willthen limit the use of generated datasets. Hence, it is necessity to adapt a reliable and efficient algorithm that canidentify data abnormalities and correct faulty datasets. The proposed methodology aims to generate anautomated algorithm that by employing statistical and mathematical analysis can identify and flag errors withinlogged data series. Following that, faulty data will be reconstructed and replaced by means of applicablehydraulic and hydrological equations and or statistical estimation. The partner company, Civica InfrastructureInc., has developed a powerful web-based, purpose-built data-storage platform named DataCurrent thatprovides efficient and reliable ways to collect, process and analyze time-series watershed data. DespiteDataCurrent's robustness, data is often manually inspected to identify and correct or remove incorrect data.This is a time consuming and costly process.The outcome of this project aside from its social andenvironmental gains will bring about significant benefits through automation of flow monitoring data qualityassurance and control (QA/QC) processes within the DataCurrent software, reduction of the time and costassociated with manual data correction procedures and enhancement of the quality and reliability of thecollected data.