Developing an Automated Reliable Methodology for the Detection and Correction of Error within a Sanitary and Storm Water Data Collection and Processing System

流域流量监测数据被市政当局和工程公司广泛用作设计、确定规模、监测、控制和优化排水和下水道系统的基础。因此，我们认为，时间的正确性和完整性-系列在水资源管理和工程中起着重要作用，因此确保流量传感器和雨量计生成的数据准确可靠至关重要。水文事件和相关数据集具有复杂的性质，各种误差来源沿着传感器故障或技术问题可能导致记录数据中的差距和不规则性，从而限制使用生成的数据集。因此，有必要采用一种可靠有效的算法来识别数据异常并纠正错误数据集。所提出的方法的目的是生成一个自动化的算法，通过采用统计和数学分析，可以识别和标记内记录的数据系列中的错误。在此基础上，通过适用的水力学和水文学方程和/或统计估计来重建和替换错误的数据。合作伙伴公司Civica StructureInc.开发了一个功能强大的基于Web的专用数据存储平台DataCurrent，为收集、处理和分析时间序列流域数据提供了高效可靠的方法。尽管DataCurrent的健壮性，数据往往是手动检查，以确定和纠正或删除不正确的数据。这是一个耗时和昂贵的过程。该项目的结果，除了其社会和有益的心理收益将带来显着的好处，通过自动化的流量监测数据质量保证和控制（QA/QC）过程中的DataCurrent软件，减少与人工数据校正程序相关的时间和成本，提高收集数据的质量和可靠性。

Watershed flow monitoring data are widely used by municipalities and engineering firms as the bases fordesigning, sizing, monitoring, controlling, and optimizing drainage and sewer systems. Hence, the correctnessand completeness of such time-series plays a significant role in water resources management and engineeringand as such it is crucial to ensure data generated from flow sensors and rain gauges are accurate and reliable.Hydrological events and associated datasets are of a complex nature and various sources of error along withsensors malfunctioning or technical issues may result in gaps and irregularities within logged data which willthen limit the use of generated datasets. Hence, it is necessity to adapt a reliable and efficient algorithm that canidentify data abnormalities and correct faulty datasets. The proposed methodology aims to generate anautomated algorithm that by employing statistical and mathematical analysis can identify and flag errors withinlogged data series. Following that, faulty data will be reconstructed and replaced by means of applicablehydraulic and hydrological equations and or statistical estimation. The partner company, Civica InfrastructureInc., has developed a powerful web-based, purpose-built data-storage platform named DataCurrent thatprovides efficient and reliable ways to collect, process and analyze time-series watershed data. DespiteDataCurrent's robustness, data is often manually inspected to identify and correct or remove incorrect data.This is a time consuming and costly process.The outcome of this project aside from its social andenvironmental gains will bring about significant benefits through automation of flow monitoring data qualityassurance and control (QA/QC) processes within the DataCurrent software, reduction of the time and costassociated with manual data correction procedures and enhancement of the quality and reliability of thecollected data.