STTR Phase I: A Self-Learning Approach for In-Vehicle Driver and Passenger Monitoring Through a Sensor Fusion Approach

这项小型企业技术转让（STTR）第一阶段项目的广泛影响将来自引入最先进的驾驶员监控系统，该系统使用人工智能来检测分心驾驶或不良驾驶行为。它还可以用于驾驶员培训和教育，以及提高驾驶员的注意力。该系统将有助于减少事故，创造更安全的道路和工作环境。最终用户包括汽车原始设备制造商（oem）、商业车队运营商、出租车和拼车公司、重型机械和起重机运营商、铁路和航空运营商，以及专门运输系统的运营商，如校车服务和包车。这项小型企业技术转移（STTR）第一阶段项目将利用车辆内不同摄像头和惯性传感器的数据来监测和评估驾驶员的注意力。将使用人工智能（AI）方法分别评估驾驶员的目光和上半身姿势，并将结果综合起来，得出驾驶员分心程度的总体估计。所提出的框架即使在高度面部遮挡的情况下也有望产生可靠的结果。该项目的技术目标包括：1)探索使用深度和RGB传感器跟踪驾驶员身体运动的监督和无监督方法，解决当前基于视觉的算法在现实驾驶条件下的挑战和缺陷；2)设计一种新颖的深度学习框架，将驾驶员的身体姿势和注意力水平结合起来，推断驾驶员的活动（例如，使用便携式设备、饮食等活动）；3)开发新的驾驶员视觉注意模型，获得估计驾驶员注视、估计肩部姿势和关节位置的置信度；4)开发集成多个传感器的多模态端到端深度学习框架，为监控和辅助驾驶员提供重要功能；5)在低功耗商用硬件上实现系统，具有成本效益和可扩展性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The broader impact of this Small Business Technology Transfer (STTR) Phase I project will result from the introduction of a state-of-the-art driver monitoring system using artificial intelligence to detect distracted driving or poor driving practices. It can also be used for driver coaching and education, as well as to improve driver attention. The system will help minimize accidents and create safer roads and work environments. End users include automotive original equipment manufacturers (OEMs), commercial fleet operators, taxi and ride-sharing companies, heavy machinery and crane operators, rail and aviation operators, and operators of specialized transportation systems, such as school bus services and charter vehicles.This Small Business Technology Transfer (STTR) Phase I project will exploit data from different camera and inertial sensors inside a vehicle to monitor and assess the attention of the driver. The driver’s gaze and upper body pose will be evaluated separately using artificial intelligence (AI) methods and the results combined to generate an overall estimate of the level of driver distraction. The proposed framework is expected to generate reliable results even in cases of high face occlusion. The technical objectives of the project include to: 1) Explore supervised and unsupervised methods to track the driver's body movement using depth and RGB sensors, addressing the challenges and drawbacks of current vision-based algorithms in real-world driving conditions; 2) Design a novel deep learning framework to integrate the driver's body pose with his/her attention level to infer driver's activities (e.g., such as using portable devices, eating, drinking, and other activities); 3) Develop new models of driver visual attention to obtain confidence levels in the estimated driver's gaze, estimated shoulder pose and joints positions; 4) Develop multi-modal end-to-end deep learning frameworks that integrate multiple sensors to provide important features for monitoring and assisting the driver; 5) Implement the system on low-power commodity hardware that is cost-effective and scalable.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.