ATD: Activity Aware Bayesian Deep Learning

ATD：活动感知贝叶斯深度学习

• 批准号: 2319470
• 负责人: William Basener
• 金额: $ 9万
• 依托单位: University of Virginia Main Campus
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2319470&HistoricalAwards=false
• 关键词: ATD; Activity; Aware; Bayesian; Deep

This project will research next-generation deep learning AI models for comprehending objects, activity, and context from overhead imagery and sensor data. This work will use Large Language Models (LLMs) to create machine reasoning that replicates human reasoning at a scale, speed, and complexity unachievable with human analysts. Different types of data collected from satellites and aircraft are currently processed separately, leading to siloed information without context. This research will use LLMs to synthesize multiple data sources with context, location, and time, producing Activity-Aware Deep Learning AI. Current deep learning AI can take pixel-based information in images to object-based (groups of pixels) information, and the current state-of-the-art scene-based (groups of objects) information. This project will enable a new level of activity-based information: what are the objects doing in the scene? what is the broader context? For example, a blue tarp in a US suburb is likely covering an object to protect it from weather, but multiple blue tarps along streets following a natural disaster may be indications of people in makeshift shelters in need of help. The Activity-Aware DL models developed research will comprehend these different situations using LLMs, with no activity-specific training beyond the logic already present in the LLMs. Software developed will be made available as open source, and new editions of textbooks on the area written by the investigator will be released. There are existing models for determining object-based information from sensor data. For example, convolutional neural networks can identify objects in high resolution imagery, and Bayesian models can accurately identify chemical species present on the ground in hyperspectral imagery. Even simple prompts into LLMs including just objects and location can produce activity-aware information. For example, the text prompt “Why are there rows of XYZ military vehicles []?” where [] can be filled in with “outside Location A”, “In Location B”, or “at Location C” will produce different conclusions when put into the ChatGPT LLM without explicit context training. The current project will develop neural network architectures that can translate ‘what is present’ class probabilities from an object-based model, combine this with geospatial information, and generate a text prompt input into an LLM to determine activity and context. Particular attention will be focused on developing prompts that do not generate factually inaccurate output from the LLM. Models will be developed in TensorFlow software to facilitate sharing. Bayesian probabilities will be used throughout the models to provide regularization and interpretability of the models, facilitating future ongoing research in this area.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目将研究下一代深度学习人工智能模型，用于从头顶图像和传感器数据中理解对象、活动和上下文。这项工作将使用大型语言模型(LLM)来创建机器推理，以人类分析师无法实现的规模、速度和复杂性复制人类推理。从卫星和飞机收集的不同类型的数据目前是分开处理的，导致孤立的信息没有背景。这项研究将使用LLMS来综合具有上下文、位置和时间的多个数据源，产生活动感知的深度学习人工智能。目前的深度学习人工智能可以将图像中基于像素的信息转化为基于对象(像素组)的信息，以及当前最先进的基于场景(对象组)的信息。这个项目将使基于活动的信息达到一个新的水平：对象在场景中做什么？更广泛的背景是什么？例如，美国郊区的一块蓝色防水布很可能覆盖着一个物体，以保护其免受天气影响，但自然灾害后街道上的多块蓝色防水布可能表明有人在临时避难所里需要帮助。开发的活动感知学习模型的研究将使用LLMS来理解这些不同的情况，除了LLMS中已经存在的逻辑之外，没有专门针对活动的培训。开发的软件将作为开放源码提供，并将发布由调查员编写的关于该地区的新版教科书。现有的模型可以从传感器数据中确定基于对象的信息。例如，卷积神经网络可以在高分辨率图像中识别目标，贝叶斯模型可以在高光谱图像中准确识别地面上存在的化学物质。即使是对LLM的简单提示，只包括对象和位置，也可以产生活动感知信息。例如，文本提示“为什么有一排排XYZ军用车辆[]？”其中[]可以填充“位置A之外”、“位置B”或“位置C”，如果放入ChatGPT LLM，在没有明确的上下文训练的情况下，会产生不同的结论。目前的项目将开发神经网络结构，该结构可以从基于对象的模型中转换“当前是什么”的类别概率，将其与地理空间信息相结合，并将文本提示输入到LLM中以确定活动和上下文。将特别注意开发不会从LLM生成事实不准确的输出的提示。模型将在TensorFlow软件中开发，以便于共享。贝叶斯概率将在整个模型中使用，以提供模型的正规化和可解释性，促进未来在该领域的正在进行的研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。