ATD: Activity Aware Bayesian Deep Learning

该项目将研究下一代深度学习AI模型，用于从高空图像和传感器数据中理解对象，活动和上下文。 这项工作将使用大型语言模型（LLM）来创建机器推理，以人类分析师无法实现的规模，速度和复杂性复制人类推理。 从卫星和飞机收集的不同类型的数据目前分别处理，导致信息孤立，没有背景。这项研究将使用LLM来合成具有上下文、位置和时间的多个数据源，从而产生活动感知深度学习AI。 当前的深度学习AI可以将图像中基于像素的信息转化为基于对象（像素组）的信息，以及当前最先进的基于场景（对象组）的信息。 这个项目将使基于活动的信息达到一个新的水平：场景中的对象在做什么？更广泛的背景是什么？例如，美国郊区的蓝色防水布可能覆盖了一个物体，以保护它免受天气的影响，但自然灾害发生后，街道上沿着的多个蓝色防水布可能表明人们在临时避难所需要帮助。研究开发的活动感知DL模型将使用LLM来理解这些不同的情况，除了LLM中已经存在的逻辑之外，没有特定于活动的培训。所开发的软件将作为开放源码提供，调查员编写的关于该领域的新版教科书也将发行。 存在用于从传感器数据确定基于对象的信息的现有模型。 例如，卷积神经网络可以识别高分辨率图像中的物体，贝叶斯模型可以准确识别高光谱图像中地面上存在的化学物种。 即使是简单的LLM提示（仅包括对象和位置）也可以产生活动感知信息。 例如，文本提示“为什么有几排XYZ军用车辆[]？”其中[]可以用“在位置A之外”、“在位置B内”或“在位置C处”填充，在没有显式上下文训练的情况下放入ChatGPT LLM时会产生不同的结论。目前的项目将开发神经网络架构，该架构可以从基于对象的模型中转换“当前存在的”类概率，将其与地理空间信息联合收割机结合起来，并生成文本提示输入到LLM中以确定活动和上下文。特别注意将集中在开发提示，不产生事实上不准确的LLM输出。 模型将在TensorFlow软件中开发，以促进共享。贝叶斯概率将被用于整个模型，以提供模型的正规化和可解释性，促进未来在这一领域正在进行的研究。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

This project will research next-generation deep learning AI models for comprehending objects, activity, and context from overhead imagery and sensor data. This work will use Large Language Models (LLMs) to create machine reasoning that replicates human reasoning at a scale, speed, and complexity unachievable with human analysts. Different types of data collected from satellites and aircraft are currently processed separately, leading to siloed information without context. This research will use LLMs to synthesize multiple data sources with context, location, and time, producing Activity-Aware Deep Learning AI. Current deep learning AI can take pixel-based information in images to object-based (groups of pixels) information, and the current state-of-the-art scene-based (groups of objects) information. This project will enable a new level of activity-based information: what are the objects doing in the scene? what is the broader context? For example, a blue tarp in a US suburb is likely covering an object to protect it from weather, but multiple blue tarps along streets following a natural disaster may be indications of people in makeshift shelters in need of help. The Activity-Aware DL models developed research will comprehend these different situations using LLMs, with no activity-specific training beyond the logic already present in the LLMs. Software developed will be made available as open source, and new editions of textbooks on the area written by the investigator will be released. There are existing models for determining object-based information from sensor data. For example, convolutional neural networks can identify objects in high resolution imagery, and Bayesian models can accurately identify chemical species present on the ground in hyperspectral imagery. Even simple prompts into LLMs including just objects and location can produce activity-aware information. For example, the text prompt “Why are there rows of XYZ military vehicles []?” where [] can be filled in with “outside Location A”, “In Location B”, or “at Location C” will produce different conclusions when put into the ChatGPT LLM without explicit context training. The current project will develop neural network architectures that can translate ‘what is present’ class probabilities from an object-based model, combine this with geospatial information, and generate a text prompt input into an LLM to determine activity and context. Particular attention will be focused on developing prompts that do not generate factually inaccurate output from the LLM. Models will be developed in TensorFlow software to facilitate sharing. Bayesian probabilities will be used throughout the models to provide regularization and interpretability of the models, facilitating future ongoing research in this area.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.