SGER Proposal: A "Transderivational" Search Engine for Creative Analogy Generation in Mixed-Media Design

该项目将开发一个基于联觉的跨导搜索引擎，联觉是指两种或多种感官交叉（例如，当看到一种颜色导致一个人听到一种声音），以帮助人们发现文本、1D音频、2D图像、3D几何和4D运动数据之间的联系。该项目的灵感来自艺术家和设计师的能力，他们能够在不同的文物之间找到相似之处，并将它们组合在一起，形成一个连贯而新颖的叙事。这项研究的智力价值在于匹配算法的发展，该算法通过观察媒体内容的结构相似性来建议不同媒体形式之间的类比。其结果将是艺术、计算机图形学、机器学习、认知心理学和人机交互（HCI）交叉的变革性技术。跨衍生搜索将增强类比生成中的联觉效应，并将自然地使其成为广泛的头脑风暴追求。在不同形式的媒体（例如，音频和3D形状）之间寻找类比还没有被探索，也没有太多关注非文字搜索引擎。字面搜索仅依赖于明确的含义(例如，单词three？并对图像进行编号(3)和分类，以确定相似性。相反，该项目将通过使用统计形状分布、频率分析和机器学习技术等分析方法来寻找结构相似性来比较媒体样本，以发现混合（多维）媒体样本之间的关系。这项研究的更广泛的影响是通过跨导搜索（对语言和认知处理至关重要）推进人工智能，并在搜索技术上开辟了新的研究问题。通过展示搜索技术与创意设计和多媒体管理的相关性，教育方面的影响是吸引更多的女性和少数族裔进入计算机科学，并提高计算机科学课程的保留率。在计算机科学入门级课程中，学生将使用转导数搜索工具来构建基本的媒体管理软件。在机器学习、计算机视觉和图形学的高级CS课程中，转导数搜索也可以作为探索算法的测试平台。

This project will develop a transderivational search engine based on the neurological condition known as synaesthesia in which two or more senses are crossed (e.g., when seeing a color causes one to hear a sound) to help people to discover connections between text, 1D audio, 2D image, 3D geometry and 4D motion data. The project is inspired by the ability of artists and designers to find analogies between diverse artifacts and bring them together to compose a coherent and novel narrative.The intellectual merit of this research is the development of matching algorithms that suggest analogies across different media forms by looking at structural similarity within media content. The result will be a transformative technology at the intersection of art, computer graphics, machine learning, cognitive psychology, and human-computer interaction (HCI). Transderivational search will enhance the synaesthetic effect in analogy generation and will naturally lend itself to a wide range of brainstorming pursuits. Finding analogies between media of different forms (e.g., audio and 3D shapes) has not been explored, nor has there been much focus on non-literal search engines. Literal searches rely only on explicit meaning (e.g., the word ?three? and an image of the number 3) and categorization to determine similarity. Instead, this project will compare media samples by looking for structural similarity using analytical approaches such as statistical shape distributions, frequency analysis, and machine learning techniques to discover relationships between mixed- (multi-dimensional) media samples.The broader impacts of this research are in advancing artificial intelligence through transderivational search (essential to language and cognitive processing) and in opening up new research questions on search technology. The educational impacts are in drawing more women and minorities into CS and improving retention in CS programs by showing the relevance of search technology to creative design and to multimedia management. The transderivational search tools will be used by students in introductory level CS courses to build basic media management software. Transderivational search can also serve as a testbed for exploring algorithms in high level CS courses on machine learning, computer vision and graphics.