Modeling rich inter-image relationships in big visual collections

社会、行为和经济科学司提供博士后研究奖学金，为最近的博士毕业生提供获得额外培训的机会，在知名科学家的赞助下获得研究经验，并扩大他们在本科和研究生训练之外的科学视野。博士后奖学金的进一步设计是为了帮助新科学家指导他们跨越传统学科线的研究工作，并利用他们自己独特的研究资源、地点和设施，包括在国外的研究地点。该博士后奖学金支持在交叉领域交叉计算机视觉和心理学领域崭露头角的科学家，其研究项目调查人类和机器视觉数据之间的关系网络。对于一个人类观察者来说，没有一张照片是一座孤岛：它通过一个由相似性、联想和其他关系组成的网络与视觉世界的其他部分相连。例如，两张巴黎的照片有一定的相似之处；船的图像与水的图像联系在一起；一张蝌蚪的照片和一张青蛙的照片显示了同一生物在生命的两个阶段。在每一种情况下，人类都可以很容易地推断出两幅图像之间的联系。人们不仅可以识别一种关系的存在，还可以识别这种关系的性质。这些关系揭示了人类大脑如何组织视觉信息，也为如何建立自动建立视觉连接的智能系统提供了见解。后者将使该领域更接近于产生一个智能视觉网络，能够以与当前互联网能够组织文本相同的方式组织视觉信息。计算机视觉科学家和心理学家都研究过视觉数据之间的关系，但研究方向不同。在计算机视觉中，重点一直放在自然图像相似度模型上。这些模型处理复杂的刺激，但通常仅限于一种简单的关系，即外表上的相似性。心理学家已经研究了一套更丰富的关系——关联、因果、类比、反义词、转换等等——但他们的模型通常只适用于简单的、人为的刺激。这个项目通过在复杂、自然的图像之间建立微妙的视觉关系，将这两个领域的优势结合起来。目标是对人类认为相关的图像以及这些图像如何相关进行建模。另一个目标是研究某些关系，如视觉关联，如何以无监督的方式从自然视觉经验中产生。这将有助于解释人类最初是如何了解这些关系的。更好的意象间关系模型将对认知心理学产生深远的影响。特别是相似性和联想在人类学习和记忆理论中起着重要的作用。相似感使我们能够从一次视觉体验中学习，然后将我们的知识应用到未来类似的环境中。从经历中产生的联想还会影响人类对它的记忆。本项目也应用于计算机视觉系统。反向图像搜索最近成为一种流行的工具。然而，目前的系统只能检索相似的图像。如果计算机能够检索由更多种关系联系在一起的图像，就会有很多可能性。例如，可以想象一个系统可以让用户浏览艺术风格，或者推荐与裤子相配的鞋子。如果成功的话，这个项目将为世界范围内的视觉连接网络铺平道路，与当前的超文本连接网络平行。

The Directorate of Social, Behavioral and Economic Sciences offers postdoctoral research fellowships to provide opportunities for recent doctoral graduates to obtain additional training, to gain research experience under the sponsorship of established scientists, and to broaden their scientific horizons beyond their undergraduate and graduate training. Postdoctoral fellowships are further designed to assist new scientists to direct their research efforts across traditional disciplinary lines and to avail themselves of unique research resources, sites, and facilities, including at foreign locations. This postdoctoral fellowship supports a rising scientist in the interdisciplinary area overlapping computer vision and psychology, with a research project that investigates the web of relationships within visual data in both humans and machines. To a human observer, no photograph is an island: it is connected to the rest of the visual world by a web of similarities, associations, and other relationships. For example, two photos of Paris share a certain similarity; images of boats are associated with images of water; a photo of a tadpole and a photo of a frog show the same organism at two stages of life. In each of these cases, a human can readily reason about the link between two images. Not only can people identify that a relationship exists, but can also identify the nature of this relationship. These relationships shed light on how the human brain organizes visual information, and also give insight into how to build intelligent systems that automatically make visual connections. The latter will bring the field closer to producing an intelligent visual web, able to organize visual information in the same way as the current Internet is able to organize text. Computer vision scientists and psychologists have both studied relationships between visual data, but from different directions. In computer vision, the focus has been on models of natural image similarity. These models handle complex stimuli but are usually limited to one simple kind of relationship, namely similarity in appearance. Psychologists have studied a richer set of relationships - association, causation, analogy, antonymy, transformation, etc. - but their models usually only apply to simple, artificial stimuli. This project unites the best of both fields by modeling subtle visual relationships between complex, natural images. The objective is to model both which images humans consider to be related and how are those images related. An additional objective is to study how certain relationships, such as visual associations, can arise in an unsupervised manner from natural visual experience. This will help explain how humans might learn about the relationships in the first place. Better models of inter-image relationships will have deep implications across cognitive psychology. In particular, similarity and association play fundamental roles in theories of human learning and memory. A sense of similarity underlies our ability to learn from one visual experience and then apply our knowledge in a future, similar setting. The associations made from the experience additionally impact human memory of it. The present project also has applications toward computer vision systems. Reverse image search has recently become a popular tool. However, current systems are only able to retrieve look-alike images. If the computer is instead able to retrieve images linked by more diverse kinds of relationships, many possibilities open up. For example, one could imagine a system that lets users navigate through artistic styles, or that recommends shoes that match a pair of pants. If successful, this project could pave the way toward a world-wide web of visual connections that parallels the current web of hypertext connections.