Robust and Scalable On-Line NDP Designs and Applications to Semiconductor Process Optimization

0002098尽管今天在微处理器和控制系统设计领域取得了所有技术进步，但仍然缺少一个关键组件，即通用学习系统的设计（在提案中以后将其称为“学习器”）。学习者将以软件或硬件的形式获得最终产品，通过与环境的互动来学习提高其性能。除了缺乏明确的环境模型之外，明确的性能反馈也被延迟了，也就是说，它们只能在一长串动作和结果的最后才可用。这种性质的问题超出了经典自适应控制理论的范围。近几十年来，以基于神经动态规划（NDP）的强化学习（RL）为代表的新思想流派已经浮出水面。在这种方法中，学习者观察一个输入状态（可以是当前状态或预测的未来状态），然后产生一个“动作”或“控制”信号，以应用到环境中。因此，一个“评估”信号是由评论家网络创建的，对所采取行动的有效性进行评论。学习的目标是产生最佳的行为，从而获得最大的回报。分层神经网络是学习者的关键实现模块。神经网络同时提供动作信号和评估信号。学习者已经在许多困难的任务中证明了他们的有效性。然而，学习者通常是执行预测任务的神经网络，如生成动作值或动作评估值。当对环境或任务知之甚少时，学习者必须获得系统级知识，以首先产生行动，然后进行评估。这需要学习者内部的各个组成部分协同工作。此外，如何在不“作弊”的情况下，为不同的应用程序实现人机界面？这个项目将解决这些基本问题，从半导体制造的问题作为一个试验台。它将以数学学习算法的形式寻求可靠的系统设计。它将尝试从学习者那里获得更稳定和更快的结果，即以更少的学习试验获得更高的成功率。重点将放在配置、算法参数化、系统输入输出和性能测量规范，以及与学习器设计相关的所有其他问题。学习者将为工业规模的半导体制造设施开发输入释放和排队策略。本练习的目的是检查学习器设计的可伸缩性、可靠性和通用性。这项研究的成功实施将代表着迈向真正的类人系统的重要一步，该系统可以自主学习并随着时间的推移提高其性能。

0002098SiDespite all the technical advances today in the fields of micro-processors and control system design, one key component is still missing, the design of a generic learning system (which will be referred to as a 'learner' hereafter in the proposal). The learner will have a final product in the form of either software or hardware that learns to improve its performance through interactions with the environment. In addition to the lack of an explicit model for the environment, explicit performance feedbacks are delayed, i.e., they are only available at the end of a long sequence of actions and consequences. A problem of this nature is beyond the scope of classical adaptive control theory.In recent decades, new schools of thinking represented by Reinforcement Learning (RL) based on Neural Dynamic Programming (NDP) have surfaced. In this approach, the learner observes an input state (which can be the current state or a predicted future state) and then produces an 'action' or 'control' signal to apply back to the environment. Consequently, an 'evaluation' signal is created by a critic network to comment on the effectiveness of the action taken . The goal of learning is to generate optimal actions leading to a maximal reward. Layered neural networks are the key implementation blocks for the learner. Neural networks are used to provide both the action signal and the evaluation signal.Learners have demonstrated their effectiveness in a number of difficult tasks. However, learners are usually neural networks performing predictive tasks such as generating action values or action evaluation values. When little is known about the environment or the task, the learner must acquire a system level knowledge to first produce the action and then the evaluation. This requires the components inside the learner to work together. Furthermore, how can one implement a human-machine interface for different applications without 'cheating' by letting the learner truly learn on its own and 'on-the-fly'?This project will address these basic issues, using problems from semiconductor manufacturing as a testbed. It will seek reliable system designs in the form of mathematical learning algorithms. It will try to achieve more stability and quicker outcomes from the learner, namely higher success rates with fewer learning trials. Attention will be paid to the configuration, algorithm parameterization, system input-output and performance measure specification, and all other issues relevant to the learner design. The learner will develop input releases and queuing policies for an industrial scale semiconductor manufacturing facility. The purpose of this exercise is to examine the scalability, reliability, and generality of the learner design.A successful implementation of this research would represent a significant step toward a truly human-like system that learns on its own and improves its performance over time.