Robust and Scalable On-Line NDP Designs and Applications to Semiconductor Process Optimization

尽管今天在微处理器和控制系统设计领域取得了所有的技术进步，但仍然缺少一个关键组件，即通用学习系统的设计(以下将在提案中称为“学习者”)。学习者将拥有最终的软件或硬件形式的产品，学习通过与环境的交互来提高其性能。除了缺乏明确的环境模型外，明确的绩效反馈也被延迟，即只有在一长串行动和后果结束时才能得到反馈。这种性质的问题超出了经典自适应控制理论的范围。近几十年来，以神经动态规划(NDP)为基础的强化学习(RL)为代表的新思想流派层出不穷。在这种方法中，学习者观察输入状态(可以是当前状态，也可以是预测的未来状态)，然后产生“动作”或“控制”信号，以应用于环境。因此，评论家网络创建了一个‘评估’信号，以对所采取行动的有效性进行评论。学习的目标是产生最佳的行为，从而获得最大的回报。分层神经网络是学习者实现的关键模块。神经网络被用来提供行动信号和评估信号。学习者已经在一些困难的任务中证明了它们的有效性。然而，学习者通常是执行预测性任务的神经网络，例如生成动作值或动作评估值。当对环境或任务知之甚少时，学习者必须获得系统级别的知识，以便首先产生行动，然后进行评估。这需要学习者内部的组件协同工作。此外，如何通过让学习者真正的自主学习，在不作弊的情况下为不同的应用程序实现人机界面？这个项目将以半导体制造中的问题为试验台，解决这些基本问题。它将以数学学习算法的形式寻求可靠的系统设计。它将努力从学习者那里获得更稳定和更快的结果，即用更少的学习试验获得更高的成功率。将注意配置、算法参数、系统输入-输出和性能测量规范，以及与学习者设计相关的所有其他问题。学员将为工业规模的半导体制造设施开发投入释放和排队策略。本练习的目的是检查学习者设计的可扩展性、可靠性和通用性。成功地实施这项研究将代表着朝着真正的类似人类的系统迈出了重要的一步，该系统可以自主学习并随着时间的推移提高其性能。

0002098SiDespite all the technical advances today in the fields of micro-processors and control system design, one key component is still missing, the design of a generic learning system (which will be referred to as a 'learner' hereafter in the proposal). The learner will have a final product in the form of either software or hardware that learns to improve its performance through interactions with the environment. In addition to the lack of an explicit model for the environment, explicit performance feedbacks are delayed, i.e., they are only available at the end of a long sequence of actions and consequences. A problem of this nature is beyond the scope of classical adaptive control theory.In recent decades, new schools of thinking represented by Reinforcement Learning (RL) based on Neural Dynamic Programming (NDP) have surfaced. In this approach, the learner observes an input state (which can be the current state or a predicted future state) and then produces an 'action' or 'control' signal to apply back to the environment. Consequently, an 'evaluation' signal is created by a critic network to comment on the effectiveness of the action taken . The goal of learning is to generate optimal actions leading to a maximal reward. Layered neural networks are the key implementation blocks for the learner. Neural networks are used to provide both the action signal and the evaluation signal.Learners have demonstrated their effectiveness in a number of difficult tasks. However, learners are usually neural networks performing predictive tasks such as generating action values or action evaluation values. When little is known about the environment or the task, the learner must acquire a system level knowledge to first produce the action and then the evaluation. This requires the components inside the learner to work together. Furthermore, how can one implement a human-machine interface for different applications without 'cheating' by letting the learner truly learn on its own and 'on-the-fly'?This project will address these basic issues, using problems from semiconductor manufacturing as a testbed. It will seek reliable system designs in the form of mathematical learning algorithms. It will try to achieve more stability and quicker outcomes from the learner, namely higher success rates with fewer learning trials. Attention will be paid to the configuration, algorithm parameterization, system input-output and performance measure specification, and all other issues relevant to the learner design. The learner will develop input releases and queuing policies for an industrial scale semiconductor manufacturing facility. The purpose of this exercise is to examine the scalability, reliability, and generality of the learner design.A successful implementation of this research would represent a significant step toward a truly human-like system that learns on its own and improves its performance over time.