Uncheatable Benchmarks

基准测试已经被用来测试从处理器的速度到存储系统的访问时间的一切。计算界严重依赖它们来评估给定硬件或软件系统的运行速度。它们在日常计算中具有基本的重要性。然而，到目前为止，对设计一个好的基准的艺术的研究主要集中在使基准在预测其在预期应用中的表现方面“现实”；使基准结果可信的问题被归类到“可信的”或第三方代理，而很少有人关注使基准本身“不可欺骗”的问题。该项目研究的问题是如何使基准不易被篡改，从而更值得信任。该计划是使用现代密码学和复杂性理论来实现这一点。基本上，对个人和组织的信任被不可能打破某些计算问题的信任所取代。这种新方法的优势是显而易见的：新计划在科学上是值得信赖的。有一大堆问题需要制定和解决，这些问题对于制定不可欺骗的基准非常重要；有些是实际问题，有些是理论问题。

Benchmarks have been used to test everything from the speed of a processor to the access time of a memory system. The computing community relies on them heavily to assess how fast a given hardware or software system operates. They are of fundamental importance in everyday computing. However, up until now, the study of the art of designing a good benchmark has focused on making the benchmark ``realistic'' in predicting how well it will perform for the intended applications; the issue of making benchmark results trustworthy has been relegated to ``trusted'' or third party agents, and little attention has been paid to the question of making benchmarks themselves ``uncheatable.'' The project studies the problem of how to make benchmarks resistant to tampering and hence more trustworthy. The plan is to use modern cryptography and complexity theory to make this possible. Basically the trust in individuals and organizations is replaced by trust in the impossibility of breaking certain computational problems. The advantage of this new approach is clear: the new schemes will be scientifically trustworthy. There is a whole array of problems that need to be formulated and solved, which are important in making uncheatable benchmarks; some are practical issues, some are theoretical.