Uncheatable Benchmarks

点击翻译按钮获取中文摘要

Benchmarks have been used to test everything from the speed of a processor to the access time of a memory system. The computing community relies on them to assess how fast a given hardware or software system operates. They are of importance in everyday computing. However, until now, the study of the art of designing a good benchmark has focused on making the benchmark ``realistic'' in predicting how well it will perform for the intended applications; the issue of making benchmark results trustworthy has been relegated to ``trusted'' or third party agents, and little attention has been paid to the question of making benchmarks themselves ``uncheatable.'' The project studies the problem of how to make benchmarks resistant to tampering and hence more trustworthy. The plan is to use modern cryptography and complexity theory to make this possible. Basically the trust in individuals and organizations will be replaced by trust in the impossibility of breaking certain computational problems. The advantage of this approach is clear: the schemes will be scientifically trustworthy. There is an array of problems that need to be formulated and solved, which are important in making uncheatable benchmarks; some are practical issues, some are theoretical.