TY - GEN
T1 - Efficient statistical methods for evaluating trading agent performance
AU - Sodomka, Eric
AU - Collins, John
AU - Gini, Maria L
PY - 2007
Y1 - 2007
N2 - Market simulations, like their real-world counterparts, are typically domains of high complexity, high variability, and incomplete information. The performance of autonomous agents in these markets depends both upon the strategies of their opponents and on various market conditions, such as supply and demand. Because the space for possible strategies and market conditions is very large, empirical analysis in these domains becomes exceedingly difficult. Researchers who wish to evaluate their agents must run many test games across multiple opponent sets and market conditions to verify that agent performance has actually improved. Our approach is to improve the statistical power of market simulation experiments by controlling their complexity, thereby creating an environment more conducive to structured agent testing and analysis. We develop a tool that controls variability across games in one such market environment, the Trading Agent Competition for Supply Chain Management (TAC SCM), and demonstrate how it provides an efficient, systematic method for TAC SCM researchers to analyze agent performance.
AB - Market simulations, like their real-world counterparts, are typically domains of high complexity, high variability, and incomplete information. The performance of autonomous agents in these markets depends both upon the strategies of their opponents and on various market conditions, such as supply and demand. Because the space for possible strategies and market conditions is very large, empirical analysis in these domains becomes exceedingly difficult. Researchers who wish to evaluate their agents must run many test games across multiple opponent sets and market conditions to verify that agent performance has actually improved. Our approach is to improve the statistical power of market simulation experiments by controlling their complexity, thereby creating an environment more conducive to structured agent testing and analysis. We develop a tool that controls variability across games in one such market environment, the Trading Agent Competition for Supply Chain Management (TAC SCM), and demonstrate how it provides an efficient, systematic method for TAC SCM researchers to analyze agent performance.
UR - http://www.scopus.com/inward/record.url?scp=36348990175&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=36348990175&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:36348990175
SN - 1577353234
SN - 9781577353232
T3 - Proceedings of the National Conference on Artificial Intelligence
SP - 770
EP - 775
BT - AAAI-07/IAAI-07 Proceedings
T2 - AAAI-07/IAAI-07 Proceedings: 22nd AAAI Conference on Artificial Intelligence and the 19th Innovative Applications of Artificial Intelligence Conference
Y2 - 22 July 2007 through 26 July 2007
ER -