Using Hierarchical Machine Learning to Improve Player Satisfaction in a Soccer Videogame

This paper describes an approach to using a hierarchical machine learning model in a two player 3D physics-based soccer video game to improve human player satisfaction. Learning is accomplished at two layers to form a complete game-playing agent such that higher level strategy learning is dependent on lower-level learning of basic behaviors.Supervised learning is used to train neural networks on human data to model the basic behaviors. The reinforcement learning algorithms Sarsa (λ) and Q(λ) are used to learn overall strategies mapping game situations to these basic behaviors. We compare learning and non-learning agents and provide game results. Performance in self-play is analyzed to obtain a deeper understanding of the agent’s learning performance.Seventy people participated in a survey in which the learning agent led to a more dynamic and entertaining experience, while the non-learning agent was a slightly more difficult opponent.