Post by SongRune on 2022-06-13 03:41:29
I found this concept super interesting. I hadn't heard of this particular Reinforcement Learning technique, so I went to go learn about it via your linked source. My problem with it, fundamentally, as I went through was that I kept looking for "Okay, but how do you feasibly apply long-term reward outcomes back to long-prior decisions that lead to them?". I did find my answer eventually, in footnote two, and though it wasn't what I had hoped, it was what I was expecting. Unfortunately, it has some strong implications for the use of this technique in an MMO Boss Combat AI environment.
So let's go over requirements: To function in an MMO Boss AI context, such a system needs:
- To be able to handle all or most group compositions without falling flat.
- To be able to be applied to multiple enemy AIs in the same encounter, and manage to learn individual tactics, teamwork, the ability to handle losing various combinations of allies, or various disruptions to positioning.
- To be able to be applied to/trained for many different bosses in many different situations. MMOs aren't small, and tend to need to have, or develop, a lot[ of content. Even if there are fewer at launch, "several thousand unique bosses" is not out of the question for a mature MMO.
The first two requirements I'm willing to concede. If you can play Dota, you can probably pull those off. Unfortunately...
It's not economically feasible to provide a significant, high-performance compute cluster for every or even most major bosses in a game at Ashes' scale. It's even less feasible to provide this cluster for 10 months for each boss encounter, or even type or style of boss encounter in the game. When you add to this that a boss has to be finished and balanced for the AI to really be training with it properly (as minor to moderate changes in design can sometimes create or invalidate tactics), you simply can't use this technique effectively for the scale of content you would find in this type of game.
Tuning the difficulty of the AI is another special consideration as well. Simple stat adjustments (even on a pure "uniform scaling" level) can have the effect of rebalancing what risks are advantageous to take against a given set of opponents, requiring the AI to relearn possibility-spaces that it has already trained on. Performance does not simply scale linearly with statistics in all cases with a pre-trained AI. Add to this the fact that these sorts of effects would apply differently (sometimes negligible, and sometimes devastating) to different player team compositions and tactics, it become even more difficult to use a simplistic scaling method without allowing the boss to be 'cheesed'. And if you find a flaw and need to rebalance or rethink some aspect of a boss? Back to the cluster for a few more months of training.
I would find this sort of thing REALLY COOL, but I don't expect to get it.
So let's go over requirements: To function in an MMO Boss AI context, such a system needs:
- To be able to handle all or most group compositions without falling flat.
- To be able to be applied to multiple enemy AIs in the same encounter, and manage to learn individual tactics, teamwork, the ability to handle losing various combinations of allies, or various disruptions to positioning.
- To be able to be applied to/trained for many different bosses in many different situations. MMOs aren't small, and tend to need to have, or develop, a lot[ of content. Even if there are fewer at launch, "several thousand unique bosses" is not out of the question for a mature MMO.
We were unable to determine the author of the quoted post.
The first two requirements I'm willing to concede. If you can play Dota, you can probably pull those off. Unfortunately...
It's not economically feasible to provide a significant, high-performance compute cluster for every or even most major bosses in a game at Ashes' scale. It's even less feasible to provide this cluster for 10 months for each boss encounter, or even type or style of boss encounter in the game. When you add to this that a boss has to be finished and balanced for the AI to really be training with it properly (as minor to moderate changes in design can sometimes create or invalidate tactics), you simply can't use this technique effectively for the scale of content you would find in this type of game.
Tuning the difficulty of the AI is another special consideration as well. Simple stat adjustments (even on a pure "uniform scaling" level) can have the effect of rebalancing what risks are advantageous to take against a given set of opponents, requiring the AI to relearn possibility-spaces that it has already trained on. Performance does not simply scale linearly with statistics in all cases with a pre-trained AI. Add to this the fact that these sorts of effects would apply differently (sometimes negligible, and sometimes devastating) to different player team compositions and tactics, it become even more difficult to use a simplistic scaling method without allowing the boss to be 'cheesed'. And if you find a flaw and need to rebalance or rethink some aspect of a boss? Back to the cluster for a few more months of training.
I would find this sort of thing REALLY COOL, but I don't expect to get it.