Skip to main content

Indie game storeFree gamesFun gamesHorror games
Game developmentAssetsComics
SalesBundles
Jobs
TagsGame Engines
(+1)

Thanks for the feedback.

I hadn’t even considered a tutorial. Now, the need for one seems quite obvious.

Originally, the game was intended to be about navigating through crowds in busy spaces. But along the way, I realized that simply creating levels and watching the agents navigate them was much more fun. You correctly identified the issue: there’s no logic to the agents' movements, which makes it uninteresting. I removed the "wait" button because it made the game too easy. I’m not sure how to make this particular mode engaging.

How quickly the agents learn the correct path is random. They might learn it within the first 30 seconds, or they might never learn it at all.

To understand what follows, it’s helpful to know how the algorithm works. When an agent receives a reward, it associates its previous actions with the total reward value it eventually earned. For instance, if an agent starts in state S, takes 3 steps, and receives a +1 reward at each step, it will store a value of approximately 3 for the actions it took. The next time it finds itself in state S, it will "know" that taking one of its actions leads to a reward of 3. I’m not sure if that’s a clear explanation, but it’s the best I could come up with (assuming I’ve understood how the algorithm works myself, of course).

This happens because their "vision" is limited to their immediate surroundings - just one cell in any direction. Consequently, they might reach some very strange conclusions about which action to take in a given cell to minimize the likelihood of future problems. You noticed this while observing the orange agents. The orange and red agents aren't hard-coded. Rather, they receive negative rewards. For example, the red agents receive +1 for reaching the finish line (as does everyone else) and -1/500 for every wall or other agent they come into contact with. The orange agents receive -1/500 for every obstacle they encounter. In theory, the red agents could choose a path without walls if they deemed it more advantageous—for instance, if the path containing walls were blocked by other agents, such as the blue ones. However, since there is no penalty for taking a longer route, they generally choose to head toward a wall. This is where the orange agents employ a clever tactic: an orange agent (let’s call it Agent A) avoids moving toward the reward, as other agents might already be there - and since its field of vision is limited to the single cell it occupies and those immediately surrounding it, it cannot know whether the path ahead is clear. Consequently, it waits until another agent (Agent B) happens to approach it. At that point, remaining stationary is no longer advantageous—as it incurs a penalty of -1/500 per turn—so the agent chooses to move elsewhere. This becomes especially apparent when randomness is disabled: the agents end up forming a grid-like pattern. Just take a look:

Furthermore, they are prepared to wait indefinitely; there is no penalty for standing still, whereas moving - which would inevitably lead to a collision with another agent - does carry a penalty.

The algorithm has a specific quirk: it lacks the concept of a "similar state." It perceives every state as entirely distinct from all others. For example, if an agent is positioned right next to the finish line and has, on previous occasions, chosen an action that successfully led it to the finish, it will unhesitatingly choose that same action again. But if another agent happens to stop nearby, the first agent no longer knows how to proceed - it has never encountered this specific situation before, even though the action that would lead it to the finish line remains perfectly viable. And now, it needs to choose the same action a sufficient number of times - but in a new situation - which will require many (possibly a great many) attempts. Yet, there are 256 such possible states for every single cell. This is the reason why their field of vision is limited to just one cell. If they could see even two cells away, the number of possible states at each point would be 16,777,216 (2^24).

All tabular reinforcement learning (RL) methods follow this pattern, but there is also "Deep RL," which utilizes neural networks. You may have heard that this approach has been used to train agents to play games like Dota 2, Doom, and others. However, such algorithms are extremely resource-intensive. Integrating them into games featuring a large number of agents would simply overwhelm the player's computer. Furthermore, there is another reason why these algorithms are only applicable in very rare cases within game development: they are entirely unpredictable. I could never have foreseen that the orange agents would come up with the strategies they did. Imagine you are a game designer who wants enemies to attack the player - for instance, by rewarding them for moving in a chain or a single-file wall (like a shield wall), while still allowing them to choose a different tactic if forming that wall would lead to certain death. At the same time, these agents are also rewarded for dealing damage to the player, healing allies, or moving as a group rather than individually. In this scenario, the agents might simply ignore the reward for moving in a chain as it’s more efficient to just rush the player and attack, as that’s faster than wasting time on formation. If you increase the reward for the chain formation, the agents might stop attacking the player altogether; they might just form a chain, break it, and form it again, or perhaps build chains of an absurd, massive size. Moreover, every time you adjust the reward structure, you have to retrain the agents from scratch, a process that could take hours. On top of that, more complex algorithms involve a multitude of parameters (such as epsilon, alpha, and gamma) that must be carefully tuned. Consequently, it is often much simpler, cheaper, and more effective to hand-code even the most complex behaviors rather than relying on reinforcement learning. The only instances where using reinforcement learning (RL) in games is justified are when a computer needs to control something that is simply impossible to hard-code (for example, entities that act like players (though I’ve never actually seen this done)) or in a specific genre of games where the core experience involves watching an agent learn (I have seen games like this).

However, the idea of using AI to replicate a player’s actions really appealed to me. Within the field of RL, there is a specific niche dedicated to algorithms designed to imitate behavior, yet I’ve never seen these applied in games, even though, from what I’ve gathered, they tend to be much more stable.

Thanks again for sharing your thoughts. I found them very interesting to read.