In my current alpha builds since version 0.07 I’ve been including some analytics. I decided to go ahead and review the data and see what I could learn from it. I’ll admit that it might be a bit early to start making sweeping changes to the gameplay due to the small sample size I have of roughly 15-20 players. I just really wanted to see what I could learn this early, and maybe later compare the small sample size to a larger test group. Without further ado, here are my findings (complete with boring charts).
The chart above shows 2 stats for each level played, the number of players to pass each level, and the number of times a player has failed at each level. So what can we see from this data? The red spikes tell me that users are having a difficult time passing certain levels, and an easier time at others. The steady decline in blue lines tells me two things. The first is that more players have started the game, than ones that have finished it. Secondly, that as each red spike occurs (challenging level) the number of players continuing to play drops a little bit. These are expected results though, honestly. The game is supposed to become more challenging, and not everyone is going to finish it. So really all this chart does is confirm what I expected, and tells me that I need to adjust the pacing of my challenging levels (also that level 74 is a beast). Onto chart two then…
This chart is to compare how many moves it took each player to reach a solution (average across all players) against the allowed maximum number of moves for each level. Why do I need this data? From this data I can see wether or not a level might be too easy or too hard. If everyone is beating level 23 well under the max then perhaps it should be lowered. If everyone is getting close to the limit on a particular level I can check the fail chart and use a combination of the data to determine that the maximum should be increased. The idea is to get the average as close to the limit as possible without causing a spike in failures.
If you’re asking why I’d want to close the gap between the two points, I’ll tell you. Have you ever won a race by mere inches, or perhaps beaten a timer by portions of a second? Like a near death experience, the “Near failure experience” causes tension, and excitement. Narrowly defeating an opponent releases more endorphins, and provides a greater sense of victory then simply crushing an enemy. Don’t get me wrong, there are times that beating a level by a huge margin makes you feel like a boss, but I’d like those kinds of wins to be more of a rare reward then accidental occurance. It all boils down to pacing the challenges of the game, and the rewards to the player.
ISSUES-
So what makes this data less reliable? As I stated earlier, such a small sample size means I could be looking at a series of edge cases and my target audience is nothing like the data here. Also, The data for moves-to-completion is built on an averaged move count. So if a player has a tough go of it, and consistently above average then they may end up failing alot. Some games will take this into account and adjust during gameplay, but (as far as I know) this is not common practice in a puzzle game. The other failing factor of this data is that it is collected during multiple versions of the game. While the levels and limits have not changed, the interaction or percieved interaction may have changed. Even that minor adjustment can give a player a different feeling about the game, which could encourage them to keep playing or to quit.
CONCLUSION-Lessons Learned
In the future I will need a larger user base for testing. I’ll also need to include a version number in the data collected. I’m not sure what other data I should be collecting, but perhaps I will also include a timing element to determine if my players are thinking through a level or simply swapping tiles at random. That is all for this weeks log, I’ll hopefully have something prettier and more exciting for next week. Turns out game development isn’t all rocket launchers and kitty cats.
