Daniel Kahneman's "Thinking Fast and Slow" is all about how people are less rational than they think they are. When he first did this work, economists were using models of people who were perfectly rational to formulate their theories of how markets behave. Kahneman's contradictory work earned him the Nobel Prize in economics.
I think game designers are in a similar place now. We assume, with some evidence, that most of our players, given the opportunity, will minmax their way to a boring victory. While I agree that it's essential to avoid dominant strategies in your game, I believe that players do not act as rationally as we often think they would.
Here I list some cognitive biases and how they can be used in your game design as additional channels for information, additional sources of challenge or additional methods for manipulating your player.
1. Priming Slurp up this hearty example of priming: Fill in the missing letter: SO_P. You probably said 'U,' though you could also have said 'A.' The reason I think it's more probable that you said 'U' is that I used the word 'slurp' in the beginning, thus 'priming' your mind for things that are slurped, like soup.
Priming is a great covert way to adjust difficulty. Rather than making a challenge easier, try to prime for the solution somehow. It's a subtle distinction, because accepted methods of level design already ease the player into challenges by presenting less difficult but similar challenges beforehand. Priming is slightly different, though. Rather than introducing an easier version of the same challenge, you can introduce the type of thinking the player would need to complete the challenge. For example, you might want the player to aim for a boss' eyeball, and prime them with signs around the level saying "Safety Goggles Must Be Worn at All Times."
The other side of the coin is that you might want to make a challenge more difficult by priming the player for other solutions. The beauty of this is that the player may never pinpoint what it was that made the challenge seem so deceptively difficult. It can induce those "Why didn't I think of that before?!" moments.
2. Availability Bias The idea here is that people assume that things which are easy to think of are more common or more important. The classic example is street crime, which has been decreasing steadily in the US, but has been reported on more and more during the same period. Because of the impression given by the media, people assume that street crime is far more common than it is.
There are a lot of things which become very 'available' in the context of a game. In particular, everything which is part of the main game loop. A common example, in a game which involves a lot of combat: when the player encounters a locked door which they want to enter, his/her first instinct will be to cut through the door with their sword, gun or whatever. The solution of chopping or shooting is very 'available.' A game designer can decide to meet or subvert the player's expectations, or recreate them entirely.
3. Anchoring Given that the Civil War started in 1861, what year was the Gettysburg Address? Nope, not 1862. It was 1863. Anchoring is when people use some baseline to estimate how much of a resource to use. People almost always undershoot how much difference there should be between the baseline and the new value.
Resource management is a big part of most games these days, and bullets and mana can get pretty scarce. It's often possible that a player will take on a challenge they simply don't have the resources to complete. Whether that's desirable or undesirable depends on the game you're designing.
The game's baseline resource requirement might be something like how much health a player usually loses in a typical encounter with a monster. If you create a monster that looks similar, but a bit more powerful, chances are the player underestimate the difference in power between these two monsters. Many designers compensate by making their monsters look more powerful than they actually are, so that the player's underestimate will match up better with the reality of the situation. Appearances can play a big role in accentuating or downplaying the effects of anchoring. Appearances also play a big role in...
4. The Halo Effect This describes people's tendency to caricature others, using what little they know about one aspect of a person to make generalizations about aspects they know nothing about. For example, picture a struggling art student who is found unfit for military service, but insists on serving anyway and becomes a war hero. It may shock you to realize that this is an accurate description of Adolf Hitler.
Players form stereotypes like this for all the NPCs in a game. Unfortunately, most games' NPCs meet these expectations in every way. The good news is that expectations like these, bolstered by the tendencies of other games, make reversals of character, like the one in Portal 2, quite unexpected.
What role do appearances play in this? Probably the biggest stereotype is that good looking people are better than less-attractive people, in various ways; morally, at their jobs, in bed, or whatever. If you take a moment to think about it, this is obviously not true, but everyone uses this as a first pass heuristic.
You can use this in your game. You can go against the grain and make better-looking characters and items which are less effective, or you can go with the flow and use appearance as an accurate gauge of utility. Which you choose depends, again, on the game.
5. Duration Neglect and the Peak-End Rule "A bias that favors a short period of intense joy over a long period of moderate happiness [and causes us to] fear a short period of intense but tolerable suffering more than we fear a much longer period of moderate pain." (Kahneman 409)
This is a more general effect. If you're designing a linear game, then this applies mainly to your intensity curve (I recommend you read the linked article). If it's a simulation, or other nonlinear game, then you need to plant opportunities for sudden intense gratification, and use sudden and intense discouragement. If your punishments are slightly-bad effects which hurt you over long time periods, your player won't get the message -- they'll just think it's gotten more difficult. If you have no punctuated rewards, but rather a long drawn-out feeling of mild pleasantness, then you've made an art game.
Those were the most important ideas I took away from the book. Of course, playtesting (done right) takes all of these into account, because it's real people playing the game as they normally would. On the other hand, the playtesting phase may be too late to introduce some of these ideas. You may have to use your rational mind to question your irrational mind. You may ask it "how do you play?"
I'll mention some other cognitive biases quickly and you can look them up if you're interested: competition in system 2, regression to the mean, eye effect, cognitive ease, law of small numbers, optimism and planning fallacy, and the focusing illusion. There was another gamasutra article about a different bias, the Dunning-Kruger effect, which is here.