Planning AI seems to be a big thing these days. NPCs decide a goal and then work through the steps required to achieve it. But getting out-maneuvered by your enemies can be frustrating. Clever AI is at its most fun when you can see them doing something smart and then foil their plan...
The problem is how to communicate intention to the player.
Walt Disney noted in the '30s that characters should anticipate their actions:
"A lot of valuable points could be brought out to the men in showing them that it is not necessary for them to take a character to one point, complete that action completely, and then turn to the following action as if he had never given it a thought until after completing the first action, anticipation of action is important."
People look at things before they interact with them. An NPC in combat might notice some cover, look at the player, consider the odds, then make a mad dash for safety.
It's important to note the NPC can't decide to take the cover without first observing it. His focus should rest briefly on the object. The glance at the player serves to punctuate the moment of consideration and suggests the value of the cover is determined relative to you.
It's all in the timing.
Gaze direction tells you what an NPC will interact with. Expression will tell you how.
The Disney animators working on Pluto in the thirties found a change of expression made it look like a character was thinking.
Expression change = thought-process
Expression can tell you how an NPC will react to your actions. A look of fear might signal a retreat, an angry look of determination might serve as a warning. In a conversation, changing expressions can give feedback on how to proceed. What if you could stop when you sense your words are having the wrong effect?
People like to project human qualities onto things. Perceived emotion suggests intelligence. Between visible observation and emotional reaction, emergent character expression will make our NPCs look a lot smarter than they really are.
Arguably the more difficult problem: Expressing things with body language is all well and good, but it's not much use if the player is not around to see it.
Thomas and Johnston note in The Illusion of Life (1981) that if you make a subtle change during a broad move, the change is lost. It doesn't matter if it's the camera or the character that's moving.
Both first and third-person games rarely keep a static view. Unless the player chooses to pay attention, any feedback must be broad enough to stand out.
How then to avoid over-acting without losing information?
A solution would be to adapt the acting to camera, timing actions dynamically to occur when the player is looking, playing broad gestures when far away and saving subtle ones for when the camera is still.
A raised eyebrow or knowing smile may be more suited to quiet moments, while frantic gesturing will read from all but a mile away.
Another option is to lead the eye through composition and set up scenes in such a way that players will inevitably see what you want them to, but the more open-ended the interaction the more adaptive the staging must be.