This article began as an answer to a multifaceted yet straightforward question: what makes a good roleplaying game? I have worked with several RPGs during my career and even written and done presentation on the subject, but I have never before approached it from a holistic point of view. There are plenty of material on what makes a good game, and naturally, most of that applies to roleplaying games too, but there are some design decisions that are specific if not unique to the genre.
Since it's impossible to give a list of features a roleplaying game should have, and there are numerous ways to approach any given design aspect, I broke down the design into four distinct axes. None of the axes alone can tell if you're on the right path with your design, but together they help to manage the big picture.
From the early days of Dungeons & Dragons, Chainmail and their precursors, randomness has been in the core of the roleplaying game design. In tabletop games, you used dice to determine the outcome of any uncertain action -- and there were a lot of them back in the day. As the discipline of RPG design advanced, the number of dice rolls slowly decreased, and some games even left them out completely. In digital games, some randomness has been replaced by player skill in more action-oriented games, but it's still an essential part of any a traditional RPG experience, especially in fantasy games.
There is rarely a sweet spot for randomness versus determinism, and the best approach varies from game to game. My process has usually started with too much randomness and then adding determinism iteration by iteration. My RPG battle design articles go into this in more detail, but here's a short recap.
In tabletop games, rolling dice is an action you take: "I rolled 17." Even though we consciously know that we can't affect the outcome of the roll (except by cheating), rolling well feels like a personal accomplishment and a failed roll is either your fault or just a bad dice. In digital games, a streak of bad luck feels like a fault of the game and can get frustrating very fast. That is why we need to give the players more options to own their action. In Rimelands, we added the reroll option for this reason. Since the game only rerolls the failed dice, the option actually reduces the randomness, and by costing a single mana point (you always have a pool of precisely five mana points), it makes rerolling an actual choice.
Another good option to add determinism is to have a relatively limited range of randomness for damage, a solution often seen in JRPGs. The worst luck becomes a bottom line, and you can only go upward. Rolling a crit feels good even if you're not the one doing the rolling, but you can never really botch due to bad luck.
A low amount of randomness usually leads to battles feeling more like puzzles in which you have to find the right combination of abilities to defeat a particular enemy. Usually, the most deterministic fights in such games are boss fights, but Persona 5 (for example) takes this to another level, requiring the player to know the exact strength and weaknesses to succeed in any fight.
The more options you give the player, the more agency they have. The player should never feel like the lost a battle due to a bad dice roll, but because of a choice they made.
Roleplaying games are often a content heavy genre. Varied battles, exploration and story, require vast amounts so character and environment assets, level design, writing and audio. Common player expectations of tens of hours of gameplay make the requirements even worse, especially when making a relatively linear story-driven experience.
There are many shortcuts. You can recycle assets by using the same enemies with different level design to bring new challenges to the battles, alter textures instead of making whole new models and use lighting to create a different mood and context for environments when using the same basic blocks. A robust level design tool is essential when creating a content-driven experience and well worth the initial extra effort.
On the other end of the scale, you can also go with less content heavy and more mechanical experience. This approach is common in roguelikes, which use runtime procedural level generation to create a new experience for each session, usually using the same assets as before. Another example is puzzle-roleplaying game hybrids such as PuzzleQuest which rely heavily on their battle mechanic and leave the story content to the background.
How well a mechanics-oriented game works depends heavily on the balance of the previous axis. Since you lack the opportunity to hand-design new challenges using the same assets, the game's core mechanics need to support a wide variety of choices and emergent challenges. Failing to provide a compelling enough core mechanic makes the game feel grindy and frustrating.
One thing that I've noticed lessens the feeling of grind is having some secondary objectives. It feels less frustrating to hack through dozens of enemies if some other meters go up along with your XP bar.
This axis is somewhat related to the previous one, but more from the perspective of the narrative experience. As with grind, the story is content and as such expensive to produce, but roleplaying has had an interactive narrative as one of their core tenets from the very beginning, and the whole genre is as much a medium of storytelling as it is a form of games.
Whether fairly linear games with a great story are better than open world games with an emphasis on player freedom is mostly a question of taste, although the real classics manage to find a very delicate balance between the two. However, the ones that do are more often than not gargantuan products with budgets of tens if not hundreds of millions.
For any smaller game, you need to have few tricks up your sleeve to balance this. One obvious option is to scale down the fidelity. If you have your story as mostly text and the art is relatively low-resolution 2D, you can fit in a lot more content with fewer resources. Another is to recycle the assets you have: instead of a story choice leading to different places with a different cast of NPCs, slightly tilt the content in a single place and use the same characters. Telltale was a master of this sleight of hand: the outcome was mostly the same, with the only difference being which character did or said what. The "X remembers this" was also a neat trick that made a choice feel important even if in reality it had little or no effect to the actual outcome.
You can go too far to another direction too. Modern Final Fantasy games are not exactly known for their free-roaming gameplay, but Final Fantasy XIII managed to take this into an extreme that scared off even the hardiest fans.
Advantages and narrative mechanics
I'm mostly talking about character generation and progression here, although this does also relate to any amount of choices the player can have. As with the previous axis, this is also very much a question of taste and the player's experience in the genre.
Both tabletop and digital roleplaying games can feel overwhelming to an unaccustomed person with tons of different statistics, skills and other character options. Especially if the first screen after starting a new game the player sees is the character generation. If you have to make very longstanding choices before having any idea how the game works, there's a huge risk you're screwing up even before you've even started.
This is an axis where Japanese and western RPGs tend to be on the opposite sites: JRPGs don't usually have any kind of character creation, and the stat progression is often very linear with one or two systems offering options, whereas western games stick closer to the tabletop model of character generation first and very granular progression choices.
In Rimelands we went halfway and ditched the character generation, having the player start with a blank slate. Then after each level up, the player could select from three classes; they could either stick with one class the whole game or select a different one each time, depending on which one seemed to offer the best options.
Depending on your target audience, you can go either way here, or even compromise and choose some mixture of the two, but you need to make a conscious choice, as this axis very much ties to your choices in the other ones. They all do. A game with approachability instead of complex options works better if it also is more story-oriented and less grindy and vice versa. Naturally, you can mix them other ways too, but you need to be careful and know how this affects the whole experience.
You can balance depth and approachability by providing proper onboarding and expressive mechanics. A good example is advantages and disadvantages, which provide both narrative flair and variety in mechanics.
This method is not a be-all-end-all solution to roleplaying game design, but a tool for analysing the balance between elements in your work or breaking apart other games for study.