The tech tree quietly does a lot to reinforce the major themes of the game and to provide the heartbeat that keeps players moving through the game.
I think the most important thing that it does is ensure constant progress for the player. Civ greatly benefits from always giving the player something they want within a couple of turns. This is a large part of what feeds the one-more-turn compulsion of the games.
It serves much of the same purpose of leveling in an RPG, albeit flavored strongly by the rest of Civ. Like RPG levels, it gives you a major in-game reward on a regular basis and helps make the player feel like they've grown over time.
It really forces the game to feel like it's moving forward. It introduces new pieces regularly and so keeps the game from feeling like it's stuck in a rut. No matter what else is going on in the game, there's always a new tech in a few turns and most of those techs have something interesting attached to them.
This progression is strongly reinforced by the way that the techs tend to have something meaningful attached to each of them. There are a few that I don't care about, especially ones with non-essential military units, but mostly they have something that I want.
The game as a whole is good about making everything meaningful. It encourages you greatly to have a variety of all of the pieces, such as improvements, units and districts. This gives the game a lot of pieces to give you and so it can distribute them across the tech tree and keep the techs valuable.
This isn't completely perfect though. Things like aircraft upgrades just don't have enough of an impact in a normal game for me.
The other major theme that it does a lot to reinforce is that of progression through history. It's the feeling of having started as just a single settler and a warrior and growing to be a massive, advanced empire. The abilities to trace technological advancement and the idea of getting to such things as spaceflight from such techs as animal husbandry and archery are very satisfying.
Also, having techs unlock other techs adds value to early techs. Now, not only are they valuable for what they give, but also for what they unlock. The tension between a tech that has more immediate benefit and one that will let you get to a more powerful tech sooner is also fun for players to navigate.
Despite my above point of the meaningfulness of the techs, there are some that feel much more powerful than others, and those act as guides for decisions that have no clear answer. In this way, they greatly reduce the effective complexity of the decision of what to research for mid-tier players.
This is a little bit of an involved point, so I'm going to expand on it a little. Essentially, deciding which tech to research next is an individual skill that players develop as they play the game. When this skill is not fully developed (and it may be unreasonably difficult to ever fully develop this skill), there are going to be situations in which the player cannot easily decide between a number of options for which tech to research next, when those techs are judged simply by their immediate value. The existence of powerful techs further down the tree makes these decision points more approachable as they give the player a direction in which to research.
A sub-point here is that techs which result in a major difference in unit strengths are one of those asymmetries and an interesting one to unpack. First of all, the one in SM:AC is one of my favorite impact moments in video games because when you jump from two attack to four attack, but the defense remains the same, that's a moment that highly incentivizes attacking while you retain the technological edge. Civ6 also does a good job of making units of different tech levels differ greatly in power, so your crossbowmen one-shot the opposing archers.
That's very satisfying because it's the conversion of a scientific advantage into a military one and it's fun to be in unequal battles that you've earned. Also, it is the realization of the fantasy of winning a war due to superior technology, which is a familiar narrative, albeit one I believe to be heavily misapplied.
The big issue with this tech tree system is the degree of front-loaded complexity that comes with it. It's very hard for a beginning player to make a decision of what tech to research, even when only considering the choice menu and not the full tree. There's a lot of content that requires familiarity from the player and the choice of which tech to research requires a lot of comfort with the rest of the game as well. Complicating this with the tree structure exacerbates the issue. To some degree, the depth of the tree makes this unavoidable however.
Essentially, just having the knowledge required to make a decision is a major barrier to entry for new players. Things like the advisors are meant to help with that, but it's probably insufficient. To some extent now, new Civ games are meant for people who have played earlier Civ games and so it can be hard for players who don't have that experience.
An interesting point to think about is that the tech tree is static between games. The techs are in the same positions on the tree, cost the same amount and unlock the same pieces every time. This is probably in large part because it's much easier to design and implement that way, especially when you have to take historical pseudo-realism into account, but the results of that decision are worth examining.
You can ensure consistency in the above points through this static design. A good procedural solution could do that as well, but it is again substantially easier to manage with a hand-crafted tree.
This drastically reduces the complexity of the tech tree on later playthroughs. One of the skills of the game is simply getting better at traversing the tech tree. This naturally also leads to players no longer engaging with the decision point of which tech to research next because they feel they have a good, solved path. The rest of the game does influence the decision, but the raw power of some techs is often enough to overwhelm those wrinkles for some players.
Tying the tech tree to historical technologies greatly reduces the complexity. Logical connections between the pieces make them much, much easier to understand and historical context helps the player get an idea of what the techs do.
Also, it pushes a compelling fantasy of how technological research works. Having technologies that lead to other technologies and being able to trace that development is intrinsically very satisfying due to what research is in the real world.
This is my favorite feature from Civ6 just for all that it does. As you may have seen by now, I love things that tie different parts of the game together and having your actions in the world have such a direct impact on research is wonderful. I love how this plays and how it generates sub-quests which encourage trade-offs. I really enjoy when the game pushes you to go kill something with your slinger so that you can boost archery for instance.
Also, the flavor of the piece is wonderful. It makes sense that slingers would look to improve their weapons. It makes sense that having a number of musketmen would spur replaceable parts. The whole thing is just something that feels like it makes sense.
Civ6 does something very elegant in making the techs and civics use the same structure. However, these two things need to feel different somehow, and if it isn't in the mechanical structure, then it needs to be somewhere else. Civ6 handles this with making the reward for civics mostly be policy cards. This certainly functions effectively as a way to differentiate the two trees, but it is both difficult to understand the ramifications of policy changes, and the system feels divorced from the rest of the game.
By and large, it manages all of the above points, but the rewards don't feel as meaningful as in the tech tree, and so it doesn't do the same job of feeding the one-more-turn compulsion that the tech tree does. However, it does not exist in isolation, and so does not need to.
The first alternative to examine is that of Beyond Earth, but the visual difference merely hides an equivalent system. There are minor differences in terms of branching, but despite the difference in appearance, the system is functionally identical to that of the other Civ games and so has the same benefits and weaknesses.
On the other hand, this lack of a tech tree is one of the biggest weaknesses of Colonization. I feel like one of the major issues with that game is that there often comes a point where you feel like you've gotten into a bit of a rut. Essentially, the next real goal is quite a distance away and so the one-more-turn compulsion ends up lost and the game founders.
The Stellaris design of giving the player a choice between a group of semi-random options from an underlying tech tree might on first glance feel very similar to the Civ one, but the presentation and randomness detract from the points above about long-term planning and so the feature is much weaker. Additionally, the techs do not all feel meaningful (at least, as of before the major change to combat from a few months ago, when I last played). However, the complexity of this decision for new players is thus also greatly reduced for new players as the decision becomes which one of the limited options is most immediately beneficial. For advanced players, it reduces to something much closer to that of Civ, but not quite there.
Polytopia has the major difference of requiring the same currency for research as for everything else. This completely removes the major point of constant progression detailed above. However, each game of Polytopia is both short and dense and so it can stand to lose that piece.
Sorceror King and Master of Magic have a more implicit tech tree in that buildings unlock other buildings. This again loses the constant progression that is a critical benefit of the tech tree and so the games end up with the same problem as Colonization where there comes a point without a reward that is both compelling and reasonably near and the game starts to sputter.
The Civ games have long been known for that one-more-turn compulsion, and I think that the tech tree is the driving force behind that. It's quite safe to say that without this, Civ would not be anything like the game that it is.