One of D&D 3.5’s biggest problems was its magical system, which was by default bonkers and broken. Despite that, though, there was an almost constant attempt to expand the magic system, to fulfill every specific small variant visions of magic. The Expanded Psionics Handbook, the Tome of Magic, the Artificer and the Archivist, wings of spells in The Book of Vile Darkness and The Book of Exalted Deeds – there was a near-constant effort to expand the magic systems to do different things in different ways.
You can approach design from either a strong mechanical position, or a strong theme and Incarnum was a power idea that came hard out of a mechanical interaction. Magic of Incarnum brought its own system, called the Incarnum system.
The Incarnum system, as simple as I can explain it, is that you could create virtual items in your item slots, and then invest a small pool of points into these items to make them better. This could give you special abilities that got better, from turn to turn, and you could rearrange all these points every turn. When you needed lots of defence, you could sink those points into defenses. When you needed to kill something fast, you could put those points into offense. This system was pretty interesting and cool! You could really customise a character in a lot of ways, and there was this balancing act of choosing where your points were by default.
Incarnum however, was a really bad system, not because it was bad, but because as a player, you had to spend the bulk of your time juggling a small list of points for a small advantage. None of the Incarnum values were particularly large, and the niche utility of some of the shifting was as much a matter of pooling skill bonuses into your armour at the right moment at the right time, rather than really changing what you did. The system was designed to be careful enough as to not get out of hand like the existing spell system (which was broken), but still be an alternative worth playing (which was pretty hard, when the spell system was broken).
The real thing though that kills Incarnum is cognitive load.
Cognitive load is the concept in psychology that describes the amount of active memory you have to track to keep a task executed. It’s how you concentrate on something, it’s the work required in your brain to manage the information presented to you organised. Incarnum was a system that started with cognitive load problems, and it got worse as you levelled up.
You might sometimes hear a player describe a game as ‘smooth’ or hear a designer say something is ‘frictionless.’ Mostly, that feeling is attained by making sure your design does little to demand cognitive load without a reason. It’s one thing to concentrate on a complicated turn or a crucial strategy, but you don’t want the everyday operations of play to require you to make a lot of complex planning and contend with juggling information.
There’s a reason designers preach the idea of simplicity. It’s not for its own sake, it’s because you want to make it as easy as possible for the players to make decisions about what they want to do in your game, rather than have to do math on working out how what they want to do can work.