I can see where you’re coming from, but consider this: When it comes to games that have an emphasis on combat (and dnd certainly qualifies), it can be useful for GMs to think of their encounter XP budget as their narrative budget. In other words, the more dramatic a fight is, the more difficult it ought to be. Also, inside that encounter, the monsters that eat up most of your encounter’s XP budget deserve the most narrative spotlight.
Imagine a GM has been running for a group for some time. Now, a climactic fight is upon them. By all rights, this should be a narratively momentous occasion for the table. Only it isn’t. The system’s encounter design fails and the whole fight gets completely trivialized. But such is dnd and every once in a while, your planned encounters just end up falling on their faces.
And we accept this. Because that’s just how it’s always been, right?
Only imagine this happening in another system. What if the gang was playing a forged in the dark system instead of dnd. The situation in the fiction is dramatic and challenging. The GM rightfully calls for many risky and desperate rolls and judiciously applies standard and limited effect. The party should be in for a world of hurt. Except now their dice pools all of a sudden are tripled because the system somehow breaks. In fiction-first games like FitD, this would be unforgivable. The table would be correct to just ignore the core resolution mechanics here because they fail to represent the fiction at play.
But that’s exactly what happens with dnd on a regular basis. And it doesn’t have to be that way. Encounter balancing in pf2e and dnd 4e (with asterisks) works rather well.
But why should we care to have balance? Oh, tons of reasons. Just the example I’ve raised here is that balanced combat allows GMs to set out challenges for the party that match the fiction well. This is notably different from having every encounter be equally challenging. Just that they roughly match what the GM had envisioned. And there are lots and lots more reasons.
This isn’t to say that balanced combat is superior to swingy, unpredictable combat. Both can be lots of fun. The key is to understand the type of game the table wants to play and lean towards that direction.
I agree to a certain extent. I don’t think 4e really comes into its own before a GM applies post-Monster Manual 3 math and gives defenses + expertise feats out for free. It works, more or less, but requires the GM to be cued in more than what they would get from just reading the GM guides (which are mostly excellent).