Yeah, DM fiat. For the most part, it is simply not worth the bother to track damage to a stone wall or pillar, though critical fumbles are always fun. A pitched battle in the castle of a lord the PCs are trying to impress would have consequences, though - scorched murals, chipped tile, smashed vases, terrified servants, and other things you would really prefer your guests avoid. I am still looking for the right party to collapse a building around (preferably while on fire; fire makes everything better) to add another dimension to the fight - a hard time limit, changing terrain, randomly falling timbers, limited visibility, and temporary scuttle holes for small monsters. They need to have enough options that escape is reasonably likely, but low enough level that being trapped in rubble after being hit on the head with a roof is a meaningful threat.
Damaged terrain can also serve as a warning to canny characters that they have entered the domain of a dangerous monster. Great slabs of rock clawed out of the wall, odd puddles, and a squashed adventurer with a satchel full of alchemist's fire could indicate a high-strength cold-using fire-vulnerable critter ahead.