The application of this technique to a problem as complex as d&d optimization could be the subject of a master's thesis, I think.
The primary problem would be to design the evaluation (cost) function, I think. I'm pretty sure melee damage (complex as it is) can be calculated. Other criteria could be any of the statistics of a build, HP, AC, saves, highest level of spells, type of spell list, ... I'm not sure whether any of this helps though since the most complex problem we've done in class was the traveling salesman problem. I don't even know where to start.
edit: Or.. we could randomly generate a large data set of legal builds and then judge whether they're good or not and feed the machine with it. Which, I'm slowly realizing, is probably what you wanted to do.