When I began the 9th draft of my novel at the beginning of the year, almost all of it changed since the first draft. The biggest change was that my cast expanded from two viewpoint characters to fourteen. I have to write new scenes pretty much as I go along but the broad strokes of the plot remained the same; it's all set in the same story universe, so the major events that occur would be the same for everyone.
I'm writing the first in a series of novels, and the idea was to get everything done in five volumes. This was back in the days when I had only two viewpoint characters. Multiplying out the cast would make the first story too large in terms of page numbers, so I resolved to split it into three parts. The first part, the novel I'm writing now, would end with American fighter pilots shooting down an alien spacecraft and giving us our first look at our otherworldly enemy.
This. Is. Stupid. There are three outcomes possible if I stayed this course. First, the reader would be blown away by the aliens I'd written and excited for more. Second, the reader would be disappointed and quit on me. Third, and most likely, the reader would get tired and give up before reaching the end.
Going back to that soldier in Denmark. The scene I was writing was originally about him fighting severe civil unrest in Copenhagen. Literally five or ten minutes before I sat down to write, I changed my mind. Civil unrest be damned. The aliens were landing. This move was completely unexpected, but after two hundred plus pages without seeing the aliens directly, I felt like the reader just had to come face to face with them.
This also meant that my beloved note cards were useless. The landing of the aliens, as you might expect, is a major event, not some minor change like switching menu orders for a character's lunch. I still expect to get this first big chunk of the series done in three parts, but now I really have no idea where I'm going.
There are plenty of reasons why it's good to abandon the plot you started out with and move on in a new direction. Being a war story, my novel now has a slightly added sense of realism. That is, you've got two forces fighting each other, each doing their own thing in ways that take their opponents by surprise.
Regarding craft, this change of plot has two boons to it. First, it keeps the work fresh. If you always know what's going to happen, you risk losing interest and give up. Or, if you're locked with a long-term project like a novel, you come back to your keyboard with a sense of frustration, saying to yourself, "I know I have to work on this, but it's such a chore." A writer's work shouldn't be a chore. It should be a pleasure.
Also, this keeps your storytelling skills sharp. When you write a plot on paper and stick to it word for word, you don't give your brain any exercise. By leaving the future and the outcome of the story unclear, even to you, you keep asking yourself what comes next. You keep exploring the options and making decisions.