The right mental model is that, in modern out-of-order CPUs, there is no difference between executing "in speculation" and "not in speculation". There is just a huge pile of u-ops (~200 on big Intel chips) in the reorder buffer, and some of them may be unexecuted branches. The fetch unit plows right on through branch instructions using branch prediction and jams code "from the future" into the reorder buffer.
The CPU keeps on executing whatever instructions are ready (i.e. have data dependencies met), irrespective of the in-flight branches. When a mispredicted branch is retired (~0.1-1% of the time), the CPU can throw away the entire reorder buffer and start over. A CPU can also abort-on-execute, throwing away only work related to the given mispredicted branch. So probably 99% of all cycles are spent with at least one unexecuted branch somewhere in the reorder buffer--the CPU is essentially always speculating.
The CPU keeps on executing whatever instructions are ready (i.e. have data dependencies met), irrespective of the in-flight branches. When a mispredicted branch is retired (~0.1-1% of the time), the CPU can throw away the entire reorder buffer and start over. A CPU can also abort-on-execute, throwing away only work related to the given mispredicted branch. So probably 99% of all cycles are spent with at least one unexecuted branch somewhere in the reorder buffer--the CPU is essentially always speculating.