I only did a Master's and avoided the academic route (wasn't interested). My lab peers at the time were 5 PhD students. Of that group at a good Canadian University, 3 are still post-docs (4-6 years after graduation), 1 has a decent-paying research position (non-teaching) and 1 is a faculty member... in the Middle East. Meanwhile, I work in an R&D field alongside many current faculty members, a surprising number of whom are 65+ and will die before they retire.
In general, my observation has been that the students that quit after a Masters tend to do better financially than those who continue on. That's not to say that Master's graduates are hitting the big time; they just tend to end up getting their market value.
I think the downside of no PhD is that you're much more likely to hit a glass ceiling. There are obviously exceptions but you're more likely to be put in charge of a research project in industry if you have a PhD than not.
Clearly an MS or BS level science student can absolutely eventually do the same things as a PhD or postdoc. The latter just have scientific approaches that have usually been subject to repeated and rigorous challenge by their peers, which means they are typically (though by no means always) more rigorous out of the box. In my time in industry I've found that in many cases that kind of critical challenge doesn't happen as much as in academia.
In the end I think people in hiring positions use the PhD letters as a screening tool for how strong a scientist the person is, without necessarily really evaluating the individual.
I think the downside of no PhD is that you're much more likely to hit a glass ceiling.
I would agree in the context of staying within the R&D field. However, I know more than a few people who used the extra years not spent doing PhD post-doc to develop their careers in a different direction (sales, biz-dev) that allowed them to stay in their field and shattered those glass ceilings. You can still do these things with a PhD, but it will be more difficult (people pigeon-hole you, and you're playing catch-up time-wise).
In the end I think people in hiring positions use the PhD letters as a screening tool for how strong a scientist the person is, without necessarily really evaluating the individual.
This is key. I've actually found that a PhD. is a relatively poor predictor of how well a person might function in an industry R&D role. Please note that I'm not saying it's a negative predictor. I've just found there's little to no correlation. I've worked with incredibly smart and capable people with PhD's and others who could not function outside of a low-consequence lab setting.
I did my undergraduate in Physics and Mathematics about ten years ago now. When I started the program I wanted to be a researcher and delve into the secrets of the universe. I quickly discovered that it wasn't for me and moved into Computer Science.
One piece of advice that stuck with me the most was from the director of career development, who had previously been a research manager at IBM. He told us that if we wanted to go into industry just skip graduate school and focus on getting lab experience. He told us if we really wanted to do graduate work to just get a MS and then go into industry. He told us to only do a PhD if you wanted to go into academia because not only was it a waste of time compared to what you could learn and earn in industry, it would pigeon hole you into a very specific field.
He was very adamant that a key skill in business was being flexible and PhD programs most certainly aren't.
I think I disagree with you and OP, but for different reasons.
In R&D, it's a matter of career path. A PhD is expected to become a lead, someone to guide a research program. A Master's degree holder will be closer to the action and likely develop technical skills that make them an asset lower on the org chart.
The problem, I think, stems from the fact that PhDs are also expected to master their techniques and to perform well as a requirement for advancement. They have a career path that extends to management, but are using mostly technical skills at the beginning. The Master's degree holder does not have this difference.
I think "PhD is a poor predictor of how well a person might function in industry R&D" is spot on. PhD is a poor predictor of most things that hiring managers are looking for, I might say!
I had a similar experience as you. I wrapped up my Masters in Canada because the job prospects were better than if I stayed and finished a PhD. Most of my lab mates went into industry and are doing well. Only one landed a prestigious faculty position.
In general, my observation has been that the students that quit after a Masters tend to do better financially than those who continue on. That's not to say that Master's graduates are hitting the big time; they just tend to end up getting their market value.