Yes, we should leave the larger part of all known fossil fuel reserves in the ground. Scientists have calculated that if we want to keep global temperature rise under 2 degrees C (which was agreed upon in the Paris climate agreement), then we cannot use more than 1/3 of all currently known reserves.
...globally, a third of oil reserves, half of gas reserves and over 80 per cent of current coal reserves should remain unused from 2010 to 2050 in order to meet the target of 2 °C. (source)
Fossil fuels that are in the ground are usually stored safely there and do not contribute to global warming. But once you take them out their stored carbon content is almost always added to the atmosphere in the form of greenhouse gas emissions at one time or another. Your statement that
Using fossil fuel to make plastics does not necessarily produce more CO2 than production of bio-plastics
is only true in theory. It only holds for plastics that are stored safely at the end of their lifetime. In practice however large amounts of fossil-fuel-based waste plastics are incinerated and when that happens they add CO2 to the atmosphere. When bio-plastics are incinerated they also add CO2, but the difference here is that the biomass that was used to create the plastic has taken CO2 out of the atmosphere when it was growing. That's why bio-plastics are carbon neutral (if you neglect the emissions from their manufacturing process).
I do agree that we should 'use fossil fuels wherever their end use would have less environmental impact than the alternatives', but there are very few uses of fossil fuels for which there isn't a more environmentally-friendly alternative. The only thing I can think of right now is jet fuel, but according to this question they are working on that, and of course reduction (less flying) would be even better from an environmental point of view.
There is more information in this Guardian article: Leave fossil fuels buried to prevent climate change, study urges
