There are several environmental concerns when driving a hybrid.
One is the rare earth metals used in permanent magnet motors such as neodymium. However, it has been already established that electric cars end up being more sustainable than gasoline cars, and also the electric motors in hybrid vehicles aren't bigger than those in electric cars (actually in some cases they may even be smaller), so this is not an issue.
Another is the battery. However, a non-plug-in hybrid car has typically 1-2 kWh battery, whereas a plug-in hybrid car has 10-20 kWh battery and an electric car has 50-90 kWh battery. So you can see from this that the amount of metals in a non-plug-in hybrid is insignificant, because the battery is so small.
However, a non-plug-in hybrid very often uses a different battery chemistry. Prius uses nickel metal-hydride whereas electric cars practically all use lithium-ion today. In Prius, the main concern is lanthanum (10-15 kg in battery) whereas in electric vehicles the main concern is cobalt (10 kg in battery). World cobalt reserves are 7.1 megatonnes, and world rare earth reserves are 120 megatonnes. If we assume 25% of rare earth metals obtained from a mine is lanthanum, that would mean we have 30 megatonnes of reserves. So it can be seen that cobalt is limiting electric vehicles more than lanthanum is limiting non-plug-in hybrids. By the way, the lanthanum reserves would allow 2-3 billion hybrid vehicles using nickel methal-hydride batteries. That's more than the number of cars on the roads today.
Also it's possible to consider that battery recycling is very efficient and nearly all of the metals in batteries can be recycled. The battery can be treated as a deposit of valuable metals far richer than the deposits found in mines.
Overall, because gasoline cars use oil at a far greater rate than Prius uses oil, and because lanthanum isn't consumed by hybrid vehicles, it's probably better to reduce oil use by putting the lanthanum inside hybrid vehicle batteries and then at end of life recycle the lanthanum, than to leave the lanthanum in the ground and continue using oil at a great rate.
In the end, the main concern of non-plug-in hybrid vehicles is that they aren't good enough. We shouldn't reduce oil use and carbon dioxide emissions by 30% (which is what Prius does), but by at least 90% if not more (which is what electric vehicles in a clean electricity grid do).
Disclosure: I drive a Toyota non-plug-in hybrid, but my next car will be a fully electric car. The only reason I drive a non-plug-in hybrid was that back when I bought the car, the price of reasonable comparable electric cars was at least twice of what it is today, and plug-in-hybrids had concerns of battery wear and weren't available from Toyota back then (I only buy Toyotas because I like reliable cars). Also back then I didn't have a realistic possibility to charge a plug-in-hybrid, because the only electrical outlet available was behind a 2-hour clock switch.