This is quite a difficult question to answer with precision, because the impacts of mining depend a lot on how you do it. I can offer a few references which give a guide to the scale of the problem.
There are many environmental concerns around the impact of Lithium mines on the environment and local communities (e.g. Reuters article on Salar de Atacama).
Quantifying Impacts
The difference in scale, maturity and categories of pollution involved make a detailed comparison of all the toxins involved difficult. I will look at two broad areas of impact: the area of land taken up and the volumes of water used in the processing chain.
Assumptions
To make the comparison, consider a Nissan Leaf with a 40kWh battery. Taking the low end of Lithium content from wikipedia of 0.15kg per kWh (the efficiency of batteries is rapidly increasing, so we can expect improvements beyond this figure) we require 6kg of Lithium.
For our petrol car, consider a car driving 12,000 km per year (the UK average) at 6L per 100km, and hence consuming 2,000 L per year.
We will also consider a 10 year lifetime for the battery.
It could be argued that this comparison is too pessimistic about the Lithium because it SHOULD be recycled and have a much longer lifetime than 10 years. However, current recycling rates in, for example, Australia are 2% (Inst. Energy Research) .. so we should not count this benefit too early.
I have not dealt explicitly with petroleum extraction: if done cleanly the extraction itself is the major cause of concern for petroleum, though it would be for coal or shale gas. Toxic chemical spills are a problem. Because of the much lower volume of Lithium needed compared to oil, we can expect spillage to be a smaller concern. We might hope that the high value to weight ratio will lead to a cleaner processing chain and reduced damaging spillage, but this hope cannot be verified at this stage.
Water Use
The Institute for Energy Research say that Lithium mining (from salt flats ... see comments on open cast mines below) requires 500 gallons (2,250L) per kg of Lithium. Most of the environmental problems are associated with waste water that is heavily loaded with a toxic mix of chemicals.
We can compare this to water use in petroleum refining, according to fluence, of about 0.5L per litre of petrol.
This means that the 2,250 L/kg * 6 kg = 13,300 kg of water usage for the electric vehicle should be compared with 2,000 L/yr * 10 yr * 0.5 kg/L = 10,000 kg of water usage to create petroleum for the conventional car.
In both cases, this is much less than the typical domestic usage of 40,000 kg per person per year in the UK. The problem is that this may be heavily polluted water and the usage may be concentrated in very small and sensitive areas.
Area of Land Needed*
Lithium battery factories have a huge scale, how does this compare with oil refineries?
The Tesla giga-factory in Nevada aims to produce 50GWh of batteries per year, enough for 1.25 million 40kWh cars, and covers 180,000m2. Taken over our 10 year period reference period, this means 0.18/12.5 = 0.014 m2/car. Seen like this, it is not taking up much space.
By comparison, the world's largest oil refinery at Baton Rouge, Lousiana covers 8,000,000 m2 and produces 80,000,000 L of petrol per day. The typical car described above needs 2L per day, so Baton Rouge is supporting 40,000,000 cars, or 0.2 m2/car.
Both these numbers indicate that the area of the fuel/battery processing plant is small on a per/car basis. In terms of emissions, battery factories do not appear to have significant waste outputs. The refineries, on the other hand, have a long list of toxic by-products and problems associated with leakage in the input supply chain.
We should also look at the spatial footprint of Lithium mines. Their are two categories of mine: open cast mines from which a solid ore is extracted and brine deposits. The brine deposits, such as in Salar de Atacama (Chile), currently provide the cheapest source, but many countries are developing large open cast mines, such as at Thacker Pass (Nevada, US).
The Salar de Atacama brine pools are visible in google maps from which I estimate their area as 25 km2. This site produced 80,000 tonnes of Lithium per year in 2017 (e.g. Reuters article on Salar de Atacama). The life-time of the site is unclear, but even with a conservative estimate of 10 years, this will supply 130 million car batteries, approximately 0.2m2 per car, as for the Baton Rouge site.
The main concern at the Salar de Atacama site appears to be the rate of extraction brine, which is lowering the local water table. If this is restricted, there will be limits in Lithium supply. This is no doubt one factor leading many countries to invest in open cast mines.
The Thacker Pass open cast mine referred to above is expected to have spatial footprint of 23 km2 (close to that of the Atacama brine pools) an produce enough Lithium for 100 million cars (see mining.com -- 3.1 million tonnes of Lithium carbonate implies 0.6 million tonnes of Lithium), and hence 0.23 m2 per car.
The Thacker pass mine will not use brine pools, but there is concern in the environmental impact assessment that there are unknown impacts because of new methods being used.
Energy Use
Finally, on energy efficiency and CO2 emissions. The Nissan Leaf is rated at 114 mpg-e by the EPA , or 19kWH per 100km. If this electricity is taken from the UK grid, with 233 kg/kWh implies 4.27 kg CO2e per 100km.
For petrol, with 2.3 kg CO2e per Litre, we have closer to 13kg CO2e per 100km. So, even with today's electricity supply, the electric car is doing significantly better.
As new generation capacity is dominated by carbon free energy sources, this balance will change increasingly in favour of electric vehicles.
Conclusions
(1) Pollution associated with Lithium mining is potentially significant. Care needs to be taken with water usage and disposal of waste water. These are not new challenges, but if this is going to be done in a "green" way, there needs to be a significant improvement in the quality of waste management compared to current standards in the fossil fuel industry.
(2) Lithium battery production is a clean and compact process.
(3) Electric vehicles have a lower carbon footprint than comparable petrol vehicles if charged using energy from the UK national grid. The carbon footprint will continue to decrease as the grid moves increasingly to zero carbon emissions.