Yes, it matters, a little, but it also depends on your location.
In your car, the process of converting fuel to movement is pretty inefficient (about 25% to 30%, according to this). That movement (kinetic energy, if you're a physicist) is then converted to electrical energy with an efficiency of 75% to 80%, giving you an overall efficiency of 18.75% to 24%. You might also consider the inefficiency of getting the fuel to the service station that you bought it from.
Using more electricity in your car means a higher physical resistance experienced by the engine when it tries to turn the core of the alternator. ie it's harder to turn the alternator when more electricity is being drawn from it, and thus more fuel is used to keep the vehicle at the same speed, unless the vehicle is idling, or overrunning the engine downhill.
In your home, electricity is generated with varying efficiencies depending on the generation that supplies it. Transmission losses in the US are apparently about 6.5% (ie 93.5% efficient). It looks like an average for generation efficiency would be around 39%. This last link also says the overall efficiency of getting power to the consumer is about 33%, which seems about right, given the previous two figures. The efficiency is somewhat higher than when you use electricity from your car.
It's better to charge your phone from home.
Of course, it very much depends on how your electricity for your home is generated. If you use an electricity retailer that leans towards renewable energy, it's going to be much better for the planet. It's clear there would be very few cases where the electricity grid generation was so inefficient that your car did a better job.
All that said, cellphones don't use much power. It's probably not worth worrying about too much. For most people, there are much bigger energy efficiency improvements that can be made through actions like improving home insulation and heating/cooling systems, reducing use of hot water, and analysis of home appliance electricity use.