Red worms (Eisenia Fetida) are non-native to the USA, but whether they are harmful is difficult to say.
Earthworms in general can reduce the thickness of the layer of organic materials on the ground and thus change the amount of available nutrients. It's certainly possible that
this drives away plants that require specific conditions for reproduction or survival. For example
this article of the Minnesota Department of Natural Resources
describes some of the negative effects caused by non-native worms in Minnesota's hardwood forests
Invading earthworms eat the leaves that create the duff layer and are capable of eliminating it completely. Big trees survive, but many young seedlings perish, along with many ferns and wildflowers. Some species return after the initial invasion, but others disappear. In areas heavily infested by earthworms, soil erosion and leaching of nutrients may reduce the productivity of forests and ultimately degrade fish habitat.
The article also mentions that
they [earthworms] are not known to survive Minnesota winters. However, if they or other species are able to survive winter and escape from compost piles they could further harm native forests. If you have a compost pile in a forested area, do not introduce additional non-native earthworms. If you are concerned about spreading non-native worms with your compost, you can kill worms and their eggs by freezing the compost for at least 1 week.
Another interesting article on the matter is this blog on redwormcomposting.com. It lists some of the research findings which seems to be inconclusive on red worms. It then goes on to say that it's unlikely that red worms would thrive in the wild, at least not in Northern US states.
So in short, red worms can potentially be invasive but so far there is no evidence that they are. If you live near a forest you may want to play it safe and not start an open vermicompost heap. You would also need to make sure you kill all worms and eggs before spreading the vermicompost in your garden.