Being vegan will really reduce your environmental impact. It takes huge amounts of water and space and nutrients to grow cattle and other animals for food. I think it is something like one thousand cups of water for every cup of milk.
But, if you really want to cut down your impact, you should also only go for seasonal vegetables and fruits, so that they don't have to be transported long distances.
Unfortunately that tends to be difficult on a diet which may specify particular foods at particular times.