To expand on your basic point a bit:

Of the three major reasons for going vegan — ethical, environmental, health — ethical (avoiding the direct harming of animals) is the simplest (in a good way).

If you’re solely an ethical vegan, you may eat an unhealthy diet, and you may not always choose organic / gmo-free foods. You could eat any number of plant-based candy bars or french fries or drink yourself into the hospital with diet sodas. But you could not eat animal products or by-products.

If you’re a health-centric or environmental vegan, you’re reducing cruelty by not eating / farming animals, but that may not be a primary motivation.

If you’re solely a health-centric vegan and there were evidence that for your particular situation, eating a little meat / fish / dairy were healthier than being totally plant-based diet, then you could rationally abandon your vegan principles.

If you’re an environmental vegan and there were evidence that eggs from sustainably raised chickens were more environmental friendly than plant proteins (e.g. almonds), then you could abandon veganism in favor of protecting the environment.

If you’re an ethical vegan, you have the privilege of eating healthily or eating environmentally, but avoiding meat / animal byproducts is not a privilege. It’s not even a choice. It’s a mandate.