Hmm, my take is slightly different. I'd place heavier weight on WHAT and HOW OFTEN I am buying items over WHERE I buy them. I don't think it matters much where I get bulk beans 2x/year, for instance. Other things that Walmart sells, like their cheap and unethical clothes and mass meat, don't make me feel very good about my purchases.
Funny enough, a few years ago I had the EXACT situation as you- a Harris Teeter <1 mile from our house and a Walmart a few miles away. I rarely went to either, though- did my shopping at Aldi, which necessitated fewer trips overall but satisfied my ethical/price needs (I love their business model!). I ended up going to HT for the occasional "oh shit, I need cilantro for this recipe" moments. I'm betting you could do something similar in your area? Aldi, Costco, etc for 90% of food items?