I think you can accomplish both goals at once. You don't have to eat pounds and pounds of organic produce or $9 green juices to be healthy. Sure, you should eat some fruits and vegetables, but that doesn't have to be 100% of your diet. You also don't have to spend hours in the kitchen every day unless you really want to. Healthy food can be simple and cheap.
Carbs are not bad. Refined carbs are bad. "Carbs" that are actually more fat than carbs (like donuts and pizza) are bad. But whole grains and beans are healthy, cheap foods. And I think potatoes have an undeserved bad reputation. Sweet potatoes, in particular, are very healthy. There's an Australian guy who calls himself "Spudfit" who ate nothing but potatoes for an entire year and lost a lot of weight and improved his cholesterol, etc. Of course he ate them without the sour cream, bacon bits, etc. that people like to add. Those toppings are what get us into trouble.
One resource that might help you is drmcdougall.com. He's kind of like Mr. Money Mustache in the sense that he has "extreme" ideas that are not really that extreme once you see where he's coming from, and he tells it like it is and doesn't think excuses are helpful. His information is also completely free, and there is even a forum. He also has very simple recipes that his wife has been refining for decades.
One thing that was useful to me was to figure out whether you're a "one toe in the water at time" person or a "cannonball" person. I have very slowly, very gradually changed my diet over the past few years, but I actually think the times when I changed one big thing all at once were easier for me, and if I were to go back to where I started and do it again, I'd just make one huge overnight chnage. It was easier to say "No cheese, ever again, for any reason" (barring genuine starvation, I suppose) than to allow myself to have a little bit once in a while. If I allow myself an occasional indulgence, then I'm always bargaining with myself about when that indulgence is allowed, how much is allowed, etc. And then there's the guilt afterward. But some people feel trapped by that "never again" declaration. So it helps to know yourself.
While you're figuring out what else to eat, or not eat, consider excluding oil. It's one less thing to spend money on, and it will save you a ton of hidden calories. (I don't think counting calories is the answer, but if you pay attention to the calorie density of foods in general, you can have a huge positive effect by doing your best to lean toward less dense foods, just like aiming for better "bang for your buck" items when spending money without scrutinizing every penny spent.) Oil is extremely calorie-dense. It's easy to add several hundred calories a day just from oil without even realizing it. Cathy Fisher has some recipes for delicious salad dressings that are oil-free and easy to whip up in a blender. They store in the fridge for at least several days, or you can freeze them:
http://www.straightupfood.com/blog/2013/07/03/vinaigrette-salad-dressings/If you have, like most people, a lot of recipes that start with sauteeing onions and/or garlic in oil, you can just skip the oil and use water or vegetable broth to avoid sticking.
Sorry to go on so long... this post combines two things I'm really interested in! I hope some of this is helpful.
Oh, one more thing! Your tastes really do change. Think about the first time you had black coffee or beer. Gross, right? If you stay away from unhealthy foods and keep offering yourself healthy foods, even if you have to just have a bite or two per meal like introducing new foods to a picky toddler, you really will start to like things you never thought you'd like. I still hate beets, so I think some things will just never change, but in general, I like things much less sweet and salty than I did before, and I love so many new and different foods that I never thought I'd be able to eat! Sometimes it's a texture thing, too, so try foods in different forms. If you hate beans, try hummus, for example. Try a new vegetable raw or cooked, etc.