If you don't have reactions to them, consider adding back in: rice, legumes, potatoes. These are all cheap and filling, and fit the "whole food" bill. The research and experimentation I've done with legumes tells me that lentils are A-OK. Kidney beans not so much, and black beans are proceed with caution (must be soaked and thoroughly rinsed). I'll also do hummus and don't seem to have a problem with the chickpeas. My paleo budget was also running out of control, and adding these items back into the menu a few meals per week really helped.
Also, don't be afraid of eating plenty of fat. It is cheap by the calorie. I keep a jar of coconut butter in the cupboard and will just eat a spoonful, maybe with a piece of fruit if I need a snack. I also make little tart thingies that are coconut oil based and keep them in the freezer. If you do ready-made paleo snacks, I've discovered that Kind bars are pretty easy to replicate. Just google "copycat Kind bars" and you'll find a good article.
Also agree on the CostCo rec. They have the best price for coconut oil, and also have good prices for nuts, olive oil and frozen berries. The one here has recently started carrying Kerrygold cheese, if you eat dairy.
For meat, use your crock pot and buy inexpensive cuts like pork shoulder and chuck roast, and slow cook them for a multi-day supply of protein. Buy whole salmon; it's usually cheaper and the butcher is happy to fillet it for you. For instance, I just bought one for $4.99/lb whole, whereas fillets of the same fish were $12.99/lb. Buying the beef quarter or a freezer lamb is also a great option if you have the space and the access to local farms. I've gotten around 140# of grassfed beef, cut and wrapped to my specs for around $400. That would be a lot for one person, but maybe you can ask around and split it with someone, or go for a lamb instead.
For produce, I grow quite a bit, and also use Bountiful Baskets every few weeks for a big basket of fruit and veg, plus whatever one-offs they have that week. They do have organic baskets available, though I just go with the conventional stuff.