Rather than calculate a budget in a forward-looking manner as it sounds like OP is doing - I think I'll spend $X on insurance, $Y on travel, $Z on food, etc. - I decided to track what I actually spent in a backward-looking manner. I then approached the RE question this way: "If my actual spending divided by my current portfolio value is 100% historically safe, am I OK with that?" That is, do I feel comfortable with my current rate of spending, or do I want it to go up in retirement, or do I know it will go down for whatever reason?
OP, have you tracked your actual spending? For how long? How does it compare to your budget?
If you have tracked for several years and your budget that you're using to calculate your 4% number is higher, then I'd say you're fine. Especially since it sounds like you're assuming taxes stay the same after you retire. For me my taxes went from my largest single budget category to less than zero. Based on what you've written so far, I'm guessing that you will see some of the same effect.