First, you should be proud that you're making progress on student loan debt and living within your means!
Second, you've still got a ways to go on those student loans. You're still in a hair-on-fire-debt-emergency!
I have been in your shoes in a sense. Made big progress, knock 50% off my debt, suddenly I've unleashed an extra $100 or $200 in my monthly budget from what used to go to payments or stuff I didn't need. And The Thing, whether it's a nicer house/apartment, or new clothes, or whatever material thing suddenly appears like some sweet goddess to take all my newly liberated money and solve my problems. And as I get close, you realize it's a Medusa with snakes everywhere and fangs made of credit cards. Run before it's too late!
With some shopping I guarantee you can find a place that comes a few hundred dollars beneath your BAH budget, still meets your needs, and still allows your husband to bicycle into work. I think location, not amenities should be your primary concern because commuting costs, and costs to do the things you do all the time (errands, groceries, playdates, frolicking with your kids in the park, sitting on a park bench sipping coffee you made at home relishing your financial badassity) will outweigh small differences in housing amenities. Once you've got the location right, get the minimum fanciness that meets your needs, and reap the rewards of your financial maturity.
Consider: if this house you found is a dream, and your husband can bicycle into work, why not sell the second family car that's paid off? Gas / insurance / registration / depreciation / opportunity cost together, that's probably worth $100 a month in operating costs or more even if the car was sitting unused. This could make the "full BAH" house a better financial deal than it seemed before. Ditto if you can walk from this house to your regular activities/errands/shopping.