(Since others are addressing the issue of holding all your assets at the same brokerage, I'm going to assume that the OP actually did mean "fund:")
Should I be concerned that all our retirement accounts will be held in the same fund?
Because it is a target-date fund, no, you need not be concerned: that's exactly what target-date funds are for.
If instead your single fund were something less likely to be an appropriate asset allocation by itself, such as a sector fund (or, $DIETY forbid, an individual stock!), then you would want to be concerned.
If you had said you were 100% VTSAX (i.e., total US stock market), that's borderline: less diversified than your target-date fund, but quite possibly diversified "enough." Opinion would be divided about whether you needed to add bonds and/or international stocks or not.