I don't think there's any harm in switching to XUU, but there's no rush to go there. I would not buy a small cap fund to make up for VFV, it will only serve to needlessly complicate your portfolio.
Take your time, focus on saving first, and learn as much as you can about portfolio structuring and risk tolerance. Your biggest enemy in the investing world is not being able to stomach a loss that your portfolio can realize. The next biggest is believing you can stock-pick your way to success.
Once you've got that under control, you are pretty much guaranteed to do better than most.
Going 100% stocks is ok when you're starting out and new contributions form a relatively large portion of your total portfolio. In a market downturn, the new contributions will give you a nice boost as you buy at lower values and meaningfully bring down your Weighted Cost Average.