If I wanted to sell shares of VBTLX (at a loss) and purchase shares of VGIT, for example, how do I know whether the overlap is enough to trigger a wash sale rule? Would Vanguard warn me, or is it entirely up to the individual to figure it out?
If you're buying or selling the same security, most brokerages including Vanguard will apply the wash sale rule for you. You'll notice this on your 1099 when numbers start appearing in the Adjustments column and "wash sale" in the comments column and code "W" appearing on there.
Since you're buying and selling different securities, I'm fairly certain that it will be entirely up to you to determine whether the securities involved meet the IRS standard of "substantially identical". If they do, you'll very likely need to make your own adjustments on your tax return, and remember the adjustments you make for future purchases and sales.
Note that a wash sale is not a violent or dramatic tax situation. If you have a wash sale, all it means in most cases is that some of the loss is postponed into the future. You're not going to get arrested or be audited or have extra penalties or interest as the result of creating a wash sale.