The advantages to VTSMX would be not having to worry about the bid/ask spread, and the ability to purchase fractional shares. When you place an order to buy VTSMX (or VTSAX) you will get a price equivalent to the net asset value of the fund at the end of the trading day. All of the money will be invested.
With VTI you bid for shares like any regular stock, and the price you get is whatever you and the seller agree to. Depending on the price, this might or might not use up all your money. For example, if you want to invest $500 into VTI, but the price of a share is $60, you will buy 8 shares (8x60=$480) and have $20 cash left in your account. The advantage to VTI is the lower expense ratio.
The most you'd pay in extra fees in VTSMX is $12 per year, until you cross the $10,000 mark and convert to VTSAX.