He forgets that it is not just the skipped Starbucks. It is also the cheaper commuting, skipping the expensive gym, buying cheaper food, searching for cheaper insurances, not taking up car loans and expensive mortgages on a house, not the expensive vacations. It's the whole package.
Otherwise he probably has a point about becoming great at your job. That should make you a more attractive employee for higher and better paid positions.