It's sometimes said that you can't save your way to FI, you also have to focus on increasing earnings. However, with DP money, you don't want to risk market fluctuation, as FG pointed out. So one way to think of it is that you can develop black belt frugality, and add those savings to your existing DP fund. Focus on increasing your earnings and invest the increases into your retirement accounts with the goal of maxing out your accounts. Before you know it, you'll have the house you want and a healthy start on FI. Once you do those things, focus on building up your taxable accounts so you can RE if that's your goal.
ETA: TL;DR version - Save for a down payment, invest for the future.