Do you currently have any leftover cash to invest each month? If so, how much? How much do you put into your 401k or IRA?
My DH and I graduated in 2014 with over $150k in loans between us, but we had a salary of $110k total. We refinanced our loans (to 4.9%, 5%, and 5.3%), and had a $2k min payment each month. However, because of our high income, we were able to max our IRAs and 401ks, and still have $1k leftover each month to pay extra on our loans (for a total payment of a tad over $3k). I am very, very glad we prioritized saving equally to paying off loans (this only worked because of the interest rates we got). We hardly put anything in a brokerage account, anything leftover went to loans, so our investments are almost entirely in our pretax 401ks and IRAs.
We went from $150k in debt and ~$1k in assets in 2014 to now, where we have $200k in investments ($230k in assets total) and just $40k in loans. Once your networth tips into positive territory, and is weighted more towards assets that grow in your favor instead of against, suddenly things start accelerating, and it is a great feeling.
any increase in interest rate on your loans will hugely impact the amount of interest you pay. And, like others have said, you don't have any asset behind the loan that you can fall back on if there is a crisis (not like a mortgage or car loan), so it really depends on how secure your job REALLY is (as in, how easy and fast would it be to find another similar job if you lost yours, in a downturn?), and how risk-adverse you are. A 4% fixed loan is actually pretty great, but I wonder if there is a way you could invest more by slashing some of your other living expenses instead of changing the terms of your loan.
edit: time in the market is what is important. If you are entirely holding off investing because you can't afford it until your loan is done, then I would consider an adjustment, because being able to invest a solid chunk of change in your early 20's will give you the maximum amounts of doubling (look into the Rule of 72). But if you are currently able to sock something away in your 401k, then making significant changes may not be as necessary. I hope that makes sense.
For us, we still have one more year for our loan repayment schedule, THEN we will begin dumping money into an after-tax brokerage account. This whole time, we have been investing heavily in other vehicles, but not a brokerage account. That comes after loans are done, not before. So if you are intending to open a brokerage account with freed up money, I would say that's not worth it, but if you want to free up some cash to invest pre-tax, that could potentially work. A case study may help.