It's not necessarily correct to compare to rents of similarly-sized houses. In my experience you can rent much smaller spaces than you can cost-effectively buy. So just because one has a 1200 sqft house doesn't mean that's the amount of space needed, an 800 sqft apartment might be adequate.
Whether you should pay down the mortgage as quickly as possible or not really comes down to your risk tolerance. By keeping your percent equity in your home lower, it will free up cash for investing, as you are doing now with that $120k. This is a form of leverage and will increase your total rate of return on your assets, but it can be risky. On the other hand, you can pay off the house, but it ties up a lot of your net worth in an asset that's appreciating at only about the rate of inflation, and has relatively high expenses compared with the rate of appreciation. I personally feel that some leverage is a good way to make real estate make sense, but that many homeowners have been overdoing it.
If your goal is to pay off your house as quickly as possible, compare the net income you get from owning a house (a negative number since it's all expenses) to the returns you'd get by investing the house's value and taking a 4% SWR, then subtracting the rent (result could be positive or negative). e.g. a $300k house with $6k/year expenses, and $9k/year cost to rent would mean your net income from owning the house is -$6k/year whereas the net income from investing the amount the house is worth and renting would be $300k*4%-$9k = +$3k/year. Renting would be a better financial situation in that case. If the numbers are close, consider the effects of tax breaks that apply to those homeownership expenses, that may tip the balance toward owning.
If your goal is instead to use your mortgage as investment leverage, you'd want to make the decision as if you were a landlord and were renting the house to yourself. The easiest method there is to look at rental "income" per month and compare to the total value of the house. At the very least the monthly rent you'd have to pay elsewhere should be greater than 0.5% (1-2% is much better) of the total value of the house to justify owning it.