In a tax advantaged account (such as an IRA), just re-invest the dividends and re-balance as necessary.
In a taxable account, I suggest not reinvesting dividends - redirect them to your bank account or to a money market fund. Then, when the dividends show up, use them as part of a re-balancing. You have to pay the same tax on dividends whether or not you re-invest them right away, might as well use that to keep your asset allocation (and thus, risk) where you want it to be.