Worry? Of course not!
Small cap stocks rise on average a little more than large ones; they have more upside, more room to grow. In any given year the trend may not be evident, but in the long term you should get a little bit better return if you increase your percentage of small-cap. So you would probably benefit by increasing your small-cap percentage.
That said, the drawback of small caps is more volatility, including a greater risk that individual firms go bankrupt. So perhaps too much small cap will disturb your sleep.
In the end, it's your decision. Whatever percentage you choose, you will get a better result if you rebalance each year to maintain the percentage in each category, unless you have significant buy/sell costs.