Hi there, what you've run up against is the hard fact that gravel beds are not actually low-maintenance.
For exactly the reasons you stated below, accumulation of organic matter and whatnot, weeds will get in there and are a serious PIA to get out again. You can go the old-school Duke-Nukem road and spray it all down with herbicide on a regular basis, but since it's the 21st century let's not do that. Or you could painstakingly pick the weeds out by hand, moving the gravel around as you do, but that's tedious, so let's not do that either.
What you might consider is removing the landscape fabric entirely, and planting the areas up with a suitable perennial mix for the spot. Nature hates a vacuum, and weeds will occupy empty spaces. By planting the area up, you're shading the ground and outcompeting the weeds with desirable plants. You could probably even leave the stones in there.
Google the New Perennial Movement and Piet Oudolf online to get inspired, the new style is mixing compatible groups of perennials to simulate nature in a visually appealing way. In the end it's super low-maintenance because it looks after itself, good for nature, and looks good too.