Some thoughts:
1. Can you play BIL Bingo? Make a list of the annoying things he says/does, and each of you gets 5 on the list. The first one to hit all 5 has to find a way to work the word "bingo" into conversation, without being noticed.
2. Just say no. You are right to defer to your wife in general, as it is her brother. However, this is also your house, you are an introvert, he is overwhelming, and you have the right to do what you need to to preserve your own sanity. Don't argue, don't engage, just repeat "no thanks, that won't work," or "no thanks, I'll just stay here." It is not being rude, it is not "telling him off," it is simply setting a boundary and then sticking to it.
3. Next visit, talk with your wife in advance about what boundaries to set. You can say you are concerned because he causes her so much stress, that she swallows a bunch of stuff and then blows up, and what can you do to help set some boundaries so she doesn't get to the breaking point? E.g., there is NOT A DAMN THING WRONG with telling him to rent a car or letting him borrow yours -- he is a grown-ass man, if he wants to get himself somewhere at a time that is not convenient, he can drive himself. That is actually the very first boundary I would set, because there is nothing like forcing yourself to rejigger your schedule at the drop of a hat, with small children to deal with, to magnify all of the other problems.
4. Where I would call him out is if he insults your wife's cooking. I don't care how close you are with someone, that is unacceptable. The key here, though, is being calm and direct, not angry and attacking -- e.g., "please don't insult my wife's cooking, she worked hard to make this for us." And that's it -- again, don't engage, don't escalate or yell or insult, just calmly repeat the same phrase.
5. Read 1-2-3 Magic -- for him, not your kids! He is acting like a toddler, so treat him like one. Try to see it as less ego and more insecurity; how tough must it be to live in his head when he has to talk all the time because he is so terrified to listen to his own thoughts and can't tolerate the idea of not being the center of attention for one second? Good Lord, that sounds exhausting and miserable. [This was my DD, btw: she put on a huge show about needing to control everything precisely because she felt so fragile inside] The image that stuck with me from that book is the parent as the horse, and the kid as the horsefly buzzing around, and the horse just doesn't even notice it, just periodically swats its tail to knock it off and goes on grazing. You can't let him see that he's getting to you, because that is the payoff he is looking for -- it tells him that he matters, that he has the power in that situation. The more you can set a boundary in advance, and then stick to it, and then not engage or argue when he pitches a hissy about it, the more you establish yourself as the one with power -- and the more likely he will be to fall into line in the end.
I want to say this again, because I think it is the single-most important thing: when you either give in to his demands OR refuse but let him get you angry, you are giving him the power to control you. He pokes, you comply; he pokes again, you yell. That is exactly what he wants. The only way to reclaim your power is to have clear lines for what you will and won't do/accept, enforce them consistently, and never ever ever let him see that he is getting under your skin. No yelling, no arguing, no engaging at all -- just a calm, repeated response that enforces the boundary. [Hint: if you find yourself getting angry, it means that you have let him slide too far over the boundary already] Added bonus that not being able to get a rise out of you will drive him completely bat-shit crazy.
6. Expect giant fireworks and hissy fits regardless of what you do. When someone like him realizes he is losing the ability to get under your skin, he will immediately ramp up the behavior to get a reaction. You have to expect that and be prepared for it and let it blow over -- if you stick to the plan and don't cave on his stupid demands, it will blow over and he will learn to behave better. Or he'll decide that you're not worth his time if he can't re-establish his dominance and go pester someone else. So, win-win.