BTDT. You are in a no-win situation, because any punishment just verifies her view of herself as the victim.
So, 1, counseling. You guys need a reboot, a new way of interacting. FWIW, we framed it as "family" counseling -- in my case, I wanted DH to learn better ways of dealing with her, so I bullshitted *him* into thinking it was about *her*. You can do the opposite.
2. 1-2-3 Magic. Easier with little kids, but the key bit is that it forces you to respond *before* you lose your shit. Do not yell, do not get visibly angry, do not let her see you sweat. The more volatile the kid, the calmer you need to be. When you lose it, they feel *less* safe -- because they know you are supposed to be the rock, and so if they can poke and prod you into moving, well, what do they have that is stable?
3. Physical checkup. Doctor is good, but also do you own assessment -- sufficient sleep, healthy foods, etc. Is she in the wrong classes at school, is there an issue with friends, etc. Also, I see you said she is 12 -- OMG, yes, THE worst age with us. With my own DD, the 6 months before she had her first period were sheer hell, and we can still tell when it's coming on by the pretty dramatic change in volatility. So if this is a recent uptick, it is very likely hormonally-driven. Not that it excuses it -- as I've said to my own DD, yes, it totally sucks and it's not fair, but you are going to have these intense feelings and mood swings for probably 40 years or so, so you need to learn ways to recognize and deal with them. But have a little empathy and treat her like you used to when she was a toddler -- expecting her to be calm and pleasant when she has PMS is like taking a hungry toddler to the mall during nap time. :-) So if/when you can identify a pattern, try to set her up for success during those periods, with lower expectations, fewer plans, some nice calm reading time alone, or family movie time, etc.
4. Catch her doing something right. This is SO the hard part!! But right now you are in a riff where the only way she is getting attention is by misbehaving, and so since she craves attention, it creates a vicious cycle. You need to pair giving her less attention for the bad stuff (the 1-2-3 Magic recommendation) with more attention for the good stuff. And I mean any little thing -- if she manages to storm off instead of yelling at her brother, that is a win, let her know you noticed how she kept control of herself around him.
5. Give her more responsibility. When my DD gets into a rut of being an entitled nasty creature, I figure I have made life too easy for her, so she gets more chores. But the key is the chores are "grown-up" chores that help her feel independent and responsible and grow her skills. E.g., when my own DD was insufferably 12, it became making dinner for the family (which I chose for DD because she has always been interested in cooking -- and this week, btw, my now-16-year-old is making us dinner every night!). The first few times, I was working at home but in the other room, so she could figure it out herself but come get me with questions, and it was actually good one-on-one time. Added bonus is that it gives you an opportunity to catch her doing something right.
6. Find ways to spend time one-on-one with her. Make it fun -- you guys need some time to remember that you love each other and just relax.
7. Look for opportunities to let her open up. If you can find a calm time, you can ask if there is something going on, how she's feeling, because she is too nice a person to act this way, etc.
8. Cut way back on the criticism. I know this is counterintuitive when they are acting like total shits, but she is a walking open wound right now, and every hint of criticism is pouring salt in it. I drew the line at hurting her brother or saying nasty things to us -- that happened, she was gone immediately, no questions, no arguments (again, 1-2-3 Magic), and the rest of us continued on with whatever we were doing. But general whining, complaining, etc., eh, water off a duck's back. A/k/a pick your battles -- and when you do engage, do it with as few words and as calm a manner as possible.
9. Pay attention to what she does, not what she says. My DD tends to be very verbal, and I have noticed that she will vent and kvetch and complain to me -- and then she gets it out of her system and goes and does whatever it is she is supposed to do. This happens a lot with tests -- so the drama, "I'm going to fail!!!" and then she does just fine. If this is how she is built, you might need to work on your ability to take all the words less seriously. I found it almost impossible not to respond, because I had advice! I could make it better! But the talking was just her way of venting stress -- she didn't want me to fix it for her, and in fact my "helpful" responses just made it worse, because she read it as me not having confidence that she could solve her problems on her own.
10. Always send the message that she is a great kid and you have confidence she will get through whatever it is. Most especially when you are despairing that she will end up in jail for fratricide. :-) The only way to replace the voice in her head telling her she is worthless and bad is yours telling her she is good, even when she is bad. E.g., my "you are way too nice a kid for me to let you be mean to your brother" comment above. Or "you are way too competent for me to let you get away with not cleaning your room." The point is that she is not living up to her own standards.
11. Be a Smother occasionally. Ever seen The Goldbergs? I spent the first season laughing at the mom, thinking, OMG, I never want to be the mom who can't see her kid's faults. But then I realized that sometimes, that's exactly what our kids need. They have a whole world telling them they are stupid, fat, lazy, whatever; they need us to be the ones who tell them they're perfect and take their side [even when they're wrong] and say "you got this." I did this one time in the car, and I truly think it was a turning point. DD and I got into a squabble about missing assignments or something, and she was feeding me lots of excuses that I knew were crap, and we went from having a nice conversation to total snark to pissy stony silence in like 45 seconds. And I said to myself, crap, I really blew that. What would the Smother do? So I said, OK, I really blew that, I'm sorry I jumped on you, let me that again -- I know you've got this, so what's your plan? I asked if she needed me to help out, or if she wanted to go to her teacher herself to figure out the missing assignments -- "Because I'm your Smother and I am perfectly happy to go marching into that school and ask your teacher wtf she thinks she's doing not giving you the makeup work." [Of course, I also knew that the idea of her mother storming into school was the *last* thing DD wanted] And we ended up having a great talk, and she was actually giggling by the end.
12. Realize there's only so much you can control or fix. My DS is my DD's favorite punching bag. That is where I draw the line -- I tell her that I wouldn't let anyone treat her that way, and so I am not going to let her treat him that way. But, my God, it's like spitting into a hurricane sometimes. So I also had to acknowledge that she is going to reap what she sows. I can't turn her into a warm fuzzy person; she has an edge to her, and she always will, and in fact that is part of what makes her great. So if she is a shit to her brother, she is just going to not have a good relationship with him when he grows up. And it sucks, but in the end that's her decision to make. She is hardheaded and so determined to be independent that she will bullheadedly insist on making her own mistakes instead of learning from mine. It kills me to see her doing stupid stuff, but as long as it isn't stuff that will take her totally off-track, I give her the leeway to make her own decisions.
13. Make sure the little brother isn't poking the bear. Siblings definitely have their own language and relationships, and angelic little brothers have been known to intentionally annoy their older siblings in the hope that sis gets caught and gets busted. Not that that excuses nastiness, but it's the difference between removing one vs. removing both.
Sorry you are going through this. It sucks, and there is no 100% effective fix. But I have to say, life has gotten a lot smoother for us, knock on wood -- DD craves independence and competence, and so now that she is getting older and I am giving her lots of free rein and treating her more like an adult, the nastiness has really cut down quite a bit.