So, I guess what you're asking is whether you should use some kind of average cost for your basis, or maybe FIFO.
No, you wouldn't. The shares are "identified" as being the ones you bought at a particular broker.
If you own multiple lots with different costs within one account, then this would be more complicated. As you've described it, your assumptions are correct: you're selling the same shares you bought at that broker.
However, keep in mind that the Wash Sale Rule applies to all your shares, wherever you hold them. So, other purchases within 30 days before or after will affect your ability to harvest the loss.