Thank everyone for the helpful comments.
The listing agreement between the seller and and the listing agent absolutely matters to the buyer. The flow of money is from (1) buyer pay total price to seller, (2) seller pay 6% commission to listing agent, (3) listing agent pay 3% cooperating commision to buyer agent. Clearly if the buyer agent is not present, the buyer should ensure the seller is aware of that, and reduce the commission paid to listing agent in step 2 to 3%, thus increasing the seller's net.
Acutally how agents split up the fee is usually specified in the listing agreement. See paragraph below from Northwest MLS form 1A-SWF, typically used in Washington State:
4. COMMISSION. If during the Listing Term (a) Seller sells the Property; and the buyer does not terminate the agreement prior to closing; or (b) after reasonable exposure of the Property to the market, Firm procures a buyer who is ready, willing, and able to purchase the Property on the terms in this Agreement, Seller will pay Firm a commission of (fill in one and strike the other) _________% of the sales price, or $ ____________________ (“Total Commission”). From the Total Commission, Firm will offer a cooperating member of MLS representing a buyer (“Selling Firm”) a commission of (fill in one and strike the other) ________% of the sales price, or $ ________________.
So clearly, if the buyer comes without a buyer agent, this paragraph should be modified to reflect 0% cooperating commission. Of couse, the seller and the listing agent are still free to negotiate a new total commission percentage, it is unlikely the seller would still agree to 6%. I feel it is this circuitous route which the buyer agent commission is paid allows the real estate industry to perpetuate the fallacy such as "buyer agent is free to the buyer", "if you go without a buyer agent, the listing agent would just keep the entire commission, and you won't save money".
In some states, if the buyer comes without an agent, the listing agent becomes a dual agent. In some listing agreement, the total commission is reduced to 4.5% for a dual agent. However, in such case, I have seen the listing agent suggesting the buyer to use a colleague of the listing agent as the buyer agent. Then the listing agent does not become a dual agent, and collectively, with his colleage, they capture the entire 6%.