Interesting! My conclusion is that there's no official rule stating hierarchy between a sailing vessel and a vessel under oars.
Long Technical Discussion.
Section II
of the Inland and International COLREGS covers the conduct of vessels in sight of one another, which would include your scenario
@Uturn. Rule 11 states the Section applies to all vessels, and the definition of a vessel
"includes every description of water craft, including non-displacement craft and seaplanes, used or capable of being used as a means of transportation on water.". This seems like a hopeful path, but ultimately peters out.
Rule 12 covers conduct between vessels operating under sail (which
@techwiz has covered), but doesn't explicitly include vessels under oars. Rule 13 says any vessel overtaking another vessel is automatically the give way. But then Rules 14 for crossing and 15 for head on situations explicitly state they only apply to power driven vessels. Rule 16 covers how to give way, and Rule 17 covers how to stand on, but neither of those are helpful because vessels under oars aren't included in Rule 18 which actually lists the hierarchy of who stands on and who gives way. At best, this section tells vessels under oars that they're the give way vessel when overtaking another vessel, but gives no information on when to stand on or give way during a head on or crossing situation.
Rule 25, in Section III does explicitly discuss vessels under oars, but that only explains what lights and dayshapes such vessels must carry. Rule 25
does lump sail and oars together. I suppose a prudent mariner could use that grouping to assume vessels under oars should follow Rule 13 for standing on and giving way to other vessels, but I personally feel it breaks down because the two methods of propulsion have very different levels of manoeuvrability and hydrodynamics.
At the end of the day, I suppose you can do what all the other yachties do, and scream STARBOARD, while vigorously ramming into your own well polish, well loved, and well cared for flotation. Usually they do this from port, the fuckers.