Think of the recommended R values as being for "overhead insulation" instead of "for ceilings". It doesn't matter if there's only 6 inches of depth to fit that insulation because you have a flat roof or or if there's 100 ft of empty overhead depth to fill. The recommended R value is the recommended R value so don't get hung up because it says "for ceilings". They're talking about any overhead space with a roof on the other side.
Now obviously, there are constraints to achieving that recommended R value, with space being a big factor. Really, flat roofs suck for a number of reasons, so if it were my house I'd try to add some depth to the ceiling for insulation and reframe the roof structure there at the same time to eliminate the flat rubber roof. You could do a single pitched "shed" roof pretty easily that would be higher next to the house, and lowest at the outer wall. You could also do actual pitched trusses which could give you an actual attic for extra storage and more space for insulation at the same time. Either of those options will increase the budget, but they're probably the best way to get a decent end result here. If you want the best end result, then redoing the roof structure to eliminate the flat roof and add space for more insulation is the best route not only for comfort but also for a better performing roof and potentially more storage.
Insulation types:
Closed cell spray foam is the best. It air seals, insulates with a high R value, and provides it's own vapor barrier at the same time (when applied 3" thick or more). This could also theoretically be applied directly to the roof of the garage in whatever amount of space is there, so that the floor is insulated too. It's great for irregular spaces. The downsides are that it's not really DIY friendly and it's costly. If you've got very limited space to fit insulation, it's the best option. But it's really not cost effective when you start applying more than 3-4 inches of depth as the outer 2-3 inches do almost all of the air sealing and insulating work.
Rigid foam panels are a close second that are more DIY friendly. I'd look for the Polyiso kind since it's the same foam as closed cell spray foam. You can cut them to fit tightly between framing with super basic tools, and if you want to go the extra mile you can use cans of spray foam to seal the edges. It's great at reducing drafts and gives lots of R value per inch of thickness. You can often find these panels on local CL type sites for a fraction of what the box stores sell it for.
Batts are common and the cheapest option. Very DIY'able. They're also the least effective insulation. They do little or nothing to stop drafts, and they have lower R value/inch than the foams.
Blown in cellulose is the cheapest option per inch of thickness, but you need a lot of it so it's only really a good option if you're reframing the roof to have a bunch of attic space above. But it could give you the recommended overhead R value for the least amount of money relative to the other insulation types.
So again, if it were mine, I'd really think about redoing the roof framing over that room to add some overhead space and get rid of the flat rubber roof at the same time. That will probably give you the best end result in a number of different aspects.
If that's too much, and you're sticking with retrofitting the existing envelope then I'd tear that whole room down to the framing to see what's actually there, move any plumbing/electrical that you want to, and then add as much closed cell polyiso foam as I could fit in a given space (whether via rigid panels/DIY, or just paying to have it professionally sprayed). Batt insulation would only be a last resort for me, especially if you won't be able to get to the recommended R values for other parts of that room.
For your question about vapor barriers, the vapor barrier typically goes on the side of the wall that is warm most of the time. If it's a cold climate, that would be to the inside of the insulation, and if it's a warm climate it would be to the outside. In zone 4 where you are, it's a bit of a toss up. And if you use closed cell foam, then you don't have to worry about it.