It costs libraries something like $10-15 to fill a single ILL request, counting staff time on both ends plus return shipping costs. (I refer to true ILL requests here, not requests that are filled by other libraries within the same system, which benefit much more from economies of scale.)
It's a great service, and people should absolutely feel free to use it when the local collection doesn't fill their needs. I wouldn't personally mind paying a $3 fee, given that it's not even close to covering the cost of the service, but I'd be a bit concerned about it for other reasons. If a public library feels it has to charge for this service, that's a sign it's not funded very well in general, so I might lobby my municipality for better library funding in my community. How supportive councillors are of the library can *really* vary, and is definitely affected by what they hear from voters.
[ETA: well-run / well-funded libraries do keep the cost of ILL in mind when looking at a request, and if it's for a book that's currently in print, they may just buy a copy, especially if it looks like an item others might be interested in too. ILL requests from patrons are treated a bit like purchase suggestions, in terms of collection development.]