I don't know that there is a right way. Been a while since I did it as well, and really some of the hassles and the worries have made me simply use a broker or a dealer ever since.
Title is usually transferred by simply signing the back and then taking it to a DMV station that works with titles. There will be fees, which you can check online for in most states. Carfax helps, but obviously is no guarantee. I've used a cashier's check. I would never use cash. No solid paper trail and if you get a seller who wants to play games and claims you never paid for the car it becomes your word against theirs. Obviously it is harder once you get title transfer done, but still.
I'd never buy a car online without seeing it and driving it from a private seller, so if you mean would it be OK to see one online and offer to pay for it sight unseen, for me, no.
You can have a mechanic check it out before you complete sale. I've done that sometimes. If nothing else, how open the seller is to this will tell you a lot. Obviously a car with problems, they'll do everything possible to discourage you. I've used sites like TrueCar and they offer used cars now, so that might offer you some added protection.