OP, despite my warning above, I would encourage you to read up some on what to do then just dive in with minimal supplies. Get 10 or so small plants from a local nursery your first year to avoid dealing with seedlings, and just try some stuff out. I can get baby herbs, tomatoes, peppers, etc for $1.00 each here. Even if they're $3 a piece (try to find somewhere cheaper though), just try some stuff out. Don't get analysis paralysis and avoid getting started because you want everything to be perfect. Learn by doing! And reading. Dig up some ground, break it up, and plant some stuff! Use containers and some commercial potting soil if you'd rather.
If the first wave of plants dies off, get 5 more and try something different. Maybe put some in containers if your soil isn't great. Rinse and repeat until SOMETHING survives. Mint, chives, and oregano have always been solid tough to kill herbs for me. It's going to be a max of $30-40 per failure, a pretty cheap lesson. Don't worry too much about fertilizer, get some foliage centered fertilizer (nitrogen heavy) and use 1/2 strength to play it safe. Don't fertilize if you don't want to. Whether to use pest control is up to you, I have some family members who like Sevindust. I garden purely as a hobby on my apartment balcony, so I just let the bugs eat what they will, and spray with a bit of soapy water occasionally if they're bad. The first year, just focus on trying to grow some stuff. Next year you can worry about yields, special varieties, organic fertilizers, etc etc.