Hi! Local here!
Yes! there are actual grocery stores in downtown. Depending on where in the city your'e staying look up Shaws, Star, or Trader Joe's.
Boston is very walkable, even with a child. A day in the Public Garden's is very free and fun, also nearby would be the Charles River Esplanade and the Boston Common. For non-walking travel you'll want to pick up two plastic Charlie Cards. These can be loaded with money and used to travel about, if you think you'll travel a lot, then pick up a week pass instead (from any MBTA vending machine - it is a paper ticket). You definitely don't need a car, but you might consider doing something like Relay Rides to go to Providence and back it would be $42 round trip to take the commuter rail to Providence and back, which does at least take you directly from one city center to the other, but I'd still check Relay Rides (its like craigslist for zipcar).
I'm a big fan of Providence over Boston, its smaller and thus more walkable, and I LOVE the Rhode Island School of Design art museum and highly recommend it. Good luck with your travels!