Because really, if you are new to riding you just don't know what you want yet. My husband loves tooling around town and going on long slow rides, or adventures on rougher terrain, so all he wants are steel bikes. I'm focusing on getting fast, so I upgraded to a pretty aggressive high end aluminum bike (the aforementioned CAAD10). If I hadn't put in time on my entry level bike first, I would have probably gone with a carbon bike like the ones you are mentioning, but after learning my riding style I knew I wanted something that felt more responsive versus smooth & easy.
I 100% think it is worth it to spend money on bikes (speaking as someone who has 8 built bikes + 2 frames in her house right now...) if its what you love, and cycling is a great hobby/sport/lifestyle, BUT you first need to put the time in.
It's OK to get scared descending. I don't like to let my speed creep much past 35 on descents. That said, you should be fine stopping on SF hills. Disc brakes are only really "necessary" (read: they're never actually necessary) on looooong descents where your rim brakes might overheat. Although I think that can happen with disc brakes too? Disc brakes are also good in cruddy or wet conditions, but if you are riding on the road you'll do just find without them with proper technique. If you're having a hard time stopping you may need to switch out your brake pads or adjust your brake levers.