I want to do exactly what you describe, though I lack a portable sawmill. How much experience did you have before you started this project?
The portable sawmill is a huge money-saver, but it's also a huge time-suck. I don't care to count up the hours I spent felling and dragging trees, sawing and stacking lumber, and maintaining the mill. It's another hobby altogether, so we'll just ignore that and assume you're working with store-bought lumber.
I have quite a lot of experience with carpentry and woodworking. I grew up on a small farm where we built lots of barns and sheds. I've also renovated a couple of houses and done some volunteering with a Habitat for Humanity-type organization. But I really don't think a huge amount of experience is necessary to do a decent job of it. I would strongly suggest using plans and watching YouTube videos if you're coming at it with no experience. The plans will really help you visualize how things go together. Start with something small, like an 8X10 storage shed, and work your way up. You will learn a lot just by using the tools and making a few mistakes (best to make your mistakes on a smaller project).
Don't worry about how long things take at first - just focus on doing a good job, and the speed will come as you're always anticipating the next step. And if you have a good friend who is a decent carpenter, maybe a nice bribe of of a few cases of craft beer would help. Learning from someone one-on-one is invaluable. I once dated a girl whose father was a seasoned home-builder. I learned more in a weekend from him that I could from a decade of This Old House. Really would have loved to spend six months on one of his job sites.