I actually tried this. I got an old brick phone that could do email, texts, and phone calls. I used it for about 2 weeks, which was instructive, but ultimately I went back to my smart phone.
First, there was the issue of phone numbers, which wouldn't be a problem if you were willing to switch your service all the way over. But my new Ting phone number got an insane number of spam calls, which was more irritating than push messages ever were on my smart phone.
Second, the interface was crazy clunky. It was from the pre-touch screen era with a flip out keyboard, and navigating to email or texts took several clicks and menus and whatever, and I was constantly annoyed at how long it would take to respond to a single text.
Third, I realized the world is really really set up for smart phone use, especially living in the city. I went to urgent care and there was a long wait because I was the only one who hadn't pre-checked in on my phone via an app. I was able to use the bus tracker through text, but it came through formatted in such a way that I had to scroll like crazy to see the actual times. Finally, I realized I've become very dependent on our grocery shopping and meal planning app, and I missed having that a lot.
All these are very minor inconveniences, but in the end I decided that I would keep my smart phone and just use badassity/self control more. My phone goes on Do Not Disturb as soon as I get home and sits charging on the counter. That way it doesn't interfere with family time but calls and texts from my favorites (my mom, my spouse) come through. My other major smartphone-related weakness was Uber, and that one was solved by just not ordering Ubers. I think it's tempting to try and solve our weaknesses through technology, but for me, "proving" I had more self-control than I gave myself credit for has actually been nice.