If we planned and did it wisely, no, maybe not.
The way we're doing it, yes.
And by "we", we're talking about the affluent (globally). For instance the average American uses roughly 30-100 times the energy of the average low income resident (from Eritrea, Afghanistan, Morocco, El Salvador, etc...). In the aggregate, in terms of energy, ~10-35 billion low income use as much as ~.3 billion Americans. If we break it up strictly on income, it's probably worse. Many resources follow this pattern.
The worse part is that the average American could fairly easily cut their fossil fuel energy consumption by ~75%, and with a few more years of DIY work, cut it by ~80-90%.
Any evidence this could be done by Americans without skyrocketing mortality? You need to count energy usage per capita, so that means all the physical objects that took a bunch of energy to make also. Like if a computer is purchased and used in America that should be counted to Americas energy use and not China's, because those people don't get the benefit of consuming the energy required to make it.
People use less energy in those countries you mentioned because they are poor and suffer great poverty, not because they are environmentally concerned. They die from preventable diseases because they don't have access to energy because they can't afford it.
But I do encourage you to move and reduce your energy consumption to fulfill your fantasy, even swap places with an Afgani or Moroccan.
Fulfill my "fantasy"? Are you trying to be offensive? Not that it matters I guess....
Going back on topic, you're conflating two separate statements. One is that most of the impact on the environment is done by a relatively small and wealthy part of the world population. The second is that by virtue of that wealth, this group could have minimal impact without the downsides normally associated with low consumption/impact, but they choose not to.
The idea that someone needs to live like someone in a third world country is a false dilemma. That comparison was just used to illustrate where most of the impact comes from, wealthy individuals in the first world, and to a lesser extent in other places.
If you're interested in specifics, take a look at this.
http://www.withouthotair.com/It's a UK centric overview, but plug and chug for where ever you live... Look at the stuff chapter for more info on items. A 3 bedroom house for example requires ~42000kWh, which is a lot, but it's amortized over many decades (a 50 yo house would require ~1000kWh/year). For comparison, this is only 25000 miles of driving in a 20mpg vehicle, which plenty of Americans roll through in a year or two.
In the aggregate, the energy associated with making stuff for someone requires ~12000 kWh per capita per year. Someone on the frugal side would probably need half of this, or less. Food requires about the same amount, and could reasonably be cut to ~4000kWh/year with a frugal/mostly vegetarian diet. Heating, cooling, electricity, and transportation can in most cases be configured to use only renewable energy.
This puts the total energy requirements at ~10000kWh/year as opposed to the US average of ~83000kWh/year, which, like I mentioned, would minimize the majority of impacts people have on the planet, and that would also be minimized as agriculture and manufacturing adopt better production standards/use more and more renewable energy.
http://en.wikipedia.org/wiki/List_of_countries_by_energy_consumption_per_capita