Basically, urban life is becoming a luxury good in much of the U.S., in part because there isn’t enough of it to go around.
Pointer from Mark Thoma.
I think of this in terms of the four forces. The New Commanding Heights of education and health care tend to concentrate in inner cities. This helps draw affluent professionals to cities, leading to gentrification. Then you get the sorts of amenities that affluent professionals like–bike lanes, sushi restaurants, yoga studios. The urban-suburban demographics becomes sorted by taste as well as by occupation.
And income. And it becomes attractive to those with less income to associate with those with more, for more numerous, steady, and richer opportunities, and for those unable to benefit from them to leave.
Meh — in most metro areas in the U.S. the city at the urban core has a relatively low share of the housing market for affluent professionals. Consider Chicago. Yes, there are plenty of affluent professionals in the city. But there are a hell of a lot more in the many, many wealthy suburbs and exurbs. And affluent areas in Chicago are by no means space-limited. The city is almost a million below its peak population; there are many, many more neighborhoods that could be gentrified if the demand was really there — but demonstrably it isn’t. And it’s no secret that one of the main limiting factors there, as elsewhere, is that core cities remain lousy places to raise your kids. San Francisco may be the poster city for ‘urban living as a luxury good’, but the schools are mediocre and SF has become famous for its lack of children. Urban living is mostly just for childless affluent professionals. Once they procreate, they relocate.
I think you aren’t making enough of the effects of job clustering. Looking at where I live in California, you have very strong clusters in the two leading metros: tech in the Bay Area and media in LA.
By definition, you can’t have clusters of industries outside metro areas.
Admittedly, California may be a special case, with both extremely strong economic clustering and extremely strong limits on development. Possibly, given the limits on development and the persistently higher prices associated with them, no one would have bothered to move to California since 1990 if the clustering had not been so important for people in those industries.