California is losing residents at both the top and the middle of the income ladder, a bad sign for the state's tax base and its political leadership.

• Wealthy residents are moving out mainly to escape the state's high taxes.
• Families are leaving over crime and safety worries.
• Others say they are simply fed up with the state's progressive politics and culture.
• Losing high earners hits California hardest, since a small slice of top taxpayers funds a large share of the state budget.

Outlook: Expect more pressure on California's budget and continued moves to lower-tax states like Texas and Florida.