The riots are part of the story, but absolutely not the start. Population was significant declining before the riots. A number of choices, including a long history of racial segregation in housing before the 60s combined with US housing policy (the GI Bill) to create strong incentives for people to build new houses outside of the city. That, combined with a lack of dense housing in the city, created cascading problems that are difficult to recover from.

A good source on this topic is Thomas Sugrue's "The Origins of the Urban Crisis".