Or your assumption wrong. Here is a reasonable explanation: People are poor since they got bad self control, which also makes them fat. People who lack self control are easily tempted with shitty fast food, so their areas mostly serves it rather than real food creating these "food deserts".

If there was demand for food in those areas people would sell it, but there isn't.

Edit: A strong piece of evidence is that people aren't getting poorer, but they sure are getting fatter.