Note that this is heavy disputed, with estimates going from under 100 to the hundreds of thousands.
> There is consensus that a total of approximately 30 people died from immediate blast trauma and acute radiation syndrome (ARS) in the seconds to months after the disaster, respectively, with 60 in total in the decades since, inclusive of later radiation induced cancer.[2][3][4] However, there is considerable debate concerning the accurate number of projected deaths that have yet to have occurred due to the disaster's long-term health effects; long-term death estimates range from up to 4,000 (per the 2005 and 2006 conclusions of a joint consortium of the United Nations) for the most exposed people of Ukraine, Belarus, and Russia, to 16,000 cases in total for all those exposed on the entire continent of Europe, with figures as high as 60,000 when including the relatively minor effects around the globe. Such numbers are based on the heavily contested Linear no-threshold model.
I think the under-100 numbers are just from the direct effects, disregarding the excess deaths among the "liquidators" and other people exposed to radioactivity after the accident.
But you aren't going to get a Chernobyl-style disaster from a criticality accident in a glovebox; Chernobyl happened inside a containment vessel that harnessed 3 gigawatts of thermal energy in normal operation, along with massive quantities of graphite moderator which caught fire and distributed radioactive material all across Europe.
Also, Los Alamos has a population of 13'200; the nearest city is 93 km away with about 930'000. Those people are very important but evacuating Los Alamos would not be nearly as difficult as evaluating Pripyat (population 49'360) — unless strict official controls on information dissemination delayed the news until it was too late, anyway.
Basically I think people are worried about nuclear fuel processing not because of its real dangers (which are significant but of a smaller order of magnitude than the dangers of fertilizer shipping, pesticide manufacture, and inadequate building codes in earthquake zones) but because of a well-justified belief that the government lies about them, along with the well-documented fact that nuclear energy was originally developed for mass killing.