awgl
Most of the time this not a blocking problem since each step in a flow is mapped to a Docker image and/or your choice of EC2 instance (e.g. one step on a GPU, another on a memory optimized instance). You can have one step use an image with all of your python-based ML stuff, and another step have a different image with compiled exectuables that are triggered by a system call. If needed, outputs from such a system call would then need to be persisted in a database/S3 or read back into the python flow for persistence. So, it is not as seamless as a flow in all python, but it can work "good enough".
Straightforward for data/ML scientists to pick up, familiar python class API for defining DAGs, and simplifies scaling out parallel jobs on AWS Batch (or k8s). The UI is pretty nice. Been happy to see the active development on it too.
Currently using it at our small biotech startup to run thousands of protein engineering computations (including models like RFDiffusion, ProteinMPNN, boltz, AlphaFold, ESM, etc.).
Data engineering focused DAG tools like Airflow are awkward for doing these kinds of ML computations, where we don't need the complexity of schedules, etc. Metaflow, imho, is also a step up from orchestration tools that were born out of bioinformatics groups, like Snakemake or Nextflow.
Just a satisfied customer of Metaflow here. thx
- Weather: Sure, SF Bay Area wins here in terms of what you might think of as regularly experienced weather, but it is not without its extremes. Let's recall that most of California, including surrounding SF Bay Area, has been in an extreme drought for the past several years [1], threatening local agriculture and water supply. Also, SF is famous for extremely devastating earthquakes that happen every so often [2] (though, you may not categorize seismic activity as weather, but still a natural occurrence sort of thing).
- Government: SF has its fair share of recent government corruption [3, 4] as well as historical seediness/lawlessness [5], not unlike some of Chicago's history.
- Traffic: Personally, I think the Bay Area has worse traffic than Chicago. The data in this report [6] seems to agree, ranking SF/Oakland as #3 worst in the country and San Jose as #5, whereas Chicago is #8 on the list. And, really, waiting on a few trains to pass for a sports game or special event is not unique to Chicago. Try taking the BART across the bay for a Giants game, Super Bowl 50, Bay to Breakers, Pride Parade, etc. I mean, every city's transportation system hits its limit with big events/gatherings. So what? It's part of living in a city that you learn to accept.
- Wages: Can't debate it. Chicago software engineers, making average of $76k/yr [9], do get paid less than SF, at $110k/yr [10]. But, also consider that income tax in Illinois is a flat 3.75% [11], whereas California's income tax for that bracket is 9.3% [12]. 5.5% more tax in CA is not small. (Federal income tax will also be lower in Chicago with the lower salary, but I haven't looked up those numbers.) And, let's consider cost of living more in general. This handy website [13] says that "A salary of $76,000 in Chicago, Illinois should increase to $178,313 in San Francisco, California", which is mainly due to housing being 428% more expensive in SF than in Chicago. That says a lot! You might interpret that as the $110k/yr average salary in SF actually gets you less quality of living than the $76k/yr in Chicago (according to this website). Or something like that. Don't know, but my housing quality in SF is twice as much for half the space as compared to what I had in Chicago. (What am I doing here?!)
- Crime: According to this (presumably reputable) data [7, 8] Chicago has been steadily improving in terms of crime over the past more than a decade. SF has been kind of steady, and, according to that data, is actually worse per capita than Chicago. I didn't expect that at first. Chicago is a much bigger city and than SF, so that kind of helps to dilute crime among the population, I suppose. Anyway, my anecdotal evidence from friends tell me that it is improving. We must have different kinds of Chicago friends. Or maybe we should trust data instead?
- Schools: SF public schools have a lottery system too [14]. I will grant, though, that CPS does have a lot of ongoing battles with the city and state for funding, like recent closings and strikes. That stuff sucks because it's usually the poorest schools that suffer the most. Ultimately, this is a bigger societal problem than us engineers debating about where the best place for us to live is on a web forum. (Probability of existential crisis increasing...)
Okay. Got that off my chest. I love Chicago. I hope you and others do too still. I may move back soon. Writing this stuff down has, well, made me think twice about wtf I'm doing out here on the West Coast...
1. http://droughtmonitor.unl.edu/Home/StateDroughtMonitor.aspx?... 2. https://en.wikipedia.org/wiki/1989_Loma_Prieta_earthquake 3. http://ww2.kqed.org/news/2016/01/22/3-former-fundraisers-for... 4. http://www.sfgate.com/crime/article/I-am-guilty-final-S-F-co... 5. https://en.wikipedia.org/wiki/Barbary_Coast,_San_Francisco 6. http://247wallst.com/special-report/2015/08/28/10-cities-wit... 7. http://www.city-data.com/crime/crime-Chicago-Illinois.html 8. http://www.city-data.com/crime/crime-San-Francisco-Californi... 9. https://www.glassdoor.com/Salaries/chicago-software-engineer... 10. https://www.glassdoor.com/Salaries/san-francisco-software-en... 11. http://www.bankrate.com/finance/taxes/state-taxes-illinois.a... 12. http://www.bankrate.com/finance/taxes/state-taxes-california... 13. http://www.bestplaces.net/cost-of-living/chicago-il/san-fran... 14. http://www.sfusd.edu/en/enroll-in-sfusd-schools/how-to-apply...
As niels_olson points out, the primary interaction here is between the electron beam and the atomic nuclei.
So, yeah, my comment is not entirely accurate about the electron densities of the atoms. If you feel it is too misguided, I'll remove it.
This is what happens when you ask a theoretical chemist to explain an experiment. ;)
While the electrons and nucleus (i.e. protons + neutrons) of atoms are indeed 'particles', they are exhibit wave-particle duality: http://en.wikipedia.org/wiki/Wave%E2%80%93particle_duality
In brief, quantum particles act like waves sometimes (think ripples of water) and act like particles (think tiny billiard balls) at other times. The consequence of that, and a major tenet of quantum mechanics, is the 'wavefunction' of a quantum particle. A wavefunction of a particle, instead of just being a single point in space, amounts to a probability density of position and momentum.
So now that we know the above, in these images, what they are actually measuring is the spatial probability density of the electrons. The lighter values correspond to high density of electrons. The darker values correspond to less electron density. The high density occurs around the nuclei of the atoms. Thus, atomic resolution. However, note that individual electrons are not resolved in these images.
Finally, I want to recommend against thinking of atoms as electron planets orbiting a nucleus sun full of empty space in between. That thinking ignores quantum mechanics. The truth is much more fascinating, which is that electrons are wave-particles that have probabilistic densities.
P.S. Protons and neutrons are themselves made of up more elementary quantum particles: quarks!