pwang
There are a ton of subtleties in the build toolchain even within a single OS type, and these will lead to downstream frustrations with packages that just bundle pre-compiled versions of C/C++ libraries.
The conda approach treats the underlying libraries as first-class citizens in the package ecosystem; tracks their interdependencies; and most packages are built in such a way that they are relocatable on your filesystem and don't require system privileges to install. conda and the various packages in the conda universe (whether official ones from Anaconda or the community-built ones in conda-forge) all make it so that this baseline hard problem is mostly solved across all major OSes, and solved in a relatively consistent way.
You can think of conda as a cross-platform, cross-architecture, multi-language, userspace package manager. It grew into this because numerical Python's package ecosystem is so horribly complex and the users span such a huge set of install environments, that conda ended up having to be a generic rpm/brew/apt kind of thing.
TLDR - we are working to clean up this language to leave it clear that educational institutions are exempt, and that these commercial terms do not apply to third-party channels hosted at anaconda.org (which includes conda-forge).
The commercial licenses for the products and commercial repository is what supports all of this OSS development work.
The whole thing runs via PyScript/WASM, and lives locally inside the Excel spreadsheet. https://www.anaconda.com/blog/introducing-anaconda-code-add-...
That's numpy, pandas, sklearn, and friends.
I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.
Anaconda is a distro, and conda is a package manager, that works across OS platforms and hardware architectures, and installs cleanly into userland without requiring admin privileges. The only way we achieve this difficult goal is by creating a distro and build system that creates "portable" packages that can be relocated/relinked at install-time.
Ultimately, Python's challenges in this department come from the fact that it has such great integration with low-level C/C++ libraries. This gives it super powers as duct tape/glue language, but it also drags it down into the packaging tech debt of C/C++. Hmm... maybe I should write that blog post: "Python Packaging Isn't The Problem; C/C++ Is." :-)
FWIW, we are soon going to be releasing a much faster depedency resolver. We are also thinking hard about how best to address the "growing ecosystem" problem, in a future-proof way.
Part of the proceeds from the cost go to fund the Anaconda Dividend Program, which gives money to various open source projects in the pydata/scipy ecosystem.
There are a few much more substantive sites with analysis into the genetics and circumstances around the virus, which emerged since the April 2020 which your Nature article cites as its primary source.
Here's a direct debunk of that article: https://harvardtothebighouse.com/2020/03/19/china-owns-natur...
That author has written a more extensive article with much more information around the lab itself: https://harvardtothebighouse.com/2020/01/31/logistical-and-t...
And here is an analysis of RaTG13, the closest relative of SARS-CoV-2, as a "smoking gun":
It's tempting to say that Lem was way ahead of his time, but then we look at his contemporary philosophers of politics, technology, society like Jacques Ellul, Marshall McLuhan, Gilbert Simondon, and realize that the mid-20th century was awash in brilliant foresight about the unpleasant implications of a technological society.
IMO this nuanced thought was simply lost in the craziness of the late-60s and the sex/drug/rock&roll hedonism of the 70s, which then matured into stockbroker 80s, before a second wave of tech-capital-blindness emerged in the 1990s.
And now as these waves have transformed the entirety of modernity, we are faced with the unpleasant question of "where does it go from here, now that the Boomers -- whose narcissism birthed Consumerism -- are dying off?"
The sequel to ZAMM is the far more illuminating book, IMO. His metaphysics of quality continue to be a useful metaphysical frame.
The problem is that new content-centric business models have not yet emerged. So, some people remain chained to the old paradigms.
The crisis we have is a gap in vision. People don't realize that driving cost of information distribution to zero means that it no longer has enough scarcity to force the economic transfer of other scarce resources. Basically, what OSS did for commercial software, the Internet did for anything that fits within a 2D screen. Netflix, NYTimes, Fortnite, JK Rowling, and Jenna Jameson are all competitors in a space whose Pixels*Seconds value is commoditized.
Capitalism doesn't thrive unless there is an exponentiating dynamic. The only one available on the Internet is bandwidth capture. Which, for now, translates into attentional capture.
With the imminent arrival of P2P web software (e.g. Beaker Browser) and mesh networking, the tides will turn back towards a creator-centric peer network.
Actually, no. The paper may not have been explicitly clear about this, but the ENTIRE point of a "data visualization" system is to transform potentially high-dimensional datasets, with a large number of columns, into meaningful images by a series of steps. You seem to be interpreting this narrowly, and imagining that geometry is already pre-defined in the dataset, so then of course this looks like a fairly trivial 2D accumulator.
That is not the intent, nor is the common use case.
For data visualization, the question of "how do I accurately aggregate or accumulate the 25 - 1million points in this bucket" is a deep one. There is NO data visualization system that programmatically gives access to this step of the viz pipeline to a data scientist or statistician. Most "infoviz" tools gloss over this problem - they do simple Z buffering, or cheesy automatic histograms of color/intensity, etc. These are almost always "wrong" and produce unintended hallucinators.
Your first comment - about "not needing all the geometry present" - indicates that you are not understanding the nature of the problem datashader was designed to solve. There is no simple "cull" function for data science; there is no simple "Z" axis on which to sort, smush, blend, etc. At best, your data points can be projected into some kind of Euclidean space on which you can implement a fast spatial subdivision or parallel aggregation algorithm. But once that's done, you're still left holding millions of partitions of billions of points or primitives, each with dozens of attributes.... what then?
It's not about what APIs are being used to render whatever. At that level of analysis, all that anybody is every doing, is just doing memcpy and bitblt. Rather, datashader provides a framework for applying semantically meaningful, mathematical transformations on datasets as they're being accumulated, as those accumulations are converted into aesthetic/geom primitives, and as those primitives are rendered into colors. It really is "renderman for data", along with arbitrary vertex/texture shaders, driven by a dynamic rasterizer that can use whatever bins in data-space (not merely physical pixels).
BTW "Out of core" does NOT come from raytracing; in fact its history in computing is a term for anything that exceeds physical memory. We use it all the time in scientific/HPC and data science because datasets are frequently much larger than available memory.
The pipeline is also built in such a way that it permits front-end JS viewers like Bokeh to drive a very dynamic experience.
I actually have a background in 3D computer graphics, and it's precisely because of my detailed knowledge of raytracing, rasterization, OpenGL, BMRT, photon maps, computational radiometry, BDRFs, computational geometry, and statistical sampling, etc... that when I came to the field of data science & specifically the problem of visualizing large datasets, I realized the total lack of tooling in this space.
The field of information visualization lags behind general "computer-generated imagery" by decades. When I first presented my ideas around Abstract Rendering (which became Datashader) to my DARPA collaborators, even to famous visualization people like Bill Cleveland or Jeff Heer, it was clear that I was thinking about the problem in an entirely different way. I recall our DARPA PM asking Hanspeter Pfister how he would visualize a million points, and he said, "I wouldn't. I'd subsample, or aggregate the data."
Datashader eats a million points for breakfast.
Since you're clearly a computer graphics guy, the way to think about this problem is not one of naive rendering, but rather one of dynamically generating correct primitives & aesthetics at every image scale, so that the viewer has the most accurate understanding of what's actually in the dataset. So it's not just a particle cloud, nor is it nurbs with a normal & texture map; rather, it's a bunch of abstract values from which a data scientist may want to synthesize any combination of geometry and textures.
I chose the name "datashader" for a very specific and intentional reason: we are dynamically invoking a shader - usually a bunch of Python code for mathematical transformation - at every point, within a sampling volume (typically a square, but it doesn't have to be). One can imagine drawing a map of the rivers of the US, with the shading based on some function of all industrial plants in its watershed. Both the domain of integration and the function to evaluate are dynamic for each point in the view frustum.
We renamed from Abstract Rendering to Datashader for affordances of human cognition.
This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization, as part of an effort to come up with an algebraic process for visual design: https://vis.cs.ucdavis.edu/vis2014papers/TVCG/papers/2181_20...
Using their metrics around "confusers" and "hallucinators", Datashader came out as one of the few things that doesn't suffer from such intrinsic limitations.
By doing that, you've put your code into the realm of "commercial software", which engages with companies/corporate use in a whole different modality. If you've never sold software to a business, you will have no idea how much is involved in this side of things.
In addition to being treated differently by your potential customers, you will also earn the hatred and ire of your open-source-loving colleagues. Some bored college kid will see your dual-licensed software as an immoral act, and spend time building a less-awesome and more incomplete "completely free" version, which will then attract dev mindshare and users and eventually ossify into a de facto standard which you will then have to support.
That college kid will one day graduate, and think he can build a business on top of this amazing software he's made that everyone loves and wants to use, and then sit around trying to figure out how to make money from it. He may even consider making a "free-mium" model or an "enterprise" offering on top of the "open core" of his widely adopted OSS.
These will most likely fail, because - again - SELLING THINGS IS HARD.
Then you and this college kid who disrupted your dual-licensed OSS will one day meet at a symposium for "open source sustainability". It will be awkward. Teeth will be gnashed. You will get lectured about not having just used Patreon. Meanwhile the companies that use your and the college kid's OSS continue to hit their quarterly numbers for Wall Street. Executives earn out bonuses. The circle of life continues.
¯\_(ツ)_/¯
They're trying to create a new marketplace where commercial users of OSS can actually give money back to FOSS devs.
Many FOSS devs live a life of poverty because they have no idea how to actually sell things of value around their free artifacts and free labor. It turns out that selling stuff is hard, in general, and feels almost nothing like coding. ¯\_(ツ)_/¯
The founder/CEO of Tidelift is actually participating in a keynote panel this morning at the NumFOCUS summit.
I get it, it's fun to be cynical about things, but these are folks that are legitimately trying to help.
Hasn't this been a pretty well-known perspective (no pun intended) for a couple of decades now?
Then I realized it was written by Bill effing Gates. And that made me feel sad.
Firstly, because Bill is smarter than this. His lack of awareness of just how pressing this phenomenon is, indicates that he's either had his head in the sand, or the intellectual circles he swims in hasn't been ringing the alarm bells about this. (See my essay on the Changing Nature of Scarcity: https://medium.com/@pwang/the-changing-nature-of-scarcity-fc...)
Secondly, because of the irony. Bill Gates made his fortune having created a business around selling software. It would be good for him to review John Perry Barlow's "Selling Wine Without Bottles": https://www.eff.org/pages/selling-wine-without-bottles-econo...
As the world's first and most successful wine bottler, billg might find this perspective illuminating.