Well, there's a long legacy of half-baked solutions and kludges. And, especially, install instructions consistently suggest you do the worst thing: install stuff using pip.
Everyone knows that's bad advice, because it always fails in mysterious ways, and there's no good way to roll everything back.
I recommend:
python3 -m pip install pipx # Don't rely on the pip3 bin.
pipx install jupyterlab
# Now jupyter has an independent virtualenv.
# To install dependencies:
pipx inject jupyterlab numpy
# Where is everything?
pipx list
venvs are in /Users/ben/.local/pipx/venvs
apps are exposed on your $PATH at /Users/ben/.local/bin
...
Once you've done that, never install anything directly with pip3 again. If you're working on a project, use pipx to install poetry / dephell / pipenv and use them to manage the virtualenv.And you can get rid of it all:
pipx uninstall-all
python3 -m pip uninstall pipxAs an aside; it is really hard to explain why you should use anything but pip3 to people who only rarely need to interact with python.
And I agree, it's hard to explain not using pip3, so I just wouldn't. "For the best experience, pipx install neato-commando." And maybe link to the pipx website, in case someone actually clicks a link.
The video discusses python packaging in general. You have to skip through several minutes of pre-roll though. https://www.youtube.com/watch?v=tNlurLxcf68&feature=emb_logo
My (extended) team couldn't even get all the Pythonistas to agree on The One True Python Stack™.
Looking at the amount of polish on the Python docs and the fact that people still don't know how to install Python, it gives you a sense of how hard explaining things is. Maybe the Python site could do with an official "managing Python for end-users."
> Noob me found all sorts of howtos and getting started guides. Seemingly contradictory, mutual exclusive.
Python's a victim of its own success as the signal to noise ratio in online tutorials is very low; worse, the good management tools are relatively new.
Even for smaller audiences, obsolescence is a big problem. The most painful aspect of working with LaTeX was I'd struggle with one buggy package recommended by a tutorial for ages, then search for "why am I getting this error" and find the answer was, "oh, yeah, that's horribly broken, use this one instead."
[1]: https://www.python.org/doc/sunset-python-2/ [2]: https://www.python.org/doc/versions/
I ended up putting off learning Python for years as this kind of stuff kept happening (first the 2/3 transition, and then massive pip breakage).
OSX and its tooling are just ridiculous. I have no idea to this day how macs became the premier development environment.
As always xkcd has a comic for it: https://xkcd.com/1987/
Because:
(a) it's quite easy to set things up with brew, macports, and/or Nix
(b) because Python is shitty everywhere anyway, and Python isn't the be-all end-all of development work.
(c) because you get a full-featured, working, coherent, take-it-or-leave-it desktop that stops one way of endless tinkering and procrastinating available in Linux to get things "just right"
(d) because it's still a UNIX with a full support for unicy tools, not a hack like WSL or WSL2.
(e) because it has good hardware (mostly - BS keyboard-era aside) and good resale value
(f) because you get to enjoy most/all the proprietary tools you like too (from MS Office and Adobe Creative Suite, to whatever)
(g) because in 2020 Docker, remote environments, etc, make many "local dev environment" points moot anyway
The entire point of this thread is that (a) is false -- see grandparent and the xkcd joke. It's not easy. It pretends to be easy, but is usually broken in some crazy way instead. Apt is also easy, but it actually works more often than not.
(c) was relevant in 2006, when the novelty of OS X was that it was a UNIX that you could actually use as a daily driver. This is what initially got developers to move to Mac. But it's been fifteen years, and all jokes aside, the "year of the Linux desktop" for developers was probably around 2012. Linux may still have issues, but they're not worse than the hoops you have to jump through to make today's macOS behave.
Trendy tech companies are still buying macbook pros for their employees because that's what's been trendy for the last decade, not because they actually ask new hires what they prefer. Practical tech companies do, and at places like that you usually see a mix of macs and thinkpads.
I've used Linux for close to 20+ years, and Unices more, and never had to jump through any major hoops to make macOS behave.
What would those be (talking about something major, not "I can't get my favorite window manager to replace the macOS window management" -- the non-tinkering-friendliness is part of the allure to me and from what I read others too)?
On the other hand, Linux on the desktop never fails to dissapoint me in one way or another because of the need of tinkering, half-sketched apps for many things I want to do (especially anything multimedia and/or document related), driver issues to get things working (sound, compositor, 3D, bluetooth, sleep, etc), and so on. And judging from the everpresent "just use <name of another distro>" in the relevent forums, it's not something others don't have.
Thus I prefer to stick to Linux on the server and Docker, or for setups where I have investigated the hardware in advance, and only mean to use basic things (e.g. happy with just some terminals, emacs/vim, i3, and some mp3 playing).
>not because they actually ask new hires what they prefer.
Those that do found that hires generally prefer Macs. That's how they have ~ 50% of the dev surveys on Stack Overflow whereas they're just 10% of the general market...
My anecdata for this is that I bought a MacBook in 2020 for precisely this reason. I am not a programmer, mostly my use case is bioinformatics and processing large datasets. Native terminal is better than WSL (although WSL is good now) and macOS is much more reliable than linux ever has been for me.
(b) makes the "macOS is particularly bad for development because I found Jupyter/Python deps difficult there" argument moot, as messed up Python dependencies are the case in Windows and Linux as well.
And (g) says that Docker and co has superceded manually setting up Python environments for many (not necessarily all or even most) devs, mitigating concerns about managing multiple local versions different deps/libs/language versions to work with different projects (since you can now do that in different, isolated, virtual environments which are mini-OSes in themselves).
So (b) basically amounts to: "It's not macOS which is makes Python deps shitty, they are inherently shitty".
And (g) basically amounts to: "Since Docker and co make local development dep issues mostly obsolete, even if macOS was bad at local deps, it wouldn't matter as much today anyway as virtualization levels the field".
And of course, with the field levelled by (g), if you go for virtualized dev envrironments, you still get all the other benefits like e, f, c, and d.
I've been editing a tutorial one of my coworkers wrote that targets new Python users on Windows. From my findings, the grass is not greener.
Granted, geospatial Python is somewhat of a mess, but a lot of tools I have to use are somewhat messy forks of Unix tools (looking at you, pyenv-win) with tons of incompatible extensions. For development, Windows is the exception because you can transfer just about anything from Linux to macOS.
On my Mac, I can easily install all of the Python packages I need without needing to install Visual Studio, pipwin, anaconda, etc. I have bash/zsh as my default system shell. Maybe it's easier to native Windows users, but bash/brew is a much better combination than anything I've found in Windows.
WSL is a step in the right direction, but it still feels secondary. If a first-class terminal experience existed on Windows, I have a strong feeling that it could be the premier development environment, or at least closer to Mac.
Maybe I'm using it wrong, but it definitely hasn't been made clear on how to use it right.
Maybe check the other other side (Linux) - I found the grass is greener there - at least for Python (and programming tools in general). I'm very comfortable on the command-line, and moving from a pure Linux environment to OS X & brew felt like a huge downgrade, followed by random annoyances that remind you you are using inferior, non-GNU utilities:
$ ls my_dir -l
ls: -l: No such directory
Really - Mac OS? I know its minor, but that's just user-hostile and it happens every few weeks; I can't get over it.Well, you can switch to another ls in 10 seconds by "brew install gnutools" or some such.
Not to mention the same arguments could be made for FreeBSD, commercial unices, etc.
Come to think of it, I've been using Unix (including Linux) for 25 years, and never even occured to me to expect "ls my_dir -l" to work.
Yes. In most shells it's trivial to go back by a word (e.g. skip in front of the path with one shortcut).
That said, this happens so rare that it's not even something that ever registered with me as a problem. I know what arguments I want for ls. And if I don't it's usually some bizarro infrequent flag that I have to look up how it's used anyway.
And other programs where it would be actually useful don't allow it anyway (having flags after the "main" argument). Git, for one.
I did - but I never remember to type "gls" instead of "ls" - if I were that mindful, I'd always put in the flags at the start, where the BSD utils expect them. As sibling comment noted, trailing flags usually happens when I run a command and decide to amend the flags by up-arrowing and typing. A typical result is "ls -al my_dir -tr" when I realize I want to sort by mod date, post-facto.
I've been using GNU utils 95% of the time I've used Unix(-like) systems: I expect "ls -al my_dir -tr" to work: it's 2021 - recognizing flags isn't rocket science.
I've been using WinPython for a few years. It's the closest thing to "just works" that I've found, and non-programming colleagues have been able to install it successfully.
Because it works almost like an isolated "container," it's also possible to test your code on a clean install of WinPython to make sure it will run on someone else's computer before you share it.
I don't know the technical difference between WinPython and a true container, but you can have multiple WinPython installs on one machine, and they don't interfere with one another, or with a pre-existing mainstream Python installation on the same PC. So you can share your stuff without worrying about screwing up someone else's stuff.
s/macs/python
s/development environment/scripting language docker run -p 8888:8888 -v ~/Code/actual-project/:/home/jovyan jupyter/datascience-notebook
Note ~/Code/actual-project/ is a git repo or a mounted Google Drive folder. Moving from one machine to an other is pain-free.[1] https://jupyter-docker-stacks.readthedocs.io/en/latest/using...
Instead, use: docker run -p 127.0.0.1:8888:8888
/usr/bin/python3 -m venv ./venv # create a virtual environment in ./venv
./venv/bin/pip install wheel # optional
./venv/bin/pip install dep1 dep2 dep3 ...
./venv/bin/python ... # up and running
is enough to get started in most cases. Of course, all bets are off for GPU or Apple Silicon support...If you are in luck, your required libraries and appropriate versions are available as GNU Guix packages and you can use that.
As for the reason why this still hasn't happened? I'm guessing because it is because the people who care about reproducibility have found other ways to achieve it, and as a result have no need nor incentive to implement it as part of the notebook format. Also because, if someone is running notebooks at all they know enough to bootstrap the jupyter server, which is non-trivial, or they are using a managed environment (be it remote, conda, etc.) where they are blissfully unaware that all their code has massive implicit and undocumented context.
In the mean time I have implemented the start of an equivalent system for org files because the implicit environment is much worse given normal Emacs workflows. I have encountered some significant issues and considerations which the notebook ecosystem would have to deal with, and which the default python packaging ecosystem (or any ghettoized language specific packaging solution for that matter) is not equipped to handle.
The primary issues is that only language agnostic package managers can actually bootstrap the environment that is required. What do you do if someone doesn't have a toolchain to build numpy and they are on some strange arch where there are no wheels? Do you just fail? The 80% solution here is the one we have with conda or managed solutions. The 90% solution requires that all the language communities stop pretending that they are the only language in the world and stop wasting time on their bespoke package manager.
Well not really.
First, there is no link between a distribution package name (what you install with pip) and python package name (what you import).
e.g. The distribution package "foobar" can install the python package "baz". Well, it's even worst than that. There could be name space packages so multiple distribution package actually install parts of a python package. And there could be monkey patching.
Second, Python is not a static language. The import system is part of the interpretation of python code, and as such, is dynamic.
You have absolutely 0 guarantee that "import foo" on two different environments, even if they have the same version of package foo, will produce the same results. There could be an infinite amount of magic done in foo's __init__ that depend on other libraries being installed, etc.
Third, don't get me started on the actual version of these libraries.
Then comes the point, where one of your dependencies in conda simply wont install on the machine of your coworker or a windows machine of another coworker, or in your docker image and you are screwed, scrambling for the next package manager. Been there, experienced that.
I can only recommend to people not to go that route and to not get into trying to get the same environment on windows. A lot of headaches. It will be simpler to tell people to use another OS, if they want to use the same environment. Ultimately I ended up providing a virtual machine.
A managed solution -- Yes, this takes a lot of load off of the shoulders of people, who would like to actually get productive in other ways.
They've pretty much always been a moving target, haven't they? Nothing has ever been good enough that the community agrees "okay, this is it, we can stop churning on this tooling for a while".
Probably testing package managers is not a simple thing to do.
A: it works on my computer
B: then we'll ship your computer
and that's how docker was bornI used it for a semester in college, and it was next to unbearable. I can appreciate it as a fun little functional programming gimmick, but nothing more than that.
I could have done it in Python but portability was always the issue.
At my former job I had to make charts from pictures with English and Spanish lettering. Took me 30 minutes to have a working client that got installed in a dozen computers that afternoon.
I needed to also re-structure the companies file system for every computer in our company.
Personally I worked in audio video contract work from time to time and I was able to make a dozen short cut programs that actually saved my biscuit several times. I could have done it in python but the portability kicked my butt. Racket made executables super easy.
Also I went through How to Design Programs and I learned a ton.
Python is a little worse in that in pushes its dirtiness downstream to your users too.
Then remember to run pip using `python3 -m pip <command>` and use virtual environments (venv)
I stumbled on this a couple weeks ago and documented the process here https://flaviocopes.com/python-installation-macos/
I found the same problem with Ruby (old version preinstalled on macOS)
```
$ conda create -n {new_environment_name} python
$ conda activate {new environment name}
$ conda install jupyterlab
```
({new_environment_name}) ...$ python -m ipykernel install --user --name {new_environment_name} --display-name "Python ({new environment display name})"
The env won't be accessible in jupyter (lab or notebook) until you do this step.
[0] https://stackoverflow.com/questions/39604271/conda-environme...
https://www.anaconda.com/blog/anaconda-commercial-edition-fa...
conda install -c conda-forge jupyterlab=3[0] https://repl.it/ [1] https://repl.it/talk/templates/Jupyter-Notebook/49993
docker run -p 8888:8888 jupyter/scipy-notebook
Hoping to help clear some backlog, I tried to help with some ML & data pipeline house cleaning. Had never used Python before, but how hard could it be? OMG. One really doesn't appreciate ease and simplicity of multiple simultaneous JDK installs, all peacefully coexisting, until working on some python and nodejs projects.
Just running user space dev env via Docker was recently front paged here on HN. I believe that will become the norm.
Being a macOS user, I looked to see if there was something more lightweight. Like maybe a port of podman. Lighter than running a virtual Linux instance.
Again as a macOS user, I feel like I'm overdue to embrace the remote dev desktop set up. I've read that Googlers don't do any work locally; they all remote shell into servers running the tool stack. So very X Windows; what is old is new again.
It really throws me off the whole thing (Julia + Jupyter is even worse...) because I’m not keen on putting effort into stuff that nobody, myself included, will be able to use in a year or two.
While I know it has earned its praise for good reason -- as I'm sure all the data scientists I work with would attest to -- as an engineer, Python has been really frustrating to work with.
Dependency management hell isn't just a Javascript problem these days.
Actually it is:
That said, you would be up and running if you used conda.
And this is not the usual "just try distro X" Linux advocate BS when somebody has an issue with distro Y.
Conda does makes thing more user friendly by focusing on whole environments. Actually that's why it was built for.
Note that this applies for this situation. For other uses cases, e.g. deploying something to production, having used conda would have made your life more difficult.
Over an hour later I had to give up. There was initially some kind of Python version conflict on my Mac. Eventually some version of JupyterLab was installed somewhere, but it couldn't find any dependencies for the notebook. As a complete newbie to the Python ecosystem, I googled for instructions and found various environment managers, which then failed possibly because something is incompatible with Big Sur. More googling revealed instructions with complex CFLAGS environment setups to fix it, which didn't — and at that point it was very far from the supposed convenience of scripting languages anyway.
I don't think it's Python's fault. Probably all the programming toolchains are this hard to a newbie! But it was a humbling experience after 34 years of programming, not being able to load a piece of sample code in a Sunday afternoon.