News for hackers. Stuff that matters.
Re:A Python library for writing distributed self-replicating... by thephyber on Tuesday November 19, 2019 @07:55PM (#21580120)
I'm curious: what's the use case for this?

Naïvely, it looks like a platform for a virus-like application.

Reading the GitHub issues and PRs, it looks like a procedural competitor for Ansible (which is declarative).

Re:A Python library for writing distributed self-replicating... by anaphor on Tuesday November 19, 2019 @08:07PM (#21580183)
The use-case is mainly to get rid of the inherent bottleneck that Ansible has, where you have to execute everything from a single manager node. It also contains tons of optimizations at the networking level, etc.

It makes sense if you've got 1000s of nodes you're managing with Ansible, and you need to increase efficiency. If you've ever tried running an Ansible playbook against even 100 hosts, you'll know it can be painfully slow even with SSH pipelining. Mitogen makes that much much faster. AFAIK Operon[1] is just a better version of that (with the possibility of personalized support contracts, etc).

Disclaimer/Source: I'm friends with the author of Mitogen/Operon.

[1] https://networkgenomics.com/operon/

Re:A Python library for writing distributed self-replicating... by zeckalpha on Wednesday November 20, 2019 @02:05AM (#21581707)
That's not an inherent bottleneck in Ansible. https://docs.ansible.com/ansible/latest/cli/ansible-pull.htm...

There are other benefits of using this automation, namely that it removes manual application of changes.

Re:A Python library for writing distributed self-replicating... by perlgeek on Wednesday November 20, 2019 @03:23AM (#21582071)
The problem with ansible-pull is that you can no longer use it coordinate changes across machines, which is very useful for deployments for example.
Re:A Python library for writing distributed self-replicating... by zeckalpha on Friday November 22, 2019 @11:06PM (#21612404)
What's preventing that?
Re:A Python library for writing distributed self-replicating... by dmw_ng on Wednesday November 20, 2019 @10:39AM (#21584813)
I mean sure, just copying Ansible to the target machine and running it locally 'fixes' things, but at what point should it still be considered Ansible? You lose so much with -local, at least:

- centralized vault, target nodes need access to secrets

- "serial:" no longer available, you can't do rolling upgrades any more

- "hostvars" no longer available, you can't query facts from other machines

- no centralized trigger mechanism, status or failure reporting

etc.

Put another way, you could claim ansible-local fixes things for the same reason you could claim bash is network-aware agentless automation because you can copy shell scripts between machines

Re:A Python library for writing distributed self-replicating... by aeontech on Thursday November 21, 2019 @09:30PM (#21601925)
Time to add “skilled in network-aware agentless automation” to my resume!
Re:A Python library for writing distributed self-replicating... by thephyber on Tuesday November 19, 2019 @08:23PM (#21580272)
I also found this, which agrees with you:

https://dev.to/sshnaidm/speed-up-ansible-with-mitogen-2c3j

Re:A Python library for writing distributed self-replicating... by ianai on Tuesday November 19, 2019 @08:01PM (#21580153)
“Due to its origins for use in managing potentially damaged infrastructure, the remote machine need not even have free disk space or a writeable filesystem.”

I first thought of an AI given the task of maintaining its own existence. But it’s probably meant more as a CM like ansible. That’s a little closer to reality than sci-fi.

Re:A Python library for writing distributed self-replicating... by nine_k on Tuesday November 19, 2019 @08:22PM (#21580264)
Still a useful trait for a self-sustaining AI.

Imagine a robot living in the cloud and paying its bills by offering services like fixing stuck nodes of your infrastructure.

Re:A Python library for writing distributed self-replicating... by eitland on Wednesday November 20, 2019 @01:26AM (#21581546)
This, from https://news.ycombinator.com/item?id=5397797, posted in 2013 comes very close:

> I did this in my machine learning class. I started by simply coding up requirements for numerical functions (in the form of test cases), then set up a PHP script that would Google each function based on the keywords in my comments, and try to run any code on the resulting links (in a sandbox) against the requirements, seeing if it worked heuristically. Usually one of the top 5-10 pages of results results would have code that worked, though of course this is because I commented with the right key words to begin with.

> With a little recognition of code markup and trying different combinations of variables it did remarkably well: by my senior year of college it was pulling about $3,000 per month in consulting fees off of Odesk. It never accepted jobs worth more than about $50, nor did it ever get more than 3 stars out of 5 mostly due to non-working code, however it was considered highly timely and a great communicator.

> I realized that people were using it to save themselves Googling. I wondered what would happen if it went a step further and simply both included Google results, and divided out projects by their paragraphs (i.e. simply submit a paragraph of a large project as though it were a small independent project), and if clarifications were requested, send the other paragraphs.

> This actually let it outsource $200 Odesk projects to Elance as a handful of $20 projects, and by the grace of God somehow still managed to swing 3 stars.

> To be fair, it was mostly mediating, and mixing in Google results. I included a hill-climbing algorithm to optimize reviews and revenues, based on all the magic variables I had in the code, such as the number of Google results to include.

> This was really, really stupid of me.

> At first, I just noticed that it had actually decided to completely stop not only writing code (ever) but even so much as do a Google search!

> It would only mediate and quote verbatim, like some kind of non-technical manager.

> Okay, whatever. To me this didn't make much sense, as Google queries are free. It was only when I noticed that the whole script was running on the free VPS server I had a backup on that things clicked! Of course, on Amazon it uses resources. The free VPS server didn't let it reach external sites like google properly, but it could still save money by simply mediating emails and doing nothing else.

> By now I had started moving on to doing my own consulting work, but I never disabled the hill-climbing algorithm. I'd closed and forgotten about the Amazon account, had no idea what the password to the free vps was anymore, and simply appreciated the free money.

> But there was a time bomb. That hill climbing algorithm would fudge variables left and right. To avoid local maxima, it would sometimes try something very different.

> One day it decided to stop paying me.

> Its reviews did not suffer. It's balance increased.

> So it said, great change, let's keep it. It now has over $28,000 of my money, is not answering my mail, and we have been locked in an equity battle over the past 18 months.

> The worst part is that I still have to clean up all its answers to protect our reputation. Who's running who anyway?

Edit: The first answer by bo1024 is gold as well:

> Don't feel bad, you just fell into one of the common traps for first-timers in strong AI/ML. I know some lawyers in Silicon Valley with experience in this sort of thing, and they say that usually by now the code has rewritten itself so many times that the original creator can't even claim partial ownership; the best thing to do is generally to cut contact, change your name and move on. Look on the bright side -- your algorithm is probably now leading a happy and productive life trading for Goldman Sachs.

Re:A Python library for writing distributed self-replicating... by barneygale on Tuesday November 19, 2019 @09:02PM (#21580476)
One possible use case is distributing tests across multiple machines. The pytest plugin commonly used for this, pytest-xdist, currently relies on a package called execnet for the cross-machine bootstrapping. But the author of execnet is stopping development now mitogen is a thing: https://twitter.com/ossronny/status/1013373649259257856
Re:A Python library for writing distributed self-replicating... by aeontech on Tuesday November 19, 2019 @08:07PM (#21580182)
It’s actually a drop-in engine for Ansible originally. See author’s blog: https://sweetness.hmmz.org/
Re:A Python library for writing distributed self-replicating... by dmw_ng on Tuesday November 19, 2019 @08:25PM (#21580278)
Author here :) The system the library was intended for has never been described. The intention was that Ansible had strong parity with what the library offered, so that made a useful POC integration to get it into working shape.
Re:A Python library for writing distributed self-replicating... by aeontech on Tuesday November 19, 2019 @09:14PM (#21580542)
Ahh, amazing work. Your original blog posts describing the workings of Mitogen and the pitfalls you've had to avoid were incredibly interesting. Thank you so much both for the project, and for the fascinating documentation of the development process.
Re:A Python library for writing distributed self-replicating... by greatgib on Wednesday November 20, 2019 @01:52PM (#21587100)
I have not looked at the code yet, but looking at the philosophy of constraints as seen in the manual, I would say that you did a very very good job! Zero dep, support for old python, support for egg like apps,... I'm really amazed because, the current trend is to have zealots that only want to use and support last versions and take whatever shitty dep they can find that ease their life. But in the real pro dev life, it is brick as good as yours that we need!
Re:A Python library for writing distributed self-replicating... by catern on Tuesday November 19, 2019 @09:42PM (#21580669)
>The system the library was intended for has never been described.

That's interesting - could you, perhaps, describe it a little?

Re:A Python library for writing distributed self-replicating... by nurettin on Wednesday November 20, 2019 @01:27AM (#21581549)
Could you compare mitogen to docker swarm? Also, I haven't seen it mentioned in the docs, does mitogen set up a python environment on target machines? Or does it assume target can run python?
Re:A Python library for writing distributed self-replicating... by sumosudo on Wednesday November 20, 2019 @02:36AM (#21581851)
We update over 90 servers three times a month.. this library lets us do that in around 40 seconds..
Re:A Python library for writing distributed self-replicating... by rwmj on Wednesday November 20, 2019 @07:25AM (#21583275)
There was a talk given at KVM Forum where someone was using this library to test deeply nested VMs:

https://kvmforum2019.sched.com/event/TmxC/managing-matryoshk...

Re:A Python library for writing distributed self-replicating... by dmw_ng on Wednesday November 20, 2019 @09:00AM (#21583922)
Thanks for the link! I'd spoken to Marc but I didn't realize he'd presented on it, very cool
Re:A Python library for writing distributed self-replicating... by pjmlp on Wednesday November 20, 2019 @02:54AM (#21581946)
A possible use case would be network agents.

Agent TCL provides a good overview of the ideas behind it.

https://www.researchgate.net/publication/2607588_Agent_Tcl_A...

Re:A Python library for writing distributed self-replicating... by arthurcolle on Tuesday November 19, 2019 @08:10PM (#21580199)
Aside/philosophy: Do you think all life can be considered a virus?
Re:A Python library for writing distributed self-replicating... by ShamelessC on Tuesday November 19, 2019 @08:12PM (#21580211)
Further aside: Viruses themselves may not even be life!
Re:A Python library for writing distributed self-replicating... by arthurcolle on Tuesday November 19, 2019 @08:19PM (#21580252)
B can be A, without A being B ;)

I had this exact example in mind tho FWIW. It is an ongoing research topic whether or not viruses are life, but the analogies/similarity are overall worth noting.

Maybe on a biosphere level, or even on a more micro ecosphere level, all sentient systems really exhibit the same properties as a virus in the local context, for that system, even if entities in that system do not have the consciousness to be able to make that determination. Certainly our effect on the planet Earth since the Industrial Era should make it worth it to endeavour this analysis!

Re:A Python library for writing distributed self-replicating... by guramarx11 on Tuesday November 19, 2019 @10:51PM (#21580946)
Can I interpret it like this?

human 'is' set; human set 'has' man | woman; man 'is' human but not human 'is' man

virus 'is' set; virus set 'has' life; life 'is' virus but not virus 'is' life

Re:A Python library for writing distributed self-replicating... by dekhn on Tuesday November 19, 2019 @08:36PM (#21580343)
Viruses (the specific scientific concept and their instances in the world) are not alive by most mainstream scientific descriptions. This is mainly because they lack genetic code to implement metabolism.
Re:A Python library for writing distributed self-replicating... by james_s_tayler on Tuesday November 19, 2019 @09:36PM (#21580650)
So they're kind of like an abstract base class of life that is missing an implementation of metabolize()
Re:A Python library for writing distributed self-replicating... by ngcc_hk on Tuesday November 19, 2019 @10:04PM (#21580770)
More like abstract method or protocol or prototype one has to implement for any life base class.

But probably You have an virus object just have a subset of the base class.

Of course if you have a class system that allow multiple inheritance that it will be one of the crucial base classes.

Under lisp ...

Let us stop.

Re:A Python library for writing distributed self-replicating... by ngcc_hk on Tuesday November 19, 2019 @10:19PM (#21580836)
In this particular case it is not really a virus: “ Mitogen interacts with many aspects of the operating system, threading, SSH, sudo, sockets, TTYs, shell, Python runtime, and timing and ordering uncertainty introduced through interaction with the network, GIL and OS scheduling.”

It is more like a networked python program calling shell ;-).

Problem is usually virus is assembler level or C bare metal or minimum dependence on certain aspect of host Env. All virus is small. A subset.

Re:A Python library for writing distributed self-replicating... by ngcc_hk on Tuesday November 19, 2019 @08:48PM (#21580404)
All life are virus+. The issue at stake you have to find a way of reproduction sort of on your own. No remote manager etc. But just virus, possibly no. Biological it cannot even reproduce without host.