News for hackers. Stuff that matters.
Re:Deep reinforcement learning is a waste of time (2019) by YeGoblynQueenne on Sunday July 11, 2021 @05:08AM (#27799440)
Why do you have to start your comment by accusing me of moving the goal posts? I don't even like ball games. Why is it so hard to have a decent conversation on the internet?

It's a serious concern and I stand by it. It's perfectly possible -it's even easy- to make algorithms, systems, programs, what have you, that can solve one problem, perform one task, and do nothing else. Hell, you can train a dog to perform one task. Sit boy! Boy sits. And that's all it knows to do. 80% of AI research output, ever since Dartmouth, has been like that. 99% of AI research output in the last ten years is like that. My entire phylum of AI research (I'm a PhD student studying Inductive Logic Programming and even knowing nothing about that, the name should give you an idea) is roundly dismissed today as a "failure" because it couldn't get from "we can solve an instance of this problem" to "we can solve any instance of this problem" quickly enough. Researchers promised it was just a matter of "generalising" like you say, tuning and tuning until everything worked in the real world as well as it worked in blocks worlds, but the sources of funding didn't listen and then we had a winter [1].

Why is what you're doing any different when you look at it from the present? We haven't seen the future yet. In the present, you have a robot that can open one door. In the future, you have a lot of doors that wait for robots to open them.

Actually, looking at your paper, you don't even have a robot that can open one door. You have a simulated robot opening a simulated door. Talk about moving the goal posts!

Have you seen the Pythons' silly walk sketch? "It's not particularly silly, is it?" "Yes, I think with government backing I can make it very silly" [2]. That's all you got there so far.

Edit: Sorry, it's probably not your paper. But it's as far from folding laundry as you can go. I don't even see the simulated robot folding simulated laundry anywhere yet and even "simulated cloth" is a joke - the simulation itself (of the behaviour of cloth) is not even close to the real world. Simulating a robot opening a simulated door might get your paper published, but simulating a robot folding simulated shirts won't even get you a simulated paper.

______________

[1] https://www.youtube.com/watch?v=yReDbeY7ZMU

[2] https://youtu.be/eCLp7zodUiI?t=83

Re:Deep reinforcement learning is a waste of time (2019) by DukeBaset on Sunday July 11, 2021 @05:51AM (#27799618)
I am just a lay man interested in the field, but like it sort of makes sense that you can have logical rules and inference, like I mean the whole point of AI as usually conceptualized was that they understood logic, like insert your fav Dr Who/Star Trek joke here, but briefly speaking what happened? I get the charm of ML/DL but it's essentially like data fitting or something. Wouldn't you expect logic in AI? What went wrong and what is the state of the art now?
Re:Deep reinforcement learning is a waste of time (2019) by YeGoblynQueenne on Sunday July 11, 2021 @07:20AM (#27799973)
Artificial Intelligence, as a field of research, was basically created by a man named John McCarthy, who came up with the name and convened the first academic workshop on the subject (in Dartmouth, in 1956). John McCarthy (who is also known as the creator of Lisp) was the student of Alonzo Church, who gave his name to the Church-Turing Thesis and is remembered for his description of the Lambda calculus, a model of computation equivalent to universal Turing machines. These are all logicians, btw, mathematicians steeped in the work of the founders of the field of mathematical logic as it was developed mainly in the 1920's by Frege, Hillbert, Gödel, Russel and Whitehead, and others (of course there was much work done before the 1920's, e.g. by Boole, Leibnitz... Aristotle... but it only really got properly systematised in the 1920's).

So it makes sense that the field of AI was closely tied with logic, for a long time: because its founder was a logician.

There were different strains of research in subjects and with goals adjacent to and overlapping with AI, as John McCarthy laid them down. For example, statistical pattern recognition or cybernetics (the latter intuition I owe to a comment by Carl Hewitt here, on HN). And of course we should not forget that the first "artificial neuron", described by Pitts and McCulloch in 1938 (long before everything else of the sort) was a propositional logic circuit, because at the time the models of human cognition were based on the propositional calculus.

The problem with McCarthy's programme, of logic-based AI, is that logic is full of incompleteness and NP-hardness results, and the more expressive power you demand from a logic language, the more of those endless pits of undecidability and despair you fall down. So work in the field advanced veerrryyy slooowwllyyy and of course much of it was the usual kind of incremental silly-walk stuff I point out in the comment above. To make matters worse, while significant academic results drip-dripped slowly and real-world applications were few and far between, you had prominent researchers overpromising and launching themselves into wild flights of fancy with abandon based on very early actual results (certainly not McCarthy! and I don't want to point fingers but cough Marvin Minsky cough).

Then this wretched dog by the name of Sir James Lighthill :spits: was commissioned by the British Science Research Council, the holders of the Spigot of the Tap of Funding, to write a report on the progress of AI reserach and the report was full of misunderstandings and misrepresentations and the Spigot was turned off and logic-based AI research died. Not just logic-based AI- Lighthill's :spits: report is the reason the UK doesn't have a robotics sector to speak of today. Then, while all this was happening in the US and Europe, the Japanese caught the bug of logic-based AI and they decided to spend a bunch of money to corner the market for computers like they had with electrical devices and automobiles, and they instituted something called the Fifth Generation Computer project (special-purpose computer architectures to run logic programming languages). That flopped and it took with it one of the few branches of classical AI that had actually delivered, automated theorem proving.

The first link I posed in the comment above is to a televised debate between Lighthill on the one side and on the other side John McCarthy, Donald Michie (dean of AI in the UK) and er, some other guy from cognitive science in the US, I always forget his name and I'm probably doing him a disservice. You may want to watch that if you are curious about what happened. Pencil pushers without an inkling of understanding killed logic-based AI research is what happened. That was a bit like the comet that killed the dinosaurs and gave the mammals a chance. Meaning those adjacent to AI research directions, like probabilistic reasoning, connectionism and pattern recognition found an opening and they took their chance and dominated research since then. I am well aware of the connotations of my own metaphor. What can I say? Dinosaurs are cool. Ask any five-year old to explain why.

Re:Deep reinforcement learning is a waste of time (2019) by DukeBaset on Monday July 12, 2021 @05:54AM (#27808701)
Thank you for the detailed reply. Sadly killing off Logic based AI isn't the only crime we can attribute to pencil pushers. But I had some intuition I wanted to ask about.

Can a sort of vaguely type theory based - and I mean like in the sense of programming be used to reason about law and the like?

Like suppose some article says a man should pay 20% income tax but it says in some other act of some other law that a man shall pay 30% income tax. Like obviously I'm just giving an example but I mean like long cryptic legalese busting. Can we detect contradictions or show that the law or agreement esp like say something like TPP or whatever is inconsistent, by defining types and transactions?

I'm sorry I couldn't word it better. But I hope you get a gist of what I'm saying.

Re:Deep reinforcement learning is a waste of time (2019) by YeGoblynQueenne on Tuesday July 13, 2021 @06:50AM (#27819514)
I haven't really looked into that kind of thing. There's work I'm aware of in legal argumentation with logic programming and First Order Logic in general, for example:

https://link.springer.com/article/10.1007/BF00118494

A quick search on the internet also turns up this review that looks like it may have some information relevant to your question:

https://rd.springer.com/chapter/10.1007%2F3-540-45632-5_14

Re:Deep reinforcement learning is a waste of time (2019) by hervature on Monday July 12, 2021 @12:14AM (#27806987)
It is relatively easy to have a decent conversation on the internet. But when I state "a door" and you immediately ask about "any door", that is clearly moving the objective in the face of a the current state of affairs. If that is an accusation, then I am guilty.

I make and made no claims about RL being the end all be all. In fact, I am quite modest and would say that any researcher claiming their field is the path forward is incredibly egotistical and unlikely. However, to me, there is still progress being made and so there is no good reason to stop. Like everything, there is plenty of value of exploring until we understand completely why it is a bad path.

I think you may have missed the point of my post where I state that the entire field knows that RL is very far from something like folding laundry. Again, techniques are still being developed. But to state that "simulated" environment are a cop out is very silly since AlphaZero was trained in a "simulated" environment. Let alone the many algorithms being trained in simulated virtual environment for self-driving cars.

Re:Deep reinforcement learning is a waste of time (2019) by YeGoblynQueenne on Monday July 12, 2021 @05:04AM (#27808431)
Where did I 'state that "simulated" environment are a cop out'? Can you see what I'm complaining about? How can we have a productive conversation when I don't even know what you'll be responding to, when I make a comment?

You think there's any comparison between "simulating" a chessboard and simulating a road, or a door?

I think you are underplaying the difficulties of training robots with reinforcement learning and overplaying the utility of training robots in simulated environments to perform tasks in the real world. The latter in particular just doesn't work well.

Reinforcement learning is promising and its current marriage with deep learning is interesting, but no field of research goes anywhere without accepting its limitations and working to overcome them. I understand what you say in your original comment that there's work being done to address limitations, but the paper you linked is not that work. It's just business as usual - the kind of work that gets published, but doesn't change anything.

Re:Deep reinforcement learning is a waste of time (2019) by hervature on Monday July 12, 2021 @01:27PM (#27812758)
I will address your points here, but you should really review your own comments for the bad faith discussion you accuse me of doing.

> Sorry, it's probably not your paper. But it's as far from folding laundry as you can go. I don't even see the simulated robot folding simulated laundry anywhere yet and even "simulated cloth" is a joke - the simulation itself (of the behaviour of cloth) is not even close to the real world. Simulating a robot opening a simulated door might get your paper published, but simulating a robot folding simulated shirts won't even get you a simulated paper

You are right that you never said "cop out" but I am just summarizing your words. Your complaint is stating that RL doesn't tackle training in the real world. In some way, RL is "neglecting the responsibility of training a real physical robot". That is the definition of "cop out". To which I have already openly admitted that the responsibility of training a physical robot, while the long term vision, is not the intermediate goal at the moment.

No, there is no comparison between a simulating chessboard and the real world physics. All I am stating is that one just needs the simulation to be realistic enough. The problem is not the fact that it is simulated. If your complaint is that the physics engines are not good enough to train RL agents, that is completely orthogonal to the field of RL. Work is being done by these folks as well. Again, it also ignores every self-driving company who do use simulated virtual environment to train/test their algorithms.

Again, I am not overplaying or underplaying anything. I have yet to make any claim other than "RL people know we are very far from anything robotics related". I don't know how that can possibly be construed as overplaying.

If you feel that the LASER paper accomplished nothing, then we just fundamentally disagree. They did something that no one had ever done before in the field of RL. Is that not the definition of original research?

Re:Deep reinforcement learning is a waste of time (2019) by YeGoblynQueenne on Tuesday July 13, 2021 @06:45AM (#27819482)
But you're misrepresenting my comments again! I never said that training in simulation is a "cop out" because I don't think it's a cop out!

I agree that simulation results can be real resuls. I concede that the LASER paper made an original contribution to knowledge. My complain is that you overplay the result in the LASER paper when you say that "with a couple of expert examples and algorithm can learn to open a door effectively" but ommit to mention that this happened in simulation.

There's more to say here that I'd like to put into words at some point, but I don't think I'm in the right mood right now. I apologise if you were hoping for some more insightful remarks from me. Thanks for the conversation - I appreciate the time you took to give your point of view and share your expertise.