zeratul

Ask HN: Could you suggest a fast library for converting documents into a sparse matrix representation (e.g., COO or CSR) in any programming language? I'm guessing C beats most of the implementation? But there is also the issue of efficient n-gram hashing/indexing.

99.3% accuracy is not that great. Currently the best performer on this data set has 99.77% accuracy:

http://yann.lecun.com/exdb/mnist/

Many points raised here are valid concerns and should be addressed by the data scientists when designing the system.

Let's just not forget about the positives here: there is a company that openly uses term "A.I." and raised a lot of money. Am I the only one here waiting for the AI Winter to be over [1]? Maybe it's not the best A.I., yet, but they got the funding. Personally, I think that's great and I hope for others to come out of the closet.

[1] http://en.wikipedia.org/wiki/AI_winter

Brendangregg, do you know if anyone tried to use this platform to predict failures or DDOS attacks ahead of time? Or it is not feasible with this API? Thanks.

To Ryan Kirkman and Thomas Davis: could you make some of the data freely available to the Natural Language Processing community?

If this is something that interests people, then it will be worth the time to automate this process. I suspect that there are blog entries, twitts, facebook updates that already answer these questions. Now we have to mine them. "Gold standard" set of Q/A would make it easier.

EDIT: But I hope there will be a way of getting Q/A in a bulk, right? Thanks

NOTE TO SELF: This could be a data mining or text mining web app. Check if the interest grows in time.

I think no one is like Ed. There maybe be 25% of programmers that have a long standing experience, even with the new tools, but they don't share wisdom like Ed does.

Ed why can't you have your own company? Why can't you be the one who is hiring or at least teach others how to hire. I bet you can smell a talent from 1000 miles away - even if someone did not finish a good school; the world of software engineering would make so much more sense.

[EDIT:] I imagine a job interview; an interviewee says: "I didn't program this but I did program something more difficult"; interviewer makes a frowny face; Ed comes round the corner, puts his hand on the interviewer shoulder and says: "He did not call you stupid; he just said he is ready for the challenge - he has the experience".

S.A. is talking about general-purpose AI (position 2 in the RFS). This means processing natural language. There is a lot of progress but it's just slow so it's almost invisible.

Also it's a very difficult field of science. Now you need to be proficient in AI, machine learning, computational linguistics, linguistic corpora research, cognitive sciences, statistics, and sometimes physics if the text changes over time. Of course, you also need to be a good programmer. This combination of skills is very rare. Thus, very slow progress.

I suggest to start with well defined practical problems. For example, no one seems to do much with user generated reviews. There is some sentiment analysis but that is just a binary text categorization problem - not even close to general purpose AI.

It would be much more interesting to show a seller a time ordered stream of clustered reviews that depict only the most representative review for each cluster. This way a seller can see how his/her fixes/changes impact user reviews. Also it would be a great source for features and bug fixes requests. This is an ideal testing bed for clustering, novelty detection, categorization and mild inference. The inference is required because of sparseness of data.

This would create a good data set for a more general purpose AI. We would have reviews and text documenting changes and improvements of a new version of a product. Now the computer could start learning the dialog between users and product developers. Then, we are just one more step from statistical inference based question-answering system. Not a brute force system like "Watson" or a hand crafted rule base system like "Siri".

[EDIT:] I was thinking more about a decision support system that can recommend product changes. But in a way that maximizes customer satisfaction and minimizes the cost of implementation. The dialogue between past changes and customer reaction would give us the surface that needs to be optimized. This would generalize well to other domains where there is a text for request and a text for response - just to name one: clinical text in healthcare (position 5 in the RFS).

Maybe it's also good idea to revisit the reason why these systems are not widely in use. They all operate under so called "closed-world" assumptions. It means that the knowledge about the world is very limited.

Once the limitation is lifted the system has to deal with non-monotonic reasoning (http://en.wikipedia.org/wiki/Non-monotonic_logic) and that leads to multiple inheritance problem (http://en.wikipedia.org/wiki/Multiple_inheritance). Unfortunately, the multiple inheritance problem has, so far, only NP-hard solutions (http://ijcai.org/Past%20Proceedings/IJCAI-89-VOL-2/PDF/047.p...). [EDIT: think of a command "eggs are in a fridge"; to find eggs in a fridge you need to know that they are inside a container; you need to know its shape and how to open it; if your software knows that and inherits this knowledge each time when asked for eggs, then it will brake when the eggs are not in the box.]

We are in need of software that deals with large networks of human knowledge. Then we can take SHRDLU to the next level. Otherwise we are stuck with Will Smith movie (http://www.imdb.com/title/tt0343818/).

[EDIT2: there are many theoretical "suggestions" but very little software or practical internet applications]

Entangld, this is a great app. I hope you will find some time for healing.

Stress burns the mind in the same way as physical activity burns the muscle. At some point you need to give your mind a rest. I suggest this book:

http://www.amazon.com/Full-Catastrophe-Living-Wisdom-Illness...

or this one:

http://www.amazon.com/Wherever-You-Go-There-Are/dp/140130778...

This should help you keep your strengths intact regardless of your current situations ... who is to say that next endeavor will be better than the previous one.

I think posts like that should be accompanied with some tools that help to deal with grief or suicidal thinking.

Here is an online grief support group:

http://www.onlinegriefsupport.com/

Here is a phone number to a grief counseling group for those in US:

1-800-260-0094

Here is online suicide support group:

http://www.suicidepreventionlifeline.org/GetHelp/LifelineCha...

Here is a suicide prevention lifeline for those in US:

1-800-273-8255

[EDIT: suicide is a very complex disease that is far from being understood. I'm sure that prof. Yoshiki Sasai had to deal with much, much more than just two retracted articles.]

I also saw ~7000 visits first day but that was in 2012. Here are some conclusions of my web traffic analysis:

https://github.com/entaroadun/hnpickup/wiki/Hacker-News-Pick...

Here are multiple screen shots of the google web traffic analytics interface:

http://hnpickup.appspot.com/hnpickup_web_app_statistics_snap...

I would say that this is true for unstructured data that does not have availability of experts who participated in the creation of that data.

I noticed that by talking with the domain experts, for example radiologists that read head MRIs, you can greatly simplify the problem. It usually comes at the cost of experts annotating the data, which might be too expensive for some businesses. But then, I think, the data is analyzed just by scholars until they built-in the expert knowledge into their algorithms.

That's at least the pattern in healthcare data. The first one to notice a paper or source code like that dominates the market - this is just my conjecture.

I agree. What the internet needs is semantics not regular expressions. If there is a need for regular expression, it's only because semantics is not working, for example, the search engine does not have all the synonyms so I might write "do|does|done". We should be focusing more on textual query expansion (if the query hides some background knowledge), auto-tagging of web pages (if the web pages assume some background knowledge) and clever disambiguation of queries (if we don't know which background knowledge tap into).

You might be onto something. Could this be easily converted into an annotation tool for machine learning purposes? It looks like you could use it to annotate images and text. You need an interface to create annotation schema. Then, you need to have two or more annotators per text/image. And finally, you calculate inter-annotator agreement and export the data.

I know that there are couple startups that got seed money for "annotating the web". For example "Rap Genius".

Please let us know. I think they are using their own internal lat/lon mapping to encode images. I was able to get some images using their API:

https://ddnhehdam2vn0.cloudfront.net/malaysiaair2014_02/18/2...

https://ddnhehdam2vn0.cloudfront.net/malaysiaair2014_02/12/3...

but most of the numbers return 403. Above links roughly match upper left and lower right corner.