News for hackers. Stuff that matters.
Re:Artificial Intelligence Lecture Videos by inovica on Saturday October 08, 2016 @04:16PM (#12668537)
I have a side project to monitor the homepage of every domain and I want to detect the type of site - ecommerce, blog, forum etc. I've just started on it and ultimately I want to be able to automatically extract data, such as product information from ecommerce sites. Would these videos help here or is there anywhere else that someone would recommend. I know I sounds very naive here... and I am, but there might be someone here who can give me a steer. I've started looking at AI/ML... or whatever its now called and getting a bit confused!
Re:Artificial Intelligence Lecture Videos by objclxt on Saturday October 08, 2016 @06:58PM (#12669101)
> ultimately I want to be able to automatically extract data, such as product information from ecommerce sites

A little OT, but although you could probably train models to figure that out, many (if not most) eCommerce sites should already be tagging their sites with semantic tags for product information, since it assists with indexing for search engines.

Here, for example, is the Schema.org for a product (https://schema.org/Product).

Re:Artificial Intelligence Lecture Videos by inovica on Sunday October 09, 2016 @09:15AM (#12671057)
That's really useful. Depends how many of the engines (Magento etc) use this - but I'll look into it. Thank you
Re:Artificial Intelligence Lecture Videos by fnbr on Saturday October 08, 2016 @09:44PM (#12669518)
You could absolutely do it. Here's a few things you could do to get started (feel free to email me if you need more specific help, or have any additional questions- my email's in my profile):

1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site.

2. Label the data as belonging to each type of site. This is going to be tedious and take some time. You can also try hiring people via Mechanical Turk to do this.

3. Use [Weka](http://www.cs.waikato.ac.nz/ml/weka/) or [Vowpal Wabbit](https://github.com/JohnLangford/vowpal_wabbit/wiki) to run some preliminary estimations on the model. Weka and VW are great tools as they come with a lot of the configuration done out of the box, so you won't have to write any code to get started.

Check your results and see how happy you are with them. Weka has a lot of visualization capacities, so you can see how the data that you've collected aligns with the different types of sites.

Now, you can start iterating, which is the key part of any ML project. Consider which aspects of the model you think you can improve on- adding more data, adding more kinds of sites, using a different machine learner.

Re:Artificial Intelligence Lecture Videos by afro88 on Sunday October 09, 2016 @12:47AM (#12669935)
> 1. Get as much data about each site as possible. Try to find data that you think will separate the non-ecommerce sites from the ecommerce sites. This data should be in a table form- think of an Excel spreadsheet, where each row represents a different site.

This will be the hardest step. Feature extraction from html for identifying specific things like products, for any given site, is very hard (in my experience, which I will admit is pretty limited). Would love to be proven wrong though.

Re:Artificial Intelligence Lecture Videos by fnbr on Monday October 10, 2016 @01:22PM (#12678481)
I was thinking you could approach it as a document classification problem, where you extract the text from the HTML and work with that.

I had a lot of success finding duplicate bug reports by comparing the text from the bug reports with reference documents in a variety of topics (e.g. security, networking, C++), and getting a sense of how similar the text is to the reference text. That gives you a score of how relevant each subject is to the document.

You could do something similar here- download the text from 10 ecommerce sites, run some sort of topic extraction algorithm (like LDA) on it and then compare the text from the sites you're trying to classify with the text from the reference sites.

Re:Artificial Intelligence Lecture Videos by inovica on Sunday October 09, 2016 @09:15AM (#12671061)
Thank you. I'll check these out and also drop you a line
Re:Artificial Intelligence Lecture Videos by westbrookt on Saturday October 08, 2016 @05:06PM (#12668740)
It is possible for subject recognition, but the data extraction would be much more difficult. For the simplest subject recognition, you would need a corpus with all the kinds of subjects that you want to recognize, and many examples of each. It would be very tough as a first project in machine learning.

Edit Note: I'm very new to machine learning as well, but this is what I've gathered so far.