News for hackers. Stuff that matters.
Re:Show HN: Get clean readable content from any webpage by Theodores on Thursday April 24, 2014 @06:19PM (#7643466)
Thanks for posting that, it is exactly what I need.

I want to scrape a diverse selection of web pages for 'the gist' of the main content purely to put into a solr search engine. At the moment I am doing surprisingly well just going on the meta descriptions - in essence meta descriptions should be what I am after - but meaningfully scraped content is the ideal and what I need if no meta description available.

I am sure you have lots of grand plans for this, however, one thing that I wish I could do is search through my web history, at a content level.

For instance, earlier today I found some code was no longer working properly. I could remember bits of the tutorial I needed first time around but finding it again was a chore (on a narrow subject then every search result from Google is purple/visited, so you can't immediately identify the link).

If I could have done some 'search my previous browsing history' then I could have got there a lot quicker. In this 'imaginary search' only the pages that are kept open and scrolled through get 'internally archived'. There could also be snapshots of the pages in question in the results as an aid to memory.

Another thing I would like to see done with this type of tool is a sensible spell check. For decades we have had spell checks in text entry areas but never on a finished web page. A little widget to show the words with problem spellings with wavy underlining that would work on any web page would save no end of woes for people that have to proof read things online. If it also highlighted sentences too long for good copy that would be very useful too.

Can these sorts of applications be built on what you have here?

P.S. Funny that you wrote this 186 days ago:

https://news.ycombinator.com/item?id=6581317

Times have changed...

Re:Show HN: Get clean readable content from any webpage by flippyhead on Thursday April 24, 2014 @08:55PM (#7644101)
I'm curious, do you use Google History? If so, does it not meet your needs?
Re:Show HN: Get clean readable content from any webpage by crashandburn4 on Friday April 25, 2014 @06:58AM (#7645702)
Very simple request for the developer, you couldn't put up screenshots/examples of text that has been extracted along with a (cached) link so that people can see how it looks without downloading and testing it?
Re:Show HN: Get clean readable content from any webpage by willlma on Friday April 25, 2014 @03:52AM (#7645228)
Good to see an OSS version. How does it handle pagination? From my experience (Pocket, Readability, Readable (now Evernote Clearly), Boilerpipe) that's the big differentiator. But frankly, with the increased use of media queries, I find myself using this view less and less, and the only feature I miss is pagination.

The optimal reading environment on a responsive website is usually the tablet view. Narrow enough to make jumping lines easy, narrow enough to prevent big ads on the side columns. I'd love to see a tool that could force media queries on bigger displays and simply center the text. You'd still get a feel for what the web designer was going for, but it's much cleaner. On none responsive sites, it could use the same metrics as this tool to determine which elements are article elements and simply center them and remove the rest, while maintaining the fonts, colors... The medium is the message and all that jazz.

Re:Show HN: Get clean readable content from any webpage by snipek on Friday April 25, 2014 @02:48AM (#7645075)
Send email message to [email protected] , with a URL in subject line. Then you'll get a readable email of that web page. You can read it offline on mobile mail client. That's how I read long web articles these days.
Re:Show HN: Get clean readable content from any webpage by snipek on Friday April 25, 2014 @02:56AM (#7645097)
If you get used to this way, using email to read web articles, you can use it to navigate web pages in mailbox. For example, send anything to [email protected] to get hack news, and click & send mail to read each articles.
Re:Show HN: Get clean readable content from any webpage by aleksi on Friday April 25, 2014 @03:36AM (#7645177)
There is also a similar service: http://www.ukeeper.com
Re:Show HN: Get clean readable content from any webpage by snipek on Friday April 25, 2014 @02:38PM (#7648295)
Good to know. Thanks!
Re:Show HN: Get clean readable content from any webpage by kiliankoe on Thursday April 24, 2014 @06:45PM (#7643586)
Thank you for this, I just started using node-readability for a project and I'll gladly give this a spin. The second url [0] I tried passing it however unfortunately returned an empty body. node-readability is able to parse the contents of tagesschau.de seemingly without problems.

[0] http://www.tagesschau.de/ausland/krise-in-der-ostukraine112....

Re:Show HN: Get clean readable content from any webpage by bndr on Friday April 25, 2014 @02:33AM (#7645026)
Thanks! It may be encoding problem, I'll take a look at it today.
Re:Show HN: Get clean readable content from any webpage by rpedela on Thursday April 24, 2014 @08:16PM (#7643969)
How does this compare to Apache Tika?
Re:Show HN: Get clean readable content from any webpage by kephra on Thursday April 24, 2014 @10:25PM (#7644409)
I did not read the source, but have a question:

How do you deal with web pages, where the content is dynamically created using javascript/ajax? Do you execute this JS, or just ignore this growing antipattern?

Re:Show HN: Get clean readable content from any webpage by grimtrigger on Thursday April 24, 2014 @10:43PM (#7644473)
Are dynamically created websites actually common with text-heavy web sites? I've only seen them on interaction-heavy web apps.
Re:Show HN: Get clean readable content from any webpage by dredmorbius on Friday April 25, 2014 @09:39AM (#7646332)
Blogger (dynamic templates) and Business Insider are two that come to mind as suspects. Both render blank pages in Chrome w/ JS disabled.
Re:Show HN: Get clean readable content from any webpage by didgeoridoo on Thursday April 24, 2014 @11:14PM (#7644562)
Sadly, yes.
Re:Show HN: Get clean readable content from any webpage by bndr on Friday April 25, 2014 @02:36AM (#7645035)
Sadly, no dynamic js is being executed.
Re:Show HN: Get clean readable content from any webpage by TD-Linux on Thursday April 24, 2014 @10:35PM (#7644446)
This reminds me of Firefox Mobile's reader mode - something I find invaluable.

Unfortunately they don't seem to have a desktop equivalent, yet.

Re:Show HN: Get clean readable content from any webpage by atopal on Friday April 25, 2014 @06:34AM (#7645653)
Reader mode for desktop is coming (no specific version targeted though). Here is the bug: https://bugzilla.mozilla.org/show_bug.cgi?id=558882

And here are the mock-ups: http://people.mozilla.org/~mmaslaney/readermode/index.html

Re:Show HN: Get clean readable content from any webpage by bgnm2000 on Thursday April 24, 2014 @05:50PM (#7643294)
would love to see this as a bookmarklet (that hits the user's npm) for demo purposes
Re:Show HN: Get clean readable content from any webpage by bndr on Thursday April 24, 2014 @06:04PM (#7643390)
I'm not sure I understand what you mean with "that hits the user's npm"
Re:Show HN: Get clean readable content from any webpage by jzig on Thursday April 24, 2014 @06:10PM (#7643420)
He means a button he can press in the browser that does the work for him.
Re:Show HN: Get clean readable content from any webpage by recroad on Thursday April 24, 2014 @11:06PM (#7644542)
Re:Show HN: Get clean readable content from any webpage by earwolf on Friday April 25, 2014 @06:18AM (#7645626)
Pagination. Comments. Profit.