Show HN: Voyager – write a web crawler/scraper as a state machine in Rust

Re:Show HN: Voyager – write a web crawler/scraper as a state... by matsche on Wednesday December 30, 2020 @12:27PM (#25583636)

I haven't used scraper too much. I personally find the predicate approach of select.rs [0] easier to use. However in this case the selector approach just made more sense. Standalone html5ever can be a bit cumbersome to work with directly, scraper is basically an implementation of the html5ever's `TreeSink` trait, where as `select.rs` uses the hmtl5ever `RcDom` to parse the document but stores it in a more convenient way. If you look for a minimal approach you should at select.rs which basicially only depends on html5ever

[0] https://github.com/utkarshkukreti/select.rs

[ Parent | on HN ]

Re:Show HN: Voyager – write a web crawler/scraper as a state... by scudd on Wednesday December 30, 2020 @12:44PM (#25583843)

Hey just FYI, when you run the hackernews and explore examples without enabling the tokio feature flag, you get compilation errors about undeclared types for all the tokio stuff. I think these examples just need entires in the Cargo.toml requiring the tokio feature like the reddit example.

Once I add the tokio feature, they all run as expected.

[ Parent | on HN ]

Re:Show HN: Voyager – write a web crawler/scraper as a state... by matsche on Wednesday December 30, 2020 @12:51PM (#25583957)

Thanks, you're right. I fixed it just now by also requiring the tokio feature for all the tests in the Cargo.toml.

[ Parent | on HN ]