When scraping and parsing (or writing integration test DSL), I always start out with CSS selectors. But always hit cases where they lack or require hoop-jumping and then fall back on Xpath. I then have a codebase with both CSS-Sel and Xpath, which is arguably worse then having only one method.
I suspect here, one uses this tool untill CSS selector limitations are getting in the way, after which one switches to another tool(chain)
E.g. when you have a list of numbers on the website, XPath can calculate the sum or the maximum of the numbers
Or you have a list of names "Last name, First name", then you can remove the last name and sort the first names alphabetically. Or count how often each name occurs and return the most popular name.
Then it goes back to selection, e.g. select all numbers that are smaller than the average. Or calculate the most popular name, then select all elements containing that name
Element with src or href attr: //[@src or @href] or multiple conditions: //article[@state = "approved" and not(comments/comment)]
Element with more than two children: //ul[count(li) > 2] Element with matching descendents: //article[//video]
Element text containing substring: //p[contains(text(), "Foo")] Attribute containing substring: //a[ends-with(@href, ".jpg")]
Numerical attribute selection: //product[@price > round(2.5 @discount)] //product[sum(//[starts-with(name(), 'price-')]/@price) > 0]
Attribute values: //a/@href Text values with spaces normalised: //a/normalize-space(text())
Match all attributes or elements or text nodes: //user/@ or //user/node() or //user/text() or //user/comment()
Basically from any node in a document you can select its ancestors, children, descendants, siblings, attributes etc, and filtering has the same power as selecting does - in CSS there's :not() that can apply to selection or filtering, with :has() finally on the way and no :or(). CSS selectors match against HTML elements and they're great for that almost all of the time, but while you can filter by attribute value including substring and even by regular expression, for text there's :empty.
But for a query syntax you need to be able to select attributes and text content as well as elements. Either extend XPath to support #id and .class syntax
//#user-xyz//note/text() //code.language-js/@name
or extend CSS to at allow selecting attrs and text
#user-xyz note :text code.language-js @name
The former is more powerful, the latter a quick hack (if they only appear at the end of the selector anyway) with instant payoff.
Looks like that psuedo-class has not been implemented in the kuchiki library that htmlq uses though.
# Find all elements li and select the parent element for each
//li/..
# Find all element nodes with a child element named li
//*[li]
# Non-abbreviated queries
/descendant::li/parent::*
/descendant::*[child::li]
# CSS using :has
:has(> li)(: XQuery comments are marked by mirrored smilie faces, like this. :)
An empty query is not valid. There needs to be something besides the comment
<users>
{
for $user in //users
let $comments = //comment[@uid = $user/@id]
where count($comments) > 0
order by $user/lastName, $user/firstName
return <user id="{ $user/@id }">
<name>{ concat($user.firstName, " ", $user.lastName) }</name>
<comments count="count($comments)">
{
for $c in $comments return <comment id="{ $c/@id }" />
}
</comments>
</user>
}
</users>
It's also in nixpkgs, though for some reason the nixpkgs derivation is marked as linux-only (i.e. not Darwin). (Edit: probably because the fpc dependency is also Linux-only, with a linux-specific patch and a comment suggesting that supporting other platforms would require adding per-platform patches)
[0]https://en.wikipedia.org/wiki/XPath [1]https://github.com/benibela/xidel