Re:DuckDuckGo Lite now requires a User-Agent header
by opardalis on Monday September 19, 2022 @11:25PM
(#32907305)
Yeah, but your 2nd example (with the UA header included) also returns a 400 response code. I am able to POST a search query to lite.duckduckgo.com just fine using another tool while also completely omitting the user agent request header (not empty string, completely omit the UA header field). I think your openssl s_client syntax is wrong, and thats why your getting a 400 status code, 400 is usually malformed request syntax.
[ Parent
| on HN ]
Re:DuckDuckGo Lite now requires a User-Agent header
by 1vuio0pswjnm7 on Tuesday September 20, 2022 @01:37AM
(#32908033)
That's not what causes the malformed request, but thanks for the heads up. Apologies for the inadvertence.
Try this:
addcr <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
POST /lite/ HTTP/1.1
host: lite.duckduckgo.com
content-length: 6
content-type: application/x-www-form-urlencoded
user-agent: test
connection: close
q=test
EOF
addcr is from djb's daemontools.If do not want to compile/install daemontools, try something like:
x=$(printf '\r\n');
sed "s/$/$x/" <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
POST /lite/ HTTP/1.1
host: lite.duckduckgo.com
content-length: 6
content-type: application/x-www-form-urlencoded
user-agent: test
connection: close
q=test
EOF
Re:DuckDuckGo Lite now requires a User-Agent header
by 1vuio0pswjnm7 on Tuesday September 20, 2022 @07:39PM
(#32919225)
NB. If using HTTP/2, then no UA header is needed. The examples I provided use HTTP/1.1
curl --http1.1 --resolve lite.duckduckgo.com:443:52.142.124.215 -d "q=test" -A "" https://lite.duckduckgo.com/lite/
"Several characteristics of HTTP/2 provide an observer an opportunity to correlate actions of a single client or server over time. These include the value of settings, the manner in which flow-control windows are managed, the way priorities are allocated to streams, the timing of reactions to stimulus, and the handling of any features that are controlled by settings."HTTP/2 has some fingerprinting enablers that HTTP/1.1 does not. The above is from the HTTP/2 RFC.
Further reading: https://www.blackhat.com/docs/eu-17/materials/eu-17-Shuster-...
Re:DuckDuckGo Lite now requires a User-Agent header
by opardalis on Tuesday September 20, 2022 @07:59AM
(#32910507)
Ok... The above example works now, and with user-agent omitted, I still get search results. My point to others watching/following this thread is that the initial claim is false, lite.duckduckgo.com does indeed return search results if your request header does not have a user agent field.
So to be clear, there is no issue here.