News for hackers. Stuff that matters.

A comment on DuckDuckGo Lite now requires a User-Agent header. Parent comment

Re:DuckDuckGo Lite now requires a User-Agent header by 1vuio0pswjnm7 on Tuesday September 20, 2022 @01:37AM (#32908033)
That's not what causes the malformed request, but thanks for the heads up. Apologies for the inadvertence.

Try this:

   addcr <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
   POST /lite/ HTTP/1.1
   host: lite.duckduckgo.com
   content-length: 6
   content-type: application/x-www-form-urlencoded
   user-agent: test
   connection: close
   
   q=test
   EOF
addcr is from djb's daemontools.

If do not want to compile/install daemontools, try something like:

   x=$(printf '\r\n');
   sed "s/$/$x/" <<EOF|openssl s_client -connect lite.duckduckgo.com:443 -ign_eof
   POST /lite/ HTTP/1.1
   host: lite.duckduckgo.com
   content-length: 6
   content-type: application/x-www-form-urlencoded
   user-agent: test
   connection: close
       
   q=test
   EOF
[ Parent | on HN ]
Re:DuckDuckGo Lite now requires a User-Agent header by 1vuio0pswjnm7 on Tuesday September 20, 2022 @07:39PM (#32919225)
NB. If using HTTP/2, then no UA header is needed. The examples I provided use HTTP/1.1

    curl --http1.1 --resolve lite.duckduckgo.com:443:52.142.124.215 -d "q=test" -A "" https://lite.duckduckgo.com/lite/
"Several characteristics of HTTP/2 provide an observer an opportunity to correlate actions of a single client or server over time. These include the value of settings, the manner in which flow-control windows are managed, the way priorities are allocated to streams, the timing of reactions to stimulus, and the handling of any features that are controlled by settings."

HTTP/2 has some fingerprinting enablers that HTTP/1.1 does not. The above is from the HTTP/2 RFC.

Further reading: https://www.blackhat.com/docs/eu-17/materials/eu-17-Shuster-...

[ Parent | on HN ]
Re:DuckDuckGo Lite now requires a User-Agent header by opardalis on Tuesday September 20, 2022 @07:59AM (#32910507)
Ok... The above example works now, and with user-agent omitted, I still get search results. My point to others watching/following this thread is that the initial claim is false, lite.duckduckgo.com does indeed return search results if your request header does not have a user agent field.

So to be clear, there is no issue here.

[ Parent | on HN ]