Re:What technologies are under your site
by tectonic on Saturday July 09, 2011 @11:07PM
(#2746918)
Hey guys, I made this site and just gave a talk about it at SHDH. Someone must have submitted it. Thanks for all your feedback, I really appreciate it!
Re:What technologies are under your site
by 16s on Sunday July 10, 2011 @09:38AM
(#2747740)
nmap is too aggressive. It's a prelude to actual hacking attempts and labeled by IDS systems as such. Don't use it for this or you may end up in legal trouble.
Re:What technologies are under your site
by tectonic on Sunday July 10, 2011 @04:21PM
(#2748535)
Agreed. UnderTheSite makes a specific point of only considering information that would be returned when a user's browser hits a website. It doesn't scan or probe ports / urls.
Re:What technologies are under your site
by jf on Sunday July 10, 2011 @01:26AM
(#2747119)
Awww, I wish I didn't have to leave SHDH before the lightning talks started!
I wrote a Ruby program to do something similar to what you're doing: https://github.com/jpf/domain-profiler - If you ever start profiling sites using information from places other than what the server returns, perhaps what I've done can help inspire you?
Re:What technologies are under your site
by JoshTriplett on Sunday July 10, 2011 @03:49PM
(#2748447)
I'd like to suggest avoiding manual additions of technology for as long as possible. Focus on adding more ways to match specific technologies. After all, a site could always advertise more technologies in server headers or meta generator tags.
Re:What technologies are under your site
by JoshTriplett on Thursday July 14, 2011 @07:07PM
(#2765571)
A mechanism for sites themselves to advertise their stack seems like a great idea (though I'd prefer it not occur via fixed URLs like humans.txt or robots.txt, but via headers or meta tags). I'd just suggest not allowing arbitrary additions to a site's stack without any way to verify them.
Re:What technologies are under your site
by chime on Saturday July 09, 2011 @10:42PM
(#2746872)
Minor bug: For my site, it says YUI. Having written it from scratch, I'm pretty sure there is no YUI anywhere. It appears to match jqueryui.js as yui.js.
Re:What technologies are under your site
by tectonic on Sunday July 10, 2011 @12:12AM
(#2747014)
Does probe try and fetch signal urls? (Like admin pages.). UnderTheSite.com makes a point of only looking at data that would normally be fetched by your browser. Probing a site for urls could be considered offensive.
Re:What technologies are under your site
by mike-cardwell on Sunday July 10, 2011 @12:24PM
(#2748004)
I'd add SSL/https. I'd also add "Strict-Transport-Security" and "Content-Security-Policy", both of which can be seen by looking at HTTP response headers.
Your analyzer mistakenly detected my website using Microsoft IIS and ASP.NET. Which is weird since it also detected Google App Engine (which is correct). ASP doesn't run on GAE.
Just a thought, do you think website owners should mention technologies they use in HTTP header? For example your analyzer can't detect that I'm using Java and Spring framework.
Re:What technologies are under your site
by tectonic on Sunday July 10, 2011 @01:43AM
(#2747139)
I'm not sure that there is a django matcher yet. Django is hard to reliably detect. Do you know of a good signature (like a header or form token) that it always uses?
Re:What technologies are under your site
by JoshTriplett on Sunday July 10, 2011 @12:12AM
(#2747013)
What triggers the error message "The pattern that you entered appears to be too general - can you make it more specific for this technology?"? I tried to add a technology for the use of rel="nofollow", using an XPath expression //a/@rel[contains(., "nofollow")] , and got that error message. What do I need to do to make it more specific?
Re:What technologies are under your site
by shuri on Saturday July 09, 2011 @11:23PM
(#2746945)
my 2c. Nice design (graphics). I would make the first page seem less busy. BuiltWith is going to be a tough competitor. You need to match them (precision/recall) and add stuff that they don't have (trending techs? Add info: who is the host provider? where in the world is it hosted? Response times? Bad link stats?...?)
Re:What technologies are under your site
by davidcuddeback on Sunday July 10, 2011 @11:17AM
(#2747891)
It says that my company's site [1] is running ASP.NET on Microsoft IIS, which it's not. To be fair, it also mentions Ruby on Rails, Apache, and Phusion Passenger, which are all correct. Aside from these minor glitches, this is a pretty cool project.
Re:What technologies are under your site
by arctangent on Sunday July 10, 2011 @05:13AM
(#2747403)
Great idea. How about saving the results you generate and allowing people to search for sites based on the technologies they use. For example, I might want to see all sites using jQuery which are using something other than Apache.
Re:What technologies are under your site
by nfm on Sunday July 10, 2011 @02:00AM
(#2747161)
Bang on for my app's site! Very cool.
In terms of features, I'd love to see more emphasis on the aggregate/comparison data. For example, most popular server side framework, most popular JS libraries, most popular hosting platforms and so on.
Do a reverse lookup on the site's IP and add matchers for the hostnames that show up.
Mention if they're round robining their DNS.
Add nmap OS fingerprinting.
Do a traceroute and log the IP of the closest router (final hop) to the site and add matching for that.
Add a wiki interface and build a crunchbase like app.
Add archiving of data and monitoring over time (as netcraft did in their original app).
I wrote a Ruby program to do something similar to what you're doing: https://github.com/jpf/domain-profiler - If you ever start profiling sites using information from places other than what the server returns, perhaps what I've done can help inspire you?
First: http://drktd.com/8Jgn
After clicking the 'Add It!' link: http://drktd.com/8Iaf
The wording on the previous page had me expecting a page for manually adding technologies to my site's stack a la Bagcheck.com.
Link: http://underthesite.com/technologies/YUI-Library/matchers/12...
More importantly, I like that users can easily add their own technologies and matchers.
Your analyzer mistakenly detected my website using Microsoft IIS and ASP.NET. Which is weird since it also detected Google App Engine (which is correct). ASP doesn't run on GAE.
Just a thought, do you think website owners should mention technologies they use in HTTP header? For example your analyzer can't detect that I'm using Java and Spring framework.
http://underthesite.com/technologies/jQuery/matchers/5?site=...
http://whatdoyouworkfor.appspot.com/index.html
http://dustinphoto.iriscouch.com/gerrit/_design/app/index.ht...
I could reach them with no problems. These sites are just from some comments on this thread:
http://news.ycombinator.com/item?id=2746449
Now, the following is a hack, but CSRF stuff[0] gives a good indication:
If there's a form, there's a good chance you'll find a construct like:
Also, there may be a 'csrftoken' value in the cookie.You will also most probably find the jQuery function that sets X-CSRFToken on XHRs (see the doc[0] at #ajax). For prototype it'll look like this [1]
[0] https://docs.djangoproject.com/en/dev/ref/contrib/csrf/
[1] http://stackoverflow.com/questions/5551914/protecting-protot...
[1] http://identified.com
http://underthesite.com/sites/underthesite.com
All you can tell from it querying itself is that the work happens somewhere behind rails.
Some bugs in the information returned though.
For example: http://underthesite.com/technologies/WordPress-Batcache-Plug...
"WordPress Batcache Plugin is closed source. "
Batcache is Open Source :)
In terms of features, I'd love to see more emphasis on the aggregate/comparison data. For example, most popular server side framework, most popular JS libraries, most popular hosting platforms and so on.
I like it a lot :)
Thanks for the feedback!
I've been using a Firefox add-on for this, http://wappalyzer.com doesn't give as much info, but works fairly good.
Though cannot reach this site: http://www.olin.wustl.edu/pages/default.aspx ...why
Under the Site misses MooTools, which is being loaded asynchronously on my site via Google Libraries API
Impressively, Built With even managed to detect that I was making an AJAX call to StackOverflow's API.
Even Adobe.com, a ColdFusion site, doesn't list it.
or libraries like flourish