News for hackers. Stuff that matters.
Re:What technologies are under your site by tectonic on Saturday July 09, 2011 @11:07PM (#2746918)
Hey guys, I made this site and just gave a talk about it at SHDH. Someone must have submitted it. Thanks for all your feedback, I really appreciate it!
Re:What technologies are under your site by mmaunder on Sunday July 10, 2011 @12:23AM (#2747028)
Awesome! Suggestions:

Do a reverse lookup on the site's IP and add matchers for the hostnames that show up.

Mention if they're round robining their DNS.

Add nmap OS fingerprinting.

Do a traceroute and log the IP of the closest router (final hop) to the site and add matching for that.

Add a wiki interface and build a crunchbase like app.

Add archiving of data and monitoring over time (as netcraft did in their original app).

Re:What technologies are under your site by JoshTriplett on Sunday July 10, 2011 @12:29AM (#2747034)
nmap OS fingerprinting seems a bit aggressive, but reverse-DNS checking seems like a great way to check which hosting platform a site uses.
Re:What technologies are under your site by 16s on Sunday July 10, 2011 @09:38AM (#2747740)
nmap is too aggressive. It's a prelude to actual hacking attempts and labeled by IDS systems as such. Don't use it for this or you may end up in legal trouble.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @04:21PM (#2748535)
Agreed. UnderTheSite makes a specific point of only considering information that would be returned when a user's browser hits a website. It doesn't scan or probe ports / urls.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @12:56AM (#2747079)
Thanks for the great feedback!
Re:What technologies are under your site by known on Sunday July 10, 2011 @03:55AM (#2747300)
Re:What technologies are under your site by jf on Sunday July 10, 2011 @01:26AM (#2747119)
Awww, I wish I didn't have to leave SHDH before the lightning talks started!

I wrote a Ruby program to do something similar to what you're doing: https://github.com/jpf/domain-profiler - If you ever start profiling sites using information from places other than what the server returns, perhaps what I've done can help inspire you?

Re:What technologies are under your site by adrianwaj on Sunday July 10, 2011 @02:04AM (#2747166)
Is there a working example of domain profiler please?
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @01:26AM (#2747120)
Thanks, I'll check it out!
Re:What technologies are under your site by dorkitude on Sunday July 10, 2011 @04:49AM (#2747363)
Cool site! I'm having a lot of fun. However, I was confused by this flow:

First: http://drktd.com/8Jgn

After clicking the 'Add It!' link: http://drktd.com/8Iaf

The wording on the previous page had me expecting a page for manually adding technologies to my site's stack a la Bagcheck.com.

Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @04:50AM (#2747367)
Ah! This is a feature that I plan to add. Do you want to select technologies from a list, or write free-form text about your technology stack?
Re:What technologies are under your site by JoshTriplett on Sunday July 10, 2011 @03:49PM (#2748447)
I'd like to suggest avoiding manual additions of technology for as long as possible. Focus on adding more ways to match specific technologies. After all, a site could always advertise more technologies in server headers or meta generator tags.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @04:22PM (#2748537)
What if we propose an extension to humans.txt called technologies.txt where a site can self-describe their stack?
Re:What technologies are under your site by JoshTriplett on Thursday July 14, 2011 @07:07PM (#2765571)
A mechanism for sites themselves to advertise their stack seems like a great idea (though I'd prefer it not occur via fixed URLs like humans.txt or robots.txt, but via headers or meta tags). I'd just suggest not allowing arbitrary additions to a site's stack without any way to verify them.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @02:02AM (#2747162)
Follow @underthesite for updates as I implement all of the great suggestions that you guys have made here.
Re:What technologies are under your site by thomasfl on Sunday July 10, 2011 @04:48PM (#2748592)
Would love to see if it could detect more scripting languages and frameworks than PHP.
Re:What technologies are under your site by jbhelms on Sunday July 10, 2011 @12:14AM (#2747017)
It incorrectly stated that my site uses cold fusion.
Re:What technologies are under your site by MJR on Sunday July 10, 2011 @01:04AM (#2747089)
What's the site?
Re:What technologies are under your site by jbhelms on Sunday July 10, 2011 @02:05AM (#2747167)
Re:What technologies are under your site by suyash on Sunday July 10, 2011 @12:40AM (#2747060)
I posted it buddy, I liked your demo a lot!
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @12:57AM (#2747080)
Thanks :)
Re:What technologies are under your site by chime on Saturday July 09, 2011 @10:42PM (#2746872)
Minor bug: For my site, it says YUI. Having written it from scratch, I'm pretty sure there is no YUI anywhere. It appears to match jqueryui.js as yui.js.

Link: http://underthesite.com/technologies/YUI-Library/matchers/12...

Re:What technologies are under your site by tectonic on Saturday July 09, 2011 @11:05PM (#2746909)
You're right - This was a community-submitted matcher. I just reported the matcher as inaccurate: http://underthesite.com/technologies/YUI-Library/matchers/12...
Re:What technologies are under your site by JoshTriplett on Sunday July 10, 2011 @12:30AM (#2747037)
Please consider allowing a small icon for a technology, to make lists of technologies more immediately recognizable.
Re:What technologies are under your site by kingkilr on Saturday July 09, 2011 @11:59PM (#2746996)
Interesting, but seems a little primitive compared to the cleverness in https://github.com/mitsuhiko/probe
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @12:12AM (#2747014)
Does probe try and fetch signal urls? (Like admin pages.). UnderTheSite.com makes a point of only looking at data that would normally be fetched by your browser. Probing a site for urls could be considered offensive.
Re:What technologies are under your site by kingkilr on Sunday July 10, 2011 @12:16AM (#2747020)
Yes, it does probe at site URLs, you can look at libprobe.py to see all the indicators it takes into account.
Re:What technologies are under your site by bfirsh on Sunday July 10, 2011 @10:08AM (#2747796)
Re:What technologies are under your site by mikelbring on Saturday July 09, 2011 @10:47PM (#2746882)
Same idea as http://builtwith.com/ ?
Re:What technologies are under your site by tectonic on Saturday July 09, 2011 @11:06PM (#2746911)
Similar, but anyone can submit a new technology. It's a wiki of technology matchers.
Re:What technologies are under your site by Joakal on Saturday July 09, 2011 @11:08PM (#2746923)
Even same idea as http://netcraft.com
Re:What technologies are under your site by mike-cardwell on Sunday July 10, 2011 @12:24PM (#2748004)
I'd add SSL/https. I'd also add "Strict-Transport-Security" and "Content-Security-Policy", both of which can be seen by looking at HTTP response headers.
Re:What technologies are under your site by JoshTriplett on Saturday July 09, 2011 @11:54PM (#2746983)
I love this. I particularly like the broad definition of "technologies", and the variety of ways to write matchers.

More importantly, I like that users can easily add their own technologies and matchers.

Re:What technologies are under your site by wiradikusuma on Sunday July 10, 2011 @03:31AM (#2747265)
I checked my website, http://underthesite.com/sites/lelanggokil.com

Your analyzer mistakenly detected my website using Microsoft IIS and ASP.NET. Which is weird since it also detected Google App Engine (which is correct). ASP doesn't run on GAE.

Just a thought, do you think website owners should mention technologies they use in HTTP header? For example your analyzer can't detect that I'm using Java and Spring framework.

Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @03:46AM (#2747287)
You're correct - I'm looking into this, thanks for the bug report!
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @05:48AM (#2747438)
I think I've found and fixed the bug, thanks for the report. Does your site show correctly now?
Re:What technologies are under your site by christangrant on Saturday July 09, 2011 @11:29PM (#2746952)
fyi, If you click on the kid on the bottom right corner of http://underthesite.com/ he blinks.
Re:What technologies are under your site by arc_of_descent on Sunday July 10, 2011 @03:24AM (#2747254)
Nice. but you have to click on the kid, NOT!
Re:What technologies are under your site by bstar on Saturday July 09, 2011 @11:48PM (#2746980)
So yahoo.com is using jQuery now? There's like 10 references to YUI and zero for jQuery in the source... interesting algorithm.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @12:59AM (#2747082)
It was found on yahoo.match.com, which is a bug because it shouldn't go off domain. I'll look into it.

http://underthesite.com/technologies/jQuery/matchers/5?site=...

Re:What technologies are under your site by araneae on Saturday July 09, 2011 @11:23PM (#2746944)
Nice tool! Usually when I see a site I like I view source to see what it's made in, but it's not always that obvious.
Re:What technologies are under your site by ArchD on Sunday July 10, 2011 @03:09AM (#2747227)
To test the d3 matcher I just added, I tried the following URLs and get the error "Sorry, we were unable to reach the site":

http://whatdoyouworkfor.appspot.com/index.html

http://dustinphoto.iriscouch.com/gerrit/_design/app/index.ht...

I could reach them with no problems. These sites are just from some comments on this thread:

http://news.ycombinator.com/item?id=2746449

Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @03:19AM (#2747244)
Interesting- I'll take a look at this and see if I can figure out what's going on. Thanks for the bug report.
Re:What technologies are under your site by tectonic on Friday July 15, 2011 @02:59AM (#2766532)
So index.html is treated the same as /, and you return a 404 on /.
Re:What technologies are under your site by aorshan on Sunday July 10, 2011 @01:42AM (#2747137)
Noticed that not a single website is using django. Am I missing something?
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @01:43AM (#2747139)
I'm not sure that there is a django matcher yet. Django is hard to reliably detect. Do you know of a good signature (like a header or form token) that it always uses?
Re:What technologies are under your site by lloeki on Sunday July 10, 2011 @06:40AM (#2747512)
You could implement "maybe" matchers, that look for stuff which has a good chance of telling you what's underneath but not with 100% certainty.

Now, the following is a hack, but CSRF stuff[0] gives a good indication:

If there's a form, there's a good chance you'll find a construct like:

    <form action="." method="post"><input type='hidden' name='csrf_token' value="thetokenvalue" />
Also, there may be a 'csrftoken' value in the cookie.

You will also most probably find the jQuery function that sets X-CSRFToken on XHRs (see the doc[0] at #ajax). For prototype it'll look like this [1]

[0] https://docs.djangoproject.com/en/dev/ref/contrib/csrf/

[1] http://stackoverflow.com/questions/5551914/protecting-protot...

Re:What technologies are under your site by nbertram on Sunday July 10, 2011 @01:13AM (#2747099)
Nice site guys; reminds me of www.quarkbase.com in some ways. But I like the focus on what's under the hood - keep up the good work.
Re:What technologies are under your site by JoshTriplett on Sunday July 10, 2011 @12:12AM (#2747013)
What triggers the error message "The pattern that you entered appears to be too general - can you make it more specific for this technology?"? I tried to add a technology for the use of rel="nofollow", using an XPath expression //a/@rel[contains(., "nofollow")] , and got that error message. What do I need to do to make it more specific?
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @12:13AM (#2747016)
Good question, I think rel=nofollow must match too many websites. I'll look into it.
Re:What technologies are under your site by fus on Sunday July 10, 2011 @04:52AM (#2747368)
I cannot stand that unreal cogs image in bottom-right corner . Please make sure search works with special characters.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @04:53AM (#2747371)
What's wrong with the image?
Re:What technologies are under your site by Vivtek on Sunday July 10, 2011 @09:51AM (#2747767)
There are three cogs in a triangle. They couldn't move. That's all the explanation I can come up with - artistically I love the image.
Re:What technologies are under your site by shuri on Saturday July 09, 2011 @11:23PM (#2746945)
my 2c. Nice design (graphics). I would make the first page seem less busy. BuiltWith is going to be a tough competitor. You need to match them (precision/recall) and add stuff that they don't have (trending techs? Add info: who is the host provider? where in the world is it hosted? Response times? Bad link stats?...?)
Re:What technologies are under your site by davidcuddeback on Sunday July 10, 2011 @11:17AM (#2747891)
It says that my company's site [1] is running ASP.NET on Microsoft IIS, which it's not. To be fair, it also mentions Ruby on Rails, Apache, and Phusion Passenger, which are all correct. Aside from these minor glitches, this is a pretty cool project.

[1] http://identified.com

Re:What technologies are under your site by tectonic on Friday July 15, 2011 @02:57AM (#2766527)
This should be fixed now.
Re:What technologies are under your site by aphexairlines on Sunday July 10, 2011 @12:05AM (#2747006)
This doesn't really tell you anything about a site. Most of the interesting stuff happens on the server side.
Re:What technologies are under your site by djcapelis on Sunday July 10, 2011 @12:30AM (#2747039)
That's getting less and less true with the amount of stuff happening in Javascript.
Re:What technologies are under your site by aphexairlines on Sunday July 10, 2011 @12:55AM (#2747078)
Underthesite itself does its work behind a server:

http://underthesite.com/sites/underthesite.com

All you can tell from it querying itself is that the work happens somewhere behind rails.

Re:What technologies are under your site by arctangent on Sunday July 10, 2011 @05:13AM (#2747403)
Great idea. How about saving the results you generate and allowing people to search for sites based on the technologies they use. For example, I might want to see all sites using jQuery which are using something other than Apache.
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @05:15AM (#2747408)
Results are saved just not exposed yet. I'm still trying to figure out how best to do that.
Re:What technologies are under your site by westi on Sunday July 10, 2011 @03:54AM (#2747297)
This is cool.

Some bugs in the information returned though.

For example: http://underthesite.com/technologies/WordPress-Batcache-Plug...

"WordPress Batcache Plugin is closed source. "

Batcache is Open Source :)

Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @03:58AM (#2747308)
Fixed, thank you.
Re:What technologies are under your site by nfm on Sunday July 10, 2011 @02:00AM (#2747161)
Bang on for my app's site! Very cool.

In terms of features, I'd love to see more emphasis on the aggregate/comparison data. For example, most popular server side framework, most popular JS libraries, most popular hosting platforms and so on.

I like it a lot :)

Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @02:03AM (#2747165)
It's not fully-baked yet, but I'm starting to do just that: http://underthesite.com/compare/jquery_and_prototype-javascr...

Thanks for the feedback!

Re:What technologies are under your site by a3_nm on Saturday July 09, 2011 @11:57PM (#2746990)
I love how it gets confused by Hacker News: http://underthesite.com/sites/news.ycombinator.com
Re:What technologies are under your site by tectonic on Sunday July 10, 2011 @02:40AM (#2747203)
I challenge you to come up with a matcher for a technology detectable on HN. It's hard!
Re:What technologies are under your site by yeahsure on Sunday July 10, 2011 @06:15AM (#2747479)
Great idea - I like it.

I've been using a Firefox add-on for this, http://wappalyzer.com doesn't give as much info, but works fairly good.

Re:What technologies are under your site by namank on Saturday July 09, 2011 @11:24PM (#2746946)
Cool!

Though cannot reach this site: http://www.olin.wustl.edu/pages/default.aspx ...why

Re:What technologies are under your site by spontaneus on Saturday July 09, 2011 @10:44PM (#2746875)
I had this same idea a while back, but never had the time to build it. It could be a great way to generate some affiliate money. Good luck!
Re:What technologies are under your site by SallyG on Sunday July 10, 2011 @03:22PM (#2748392)
Good start. Looks like http://w3techs.com/sites does a bit more
Re:What technologies are under your site by paraschopra on Sunday July 10, 2011 @04:45AM (#2747356)
http://builtwith.com/ is another great alternative!
Re:What technologies are under your site by dorkitude on Sunday July 10, 2011 @04:54AM (#2747373)
builtwith.com does a better job with my site.

Under the Site misses MooTools, which is being loaded asynchronously on my site via Google Libraries API

Impressively, Built With even managed to detect that I was making an AJAX call to StackOverflow's API.

Re:What technologies are under your site by JoshTriplett on Sunday July 10, 2011 @12:49AM (#2747068)
Looks like you've already started to get spam in the technologies list.
Re:What technologies are under your site by Nican on Saturday July 09, 2011 @11:06PM (#2746914)
I would like to see the list of websites using IIS.
Re:What technologies are under your site by mcorrientes on Sunday July 10, 2011 @10:07AM (#2747792)
At first I thought this was a copy of lineofthought.com, but it has been created in a way it can stand for it's own.
Re:What technologies are under your site by ashwinurao on Saturday July 09, 2011 @11:06PM (#2746913)
A Chrome/FF extension would be much more useful.
Re:What technologies are under your site by tectonic on Saturday July 09, 2011 @11:08PM (#2746921)
There is a bookmarklet. Browser extensions are forthcoming.
Re:What technologies are under your site by jpr on Sunday July 10, 2011 @05:22AM (#2747416)
I'm going to build my next site in UTF-8, just like news.ycombinator.com.
Re:What technologies are under your site by clobber on Saturday July 09, 2011 @10:46PM (#2746881)
Tested this on a website written in ColdFusion and it's reporting back as ASP.NET
Re:What technologies are under your site by rmason on Saturday July 09, 2011 @11:35PM (#2746959)
I had the same experience with several sites that I own. No asp on the server, some others report just apache or IIS and completely miss ColdFusion.

Even Adobe.com, a ColdFusion site, doesn't list it.

Re:What technologies are under your site by arkitaip on Saturday July 09, 2011 @11:07PM (#2746919)
I tried ColdFusion powered metafilter and it reported nothing.
Re:What technologies are under your site by tectonic on Saturday July 09, 2011 @11:09PM (#2746926)
Please add a matcher for ColdFusion - I don't know it, but this is a wiki, so anyone can add it.
Re:What technologies are under your site by arkitaip on Saturday July 09, 2011 @11:48PM (#2746979)
I just understood the concept of tech matcher, which could be an incredibly powerful concept. I think you should highlight this more.
Re:What technologies are under your site by eurohacker on Sunday July 10, 2011 @04:20PM (#2748531)
does it recognize php frameworks - like cakephp, codeigniter,

or libraries like flourish

Re:What technologies are under your site by NARKOZ on Sunday July 10, 2011 @03:49AM (#2747290)