Actually - that's pretty hard to do. What would probably be more "doable" is to take a transcript (done manually by a transcriptionist) and sync up the text with the video.

You could provide links in a transcript which when clicked take you to the part of the video in question, if you'd like to see it. You could also go the other way, and use a transcript to provide subtitles.

Any ideas on how to make this happen? My intial thought is to exploit pauses and breaks in the sound and map them to sentences, etc. Voice mapping would help too, so you could map a sentence to Paul Graham and another sentence to the interviewer.

What would be pretty cool (actually, REALLY cool now that I think about it) would be desiging a transcript markup language that could be married to a video. Has anyone done this? Why not?

I know that speech recognition probably is not quite good enough yet :-( I have another interest in this: since I am not a native english speaker, I find it often hard to understand everything that is being said in a video. English subtitles would often be helpful (especially for those videos of interviews or conference speeches etc.).

Perhaps Amazons mechanical turk could come in handy here? Would a clone of the mechanical turk specialised on movie transcripts be feasible? Or is it too special interest? (Perhaps most people either understand the words or they would prefer a translation?).

As for syncing subtitles: I don't know the details anymore, but I remember from ripping DVDs that the subtitles come simply in a text file along with the movie. So presumably something like you propose already exists.

Sure, that exists because the movie studios usually ship movies out with a transcription. I don't think something like this exists for run-of-the-mill type videos. In the example of the video above, if there WERE a transcript, you could use the software I propose above to provide subtitles automatically.

There's a service in Japan where you can associate text to a point in a video. You type text like chat while watching a video, and the text is recorded with the timestamp and played back along the video. Association is bidirectional---you can jump to certain time point in the video from the text as well. Currently most people just making fun, putting funny text on videos, but it would be quite useful to attach transcription by collaboration of audiences.

Unfortunately they only provide closed service for registered users: http://www.nicovideo.jp/ (Actually they originally used YouTube as a video source and had no access restriction. But they got so much popular that YouTube had to shut down access from their site.)

Wow actually this is the idea I applied with to YC. thanks for the link It will help me improve my own product.

I'll just need to refresh my japanise ;)

There's actually two startups that have working demos of this...

use juwo - it can do this

I visited your site - but there is no information about how your software works. There's just some text that refers to a video. But there's no link anywhere to such a video.

If you want to convince me that your product can do what it says - take the video talked about in this thread, run it through your software and give us a link so we can see what you mean.

Right now, it can only handle stuff that plays in Windows Media Player. This file is a mov file that plays in Quick Time. (I ran into problems controlling QT from Javascript. If you are good in Javascript, maybe you can chip in to help?).

I shall be releasing about 12 demos of juwo on my website shortly - hopefully this week. (I work on it in my free time so have been extremely slow this year).

Great. Make sure you post a link to them.

It is released. Please see

http://news.ycombinator.com/comments?id=14254