News for hackers. Stuff that matters.
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @01:12PM (#24839888)
Author here.

I spent much longer on this project than I expected to, and I learned a lot from the experience. I wrote this in hopes that it might be useful for others who want to digitize their old photos and home videos.

I'm happy to answer any questions or take any feedback about this post. I'm by no means an expert on digitization or video processing, but I'll gladly share what I know.

Re:My Eight-Year Quest to Digitize 45 Videotapes by jeffbee on Tuesday October 20, 2020 @02:31PM (#24840794)
For what it's worth, the timebase correction on a VHS deck has nothing to do with whether the audio is getting ahead of or behind the picture. If you think about the way VHS works, it's not possible for it to have audio-video sync drift. Reasons your audio was ahead/behind include that your capture device was recording the audio on a device with a free-running clock without reference to the video clock, or that the video capture device was dropping frames.

Timebase correction just fixes up the sync pulses so they arrive in an orderly fashion, which improves the picture quality.

Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @02:40PM (#24840888)
Oh, that's interesting. Thank you!

In retrospect, I should have spent more time studying the technical side of of digitization. My process was to just Google problems as I encountered them so I could get unstuck. I never took a step back to understand the fundamentals, but I think that would have saved time overall.

Re:My Eight-Year Quest to Digitize 45 Videotapes by rhn_mk1 on Tuesday October 20, 2020 @03:27PM (#24841409)
Last time I've seen time drift in a video file was from having the wrong frame rate in the video stream. A difference between 24 and 25 is rather obvious, but 23.97 and 24 fps might be hard to notice immediately.
Re:My Eight-Year Quest to Digitize 45 Videotapes by mikepurvis on Tuesday October 20, 2020 @04:58PM (#24842304)
Vaguely related, anyone who hasn't seen it, definitely go and watch Matt Parker's legendary explanation of why NTSC runs at 29.97fps:

https://www.youtube.com/watch?v=3GJUM6pCpew

Re:My Eight-Year Quest to Digitize 45 Videotapes by rhn_mk1 on Tuesday October 20, 2020 @05:20PM (#24842450)
That was a great video. I wonder why the frequency got changed when the original reason for the choice was the power network frequency. Wouldn't that cause all kinds of interference problems?
Re:My Eight-Year Quest to Digitize 45 Videotapes by mikepurvis on Tuesday October 20, 2020 @07:08PM (#24843226)
I looked into this a bit more just now and it seems like maybe the mains power synchronization was more about ease on the camera side than for the receivers:

"The NTSC field refresh frequency in the black-and-white system originally exactly matched the nominal 60 Hz frequency of alternating current power used in the United States. Matching the field refresh rate to the power source avoided intermodulation (also called beating), which produces rolling bars on the screen. Synchronization of the refresh rate to the power incidentally helped kinescope cameras record early live television broadcasts, as it was very simple to synchronize a film camera to capture one frame of video on each film frame by using the alternating current frequency to set the speed of the synchronous AC motor-drive camera. When color was added to the system, the refresh frequency was shifted slightly downward by 0.1% to approximately 59.94 Hz to eliminate stationary dot patterns in the difference frequency between the sound and color carriers, as explained below in 'Color encoding'. By the time the frame rate changed to accommodate color, it was nearly as easy to trigger the camera shutter from the video signal itself."

From: https://en.wikipedia.org/wiki/NTSC

Re:My Eight-Year Quest to Digitize 45 Videotapes by pkamb on Wednesday October 21, 2020 @01:42PM (#24850205)
I've had this same question myself about TBC and how that would possibly solve audio-video sync drift. I just asked a question here to codify the knowledge if you're interested in answering:

https://video.stackexchange.com/questions/32492/what-causes-...

Re:My Eight-Year Quest to Digitize 45 Videotapes by 867-5309 on Tuesday October 20, 2020 @04:18PM (#24841943)
Great posts, thank you. It must be a joy to hold so many hours of footage. Even if 90% may seem boring to you, think of the legacy you have created for your descendents. I have a total of 45mins of family tapes, of which around 5mins includes footage of my late father - the only footage of him which will ever exist. It is a yearningly small amount.

I was wondering if the digitisation company provided any kind of report. The sawtooth graph of your audio and video sync issue presents itself as being a hardware rather than software issue, envisioning ammonitic erosion of a plastic spindle. I'm just wondering whether this was fixed with the (despite your heroic attempts to source) right hardware, or if a software algorithm was involved (handcoded, ML?)

Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @04:33PM (#24842077)
Thanks for reading!

Yes, I feel very fortunate that my family recorded so much, and I do look forward to sharing it with my children.

One of the interesting parts of revisiting this footage is just seeing how attitudes towards video changed over time, at least among my circles. When I was a child, home video recording seemed so novel, even to my parents, that there was an excitement and enthusiasm in a lot of the videos just based on the sheer fact that we were recording a home video. Nowadays, I don't think people are as excited to record these kind of "slice of life" videos, even my friends with children.

The digitization company did provide a report, but it wasn't very detailed. They just listed which tapes seemed to have physical degradation or imperfections. I don't know what their methodology was, but I didn't get the impression that it was anything especially advanced. I imagine that they just trained some technicians to use their equipment and then run through a standard process for each tape.

Re:My Eight-Year Quest to Digitize 45 Videotapes by neRok on Tuesday October 20, 2020 @09:22PM (#24843997)
The "tearing problem" your video files have vs the professional is actually just the overscan area, see https://en.wikipedia.org/wiki/Overscan#Analog_to_digital_res...

Simply put, the entirety of 1 "frame" has more data than required, but a CRT TV would hide those "error margins" behind the borders. Your digital capture however has grabbed all the available data, and the pro would have done the same, but they would have then cropped it down to the desired area/resolution. You can see this when you compare the videos side by side, as the baby is "longer" in the pro version.

Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @10:07PM (#24844192)
Oh, interesting. I didn't realize that. Thanks for the explanation!

>You can see this when you compare the videos side by side, as the baby is "longer" in the pro version.

The baby in the video is actually me at five months old, although I'm quite a bit "longer" today.

Re:My Eight-Year Quest to Digitize 45 Videotapes by betamaxthetape on Tuesday October 20, 2020 @03:23PM (#24841365)
I'd be really interested to know what difference (if any) the S-VHS deck made compared to the regular one you were using.

I've got a decent capture setup (semi-Professional gear that captures at up to 10-bit uncompressed 4:2:2 - over 100GB an hour!) that I obtained basically for free since it is fairly old (early 2000s). Since good VHS decks are getting harder (and more expensive) to find, I haven't got a good[1] one yet. I'd be interested to know if the difference between the two decks was noticeable!

[1] I consider the recommendations on this thread: http://www.digitalfaq.com/forum/video-restore/1567-vcr-buyin... to be fairly good at separating good VHS decks from the rest of the pack.

Re:My Eight-Year Quest to Digitize 45 Videotapes by jeffbee on Tuesday October 20, 2020 @03:32PM (#24841466)
Here's the most hackerly way to do this: rent a 10MHz, 10-bit or better data acquisition rig and hook its probe to the head amplifier of your VTR. Play all your tapes and capture the raw tape signal to your computer. This will only require 45 GB per hour, i.e. almost nothing. Process the signal after the fact, with perfect field/line sync correction, whatever audio compression/limiting/equalization you want, etc.

VHS only has 3MHz bandwidth, let's don't pretend that a software-defined VTR is not practical.

Answering your actual question: S-VHS vs VHS deck should not make any difference for playback of VHS cassettes. Cassettes recorded in S-VHS cannot be played on a VHS deck, so that would be an obvious difference.

Re:My Eight-Year Quest to Digitize 45 Videotapes by betamaxthetape on Tuesday October 20, 2020 @04:28PM (#24842040)
There are in fact proof-of-concepts similar to what you describe[1] (this one is based off the Doomsday Duplicator[2], a hardware device used for Direct RF capturing off laserdisc players).

My (admittedly basic) understanding of VHS vs S-VHS decks is that due to the stricter tolerances required by the S-VHS standard, S-VHS decks typically have better transports than regular VHS decks (lower wow/flutter, better tracking, etc...). And of course many of the high-end decks have a built in TBC.

[1] https://github.com/happycube/ld-decode/issues/16

[2] https://www.domesday86.com/?page_id=978

Re:My Eight-Year Quest to Digitize 45 Videotapes by jeffbee on Tuesday October 20, 2020 @04:56PM (#24842281)
Nice. It really warms the ol' cockles to see that level of dedication to preservation. There already has been for years NTSC decoding in the gnuradio project, so doing this to VHS if you happen to have a USRP or similar peripheral might be almost trivial.
Re:My Eight-Year Quest to Digitize 45 Videotapes by nobleach on Tuesday October 20, 2020 @06:51PM (#24843105)
Wow, this is taking me back to a former life of mine! The idea of a built-in TBC scares me a bit though. I guess it's not a big deal if you don't plan on mixing with other sources. I'd always prefer the shared black-burst generator in that case.
Re:My Eight-Year Quest to Digitize 45 Videotapes by pronoiac on Wednesday October 21, 2020 @03:40AM (#24845618)
Ooh, I like this idea. I'm a bit surprised that it's a small multiple over what MiniDV had for similar resolution streams, around 12 gig per hour.
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @03:27PM (#24841410)
Honestly, I couldn't tell the difference in quality between the two VCRs, but I'm also bad at assessing video quality. I thought the quality on my professionally digitized videos was good, but videophile readers have given me feedback that it looks like near-amateur work.
Re:My Eight-Year Quest to Digitize 45 Videotapes by hexbinencoded on Tuesday October 20, 2020 @05:24PM (#24842475)
Our family tried S-VHS players and VCRs back in the day, but ultimate found them to fall short of top-of-the-line VHS VCRs.

S-VHS decks weren't necessarily any better than VHS ones, and the two formats are incompatible.

Mitsubishi, Panasonic, and Sony made the best VHS VCRs, IIRC. The best ones for capture maybe different, but this is what I recall for analog NTSC playback. S-Video connectors, if you can get them may help, but not always.

If I were going green-field the design of a VHS/S-VHS VCR from scratch today, I use some sort of solid-state helical-scan-equivalent head that can over-capture tape domains, pre-process data to align scan lines per PAL or NTSC, and output unencrypted HDMI.

Re:My Eight-Year Quest to Digitize 45 Videotapes by cowmix on Tuesday October 20, 2020 @01:54PM (#24840390)
I'm on a 35+ year quest to digitize 350+ tapes... and the python lib you detailed is just the thing I need.

thx!

Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @02:22PM (#24840696)
Oh, awesome. Glad that it was helpful!
Re:My Eight-Year Quest to Digitize 45 Videotapes by dr_orpheus on Tuesday October 20, 2020 @01:55PM (#24840406)
That was very insightful! As a Christmas present last year we paid someone to digitize all of my parents old tapes. It was awesome, but left off at the step of editing and I was disillusioned by the thought of trying to manually edit all of the clips. I might go back and try doing that now!
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @02:22PM (#24840692)
Thanks for reading!

Once I realized I could script the editing, that became the least stressful part to me. Fixing mistakes was just a matter of tweaking some code and re-running the scripts. Capturing, to me, was harder because I constantly worried about missteps that would cause quality losses I'd only discover later.

Re:My Eight-Year Quest to Digitize 45 Videotapes by kodt on Tuesday October 20, 2020 @10:18PM (#24844236)
So I actually had a very similar experience, purchasing the exact same TOTMC capture device from Amazon and a quality JVC S-VHS VCR on eBay. And later a higher quality capture card.

I found any free video editing options out there difficult to learn and unintuitive. I ended up using Sony Vegas which was a big improvement.

In my case I only needed to offset my audio track to get it to sync and it was fine. However some of my videos have that same tell-tale edge tearing, I didn’t bother trying to fix it.

Re:My Eight-Year Quest to Digitize 45 Videotapes by ooopsnevermind on Tuesday October 20, 2020 @07:59PM (#24843573)
Did going through this ever make you think about trying to record more from other family members, parents or grandparents, still alive? Curious because I ended up building an app as a tool for my own family but it was more for making new recordings vs digitizing old ones: https://trysaga.com
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @10:10PM (#24844206)
It actually didn't, but it probably should. I feel like social media has kind of tainted capturing video of personal events. Now, it kind of has the connotation that you're capturing something that can potentially go out to millions of people, whereas pre-2000, it felt like video was just a fun thing you'd record purely for the enjoyment of those close to you.
Re:My Eight-Year Quest to Digitize 45 Videotapes by soylentcola on Wednesday October 21, 2020 @09:58AM (#24847928)
This is very familiar to me. When I was a kid, I thought home camcorders were amazing. In my high school/college years, people were still at the tail end of using disposable and pocket cameras so I have barely any photos of myself or friends from those years.

Now when I see a camera out somewhere, all I can think is "oh jeez...are they gonna publish this on freaking Facebook?" and debate whether to ask (and risk coming off like the paranoid nutjob acting like a buzzkill).

Re:My Eight-Year Quest to Digitize 45 Videotapes by soylentcola on Wednesday October 21, 2020 @09:59AM (#24847933)
This is very familiar to me. When I was a kid, I thought home camcorders were amazing. In my high school/college years, people were still at the tail end of using disposable and pocket cameras so I have barely any photos of myself or friends from those years.

Now when I see a camera out somewhere, all I can think is "oh jeez...are they gonna publish this on freaking Facebook?" and debate whether to ask (and risk coming off like the paranoid nutjob being a buzzkill).

Re:My Eight-Year Quest to Digitize 45 Videotapes by bobsmooth on Wednesday October 21, 2020 @12:53AM (#24844837)
That seems like a really cool idea. Looking forward to an Android app.
Re:My Eight-Year Quest to Digitize 45 Videotapes by silicon2401 on Tuesday October 20, 2020 @02:53PM (#24841033)
Congrats, and thank you so much for sharing this. I've been planning out how to build out a personal archive/website to run at home, just for my family and disconnected from the internet, and mediagoblin looks like the perfect solution for hosting media in a more convenient way than just folders and files.
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @03:09PM (#24841219)
Thanks for reading!

>mediagoblin looks like the perfect solution for hosting media in a more convenient way than just folders and files.

I'll just warn you now that MediaGoblin is deceptively hard to work with. I chose it because I got it up and running quickly, but over time, I'd keep running into issues that should have been simple but required digging through their source and sometimes making my own patches. It also doesn't specify dependencies very precisely, so I constantly had issues installing it when its dependencies changed out from under it and broke everything. I tried hard for a while to get my patches merged upstream to prevent others from re-doing my work[0], but the project has effectively been unmaintained for the past 2+ years, save for a few brief spurts of work every few months.

If I were starting from scratch, I'd use a static site generator like Hugo or Gatsby to just generate thumbnails and URLs for each clip. You'd have to write a little custom code, but I suspect it's less time overall than you'd spend getting MediaGoblin to work.

[0] https://issues.mediagoblin.org/ticket/5574 (their bugtracker is currently down, which should give you a sense of the state of the project)

Re:My Eight-Year Quest to Digitize 45 Videotapes by silicon2401 on Wednesday October 21, 2020 @04:46PM (#24852006)
oh, that's maybe even more helpful than the original post itself! Appreciate the heads up. I'll take your advice and start with Hugo or Gatsby in addition to checking out Mediagoblin.
Re:My Eight-Year Quest to Digitize 45 Videotapes by dredmorbius on Wednesday October 21, 2020 @02:38AM (#24845304)
Kudos on an excellent writeup of a very worthwhile project. Your coverage of details from family and privacy to methods, process, tools, vendors, services, and frustrations is excellent.

Glad this was reposted as I missed it first time 'round.

Re:My Eight-Year Quest to Digitize 45 Videotapes by rendall on Tuesday October 20, 2020 @01:36PM (#24840181)
This was a really lovely read. Thanks for sharing those details
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @02:35PM (#24840841)
Thanks for reading! I'm glad you enjoyed it.
Re:My Eight-Year Quest to Digitize 45 Videotapes by djmips on Wednesday October 21, 2020 @08:29AM (#24847112)
Could you see a product being made from your experience?
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Wednesday October 21, 2020 @08:38AM (#24847167)
I thought about that, but I don't think I'd be able to offer something significantly better than what's already available.

One of the difficult parts of the process that you can't really "productize" is cataloguing. If you receive a stranger's videotapes, you wouldn't be able to label a clip, "Grandma Alice Visits for Bobby's Kindergarten Graduation," because you don't know anyone's names or relationships or the context of the video.

There might be a product in a tool that helps people edit and tag clips after they've digitized them, but I'm not passionate enough about that work to go after a product like that.

Re:My Eight-Year Quest to Digitize 45 Videotapes by Xavdidtheshadow on Tuesday October 20, 2020 @01:52PM (#24840366)
Thanks for this detailed writeup! I've been staring down a similar problem and I'm really grateful you took the time and energy to do the legwork
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @02:25PM (#24840734)
Thanks for reading! I'm glad to hear that the post is helpful in planning your digitization project.
Re:My Eight-Year Quest to Digitize 45 Videotapes by flerchin on Tuesday October 20, 2020 @01:42PM (#24840256)
Your insight to decouple editing from exporting along with using a csv as a data layer was really satisfying. Bravo.
Re:My Eight-Year Quest to Digitize 45 Videotapes by mtlynch on Tuesday October 20, 2020 @02:28PM (#24840761)
It only took me about 7 years to realize it. : )

But, yes, definitely. That was the biggest breakthrough in this project. I'm glad it came to me, as it otherwise would have taken me hundreds more hours, or I'd have given up before finishing.

Re:My Eight-Year Quest to Digitize 45 Videotapes by codazoda on Tuesday October 20, 2020 @02:31PM (#24840800)
I didn't read it all but I very much like the design, layout, and tone of the post. Nice job.