Re:Geocities Forever – neural network generated geocities pages
by gwern on Friday May 06, 2016 @09:35PM
(#11647906)
Yes, it turns out to be char-rnn/torch-rnn: 'RNN was 512-node, 3-layer' https://twitter.com/aanand/status/728738421758414849
He wasn't training on his own computer so aside from being much too small (for modeling at least 3 languages, text+HTML+CSS, I would start with 1k neurons and go up from there), it probably wasn't trained to convergence either. The 50MB size is maybe a bit on the small size but probably not the root cause of the low text quality.
The template here looks like it would go
"<script src="jquery.js"></script> <script src="sha1.js"></script> <script src="links.js"></script>
<script> (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) })(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
</script> <html>...[until generation limit, ~1600 characters; note the absence of any valid closing tags]
The low quality of the text might reflect the demands of simultaneously modeling text+HTML+CSS in a too-small RNN or insufficient training (training a high quality char-rnn/torch-rnn to convergence on the entire Geocities corpus, or even just a few hundred megabytes, could take weeks).