I 100% don't get it when websites with lots of content and good SEO just delete everything. Fatwallet (fuck you Rakuten). Yahoo Answers (hey, I didn't say good content).
Even on my own Drupal blogs that I didn't want to maintain/update anymore, I did a giant curl job, recursively regex'd out the login/comment submission fields and dumped them on s3. Voila. And I'm someone that would take 9 attempts to do FizzBuzz.
A bookmark you had to it 20 years ago still works and I still get some Adsense cheque residuals from it for zero effort.
All the good articles and design samples stay online, and maybe the original authors can get a little credit or name recognition from any search traffic that finds it.
The one "liability" to doing this would be for visitors to not realize the site isn't being updated anymore. If vice is still publishing content outside their website they'd have to make it clear that the site is archived and new content is elsewhere, or people might get the wrong idea and assume nothing new is coming.
For a reasonable fee, we'll archive your site, and give you back a copy of the assets you can turnkey host on the original domain for cheap with a static hosting solution (s3, cloudflare, etc).
Everybody wins
I'm really curious what the cost per byte would be to make it worthwhile to offer a "host this byte forever, for one up-front fee" service.
https://help.archive.org/help/archive-org-information/
> *What are your fees?*
> At this time we have no fees for uploading and preserving materials. We estimate that permanent storage costs us approximately $2.00US per gigabyte. While there are no fees we always appreciate donations to offset these costs.
There's some discussion about this idea on this thread, including comments by ?id=markjgraham, who manages the Wayback Machine, thoughts from John Carmack:
https://news.ycombinator.com/item?id=29639222
https://twitter.com/ID_AA_Carmack/status/1473327982605385735
Just pay some donation, redirect your DNS to the way back machine and bingo.
However, the footer of that website says 2014 and the about page is broken, so not sure if it's still supported.
Also, Cloudflare has a partnership with Web Archive and they offer something similar, but I think it's only made for temporary outages and only archives the most popular pages on your site
The same thing happens with brands - someone bought the Montgomery Ward brand at a bankruptcy auction or something - and with store inventory: once the store goes bankrupt, they just sell the entire store contents, right down to the fixtures, to a liquidator who brings in the "Going out of business! Everything must go!" signs.
We’re migrating away but they’ll squeeze out what they can from those that don’t/can’t/won’t.
But they still have to provide continuous support, some amount of updates to keep customers functioning, and maybe even get some new customers as a “value” option (that’s barely functional).
Happens to forums all the time (fuck you Internet Brands and Vertical Scope).
100000x easier to do all this with static web content.
That's too attractive to malware peddlers. It's not particularly widespread currently, mainly because most of the content has been centralized into the same big silos... But what you're envisioning here is just going to get abused by abusers
Archive Team is a loosely organised group of individual volunteers that share a common interest in Internet preservation, and develop tools and share notes to serve that goal. They're basically one of your old-school Mediawiki communities, with very little budget:
Internet Archive is a full-blown multimillion dollar `501(c)(3)` nonprofit, which functions as more of a general-purpose library. They maintain physical offices and datacentres in multiple countries, host many petabytes of data, do activism, run conferences, and when they develop custom tools it tends to be somewhat more advanced than the Archive Team's decentralized web scrapers, like custom book scanning hardware:
https://archive.org/details/eliza-digitizing-book_202107
A lot of the information in the Wayback Machine, which is run by the Internet Archive, was saved and contributed by Archive Team. For example, as of writing this comment, that is true of the latest snapshot of `https://www.vice.com/en`. You can see this with the "About this capture" button on a Wayback Machine capture.
Both groups have ways to receive monetary donations.
For Archive Team though, I wonder if it would be more useful to donate compute by running their Warrior archiving VM/container, or contributing code to their GitHub:
Only skip-a-heartbeat moment was when aws sent me an email saying that you I have “one or more S3 buckets that allow read or write access from any user on the Internet”
But none of my containers had write access. All of them had public read, but yeah, it’s a website and they know this: their own route53 DNS points to the containers.
They just sent the same generic mass email to everyone with any public container.
If so, their warning could have been phrased better, but isn’t incorrect.
I got absolutely tarred and feathered at an agency for suggesting this strategy and I just want to say thank you for validating what I wanted to do.
It was more of an issue that they had a bunch of clients on an old CMS system, and they did not want to make any changes the way that the sites were built or hosted.
I can make arguments for and against either side of this idea, it all depends how you want to run your business.
They were unwilling to spend time "training" the team to learn react (in 2020).
They were unwilling to let their senior FE dev spend any time with me to correct the CSS issues I was struggling with.
They used this deception and dishonesty to say "it didn't work" and wasn't worth any more time. I build a prototype in a week. It's not like I spent month(s) on it without any ROI. They just wouldn't look at it because that would mean acknowledging I and or / my ideas had value.
The closest thing I got to an answer is that the CMS they preferred, which was chosen 10 years ago by people no longer working there... was the only way they could support client sites. Because that's what they've been using. Turnover means it's so hard for us to support anything new because we have no time...
Basically they Brawndo'd me.
It was hard to stomach getting fired by the incompetent people driving the business into the ground when I was literally pleading with them to implement money saving measures.
The horse sometimes would rather kill itself than drink the clean water you've found... that's just life. It's hard to accept.
Nothing a simple notification bar can't solve
It takes zero effort to keep the book available (I know, I self-published a silly little one), and zero effort to include it in your accounting as long as there's a data feed and a computer.
Which sci-fi novel?
I read one of his later works and it was horrid, but Golden Age was incredible. The first several chapters were hard to get into, I quit several times and the people I loaned it to never got past that. But after that it takes off like a rocket. On a reread I found that the difficulty at first was just from so much being unfamiliar.
It's in a distant future with superintelligent AI, immortality, physical abundance and pervasive virtual reality. And in that setting he finds a deeply human tale of epic heroism.
Meanwhile, the author doesn't own the rights to the cover art/design so they can't just put it up either.
I'm not sure how else you'd want to handle it.
By having a minimum amount of foresight and putting into the initial contract an agreement which lets the parties maintain the online availability in a mutualy agreeable way.
It's just us as fans who are not happy.
It isn’t about the effort. This is the same thing people say about out of print video games, too.
It isn’t about difficulty, it is about incentives. They want you to buy new books and video games, and if you are reading/playing old free ones, that is reduced demand for what they are selling. Why would they want to help you satisfy your need for free?
There was some effort done with public groups by the ArchiveTeam, but I know they didn't get everything as there was no good way to find all groups.
That is not zero effort. That is at least 1.2 efforts.
For you and me doing this would be probably an afternoon’s work? (Maybe a bit more, maybe a bit less)
For someone less technologically inclined it could be seen as a big project. They need to find someone capable of doing it, they need to ask for a quote, they need to supervise the project otherwise the contractor doing it might just do a half assed job or none at all.
Not to mention they can only think about doing this if they have an inkling that it is possible. They might be operating in a mindspace where “maintaining the servers” is a large monthly expense. For example if years ago they were sold a CMS with all the bells and wistles for some $bigbucks recurring cost. If they are savvy business people they might have done their research to figure out if “this” can be done cheaper or not. But they might not realise that it is possible to change the requirements such to achieve a massive reduction in cost. This is especially true if they treat the cost of servers as a kind of black box.
And very often the people who are providing them with IT services are not incentivised to tell about this option. Will they tell the business owner that oh by the by for half the monthly recurring cost they are paid the business could find someone who puts the page on ice and for the other half runs it for the next decade? Of course not! Doing so would under cut their income stream. That would be crazy. In fact they might spread all kind of FUD and sabotage attempts at scraping the site.
If you are a business built on top of the web you need someone who's tech savvy in-house.
Maybe they deserve their faith. But it's sad that the next generation will miss this insightful content because they gave up.
I use it/them as a search engine on steroids. Maybe it is time more people also do so.
> Not to mention they can only think about doing this if they have an inkling that it is possible.
Newspapers have a bad history of "experimenting" with enabling online comments and then deciding the experiment failed and delete them all. You're a newspaper, you're not supposed to delete history when you don't like it!!!
And then they complain when people use social media as their newspaper.
Except that internet commenters are very weird, and like the least representative sample ever. Not quite as weird as Wikipedia editors, but still really weird.
Unixtime is the number of seconds since 1/1/1970. Subtract a few hours for timezone post processing and you get 12/31/1969 as a date. Indicated time zero or null or missing value trying to get formatted as a date.
This is Dec 31 1969 at 7:00 PM in Eastern Standard Time.
I wish I was joking, but reducing the amount of stuff that is owned/managed for the sake of it is a common philosophy. Another way to put it: they're focusing.
We think it would have been extremely low effort to keep a static site running, they probably thought not having to think about it at all was worth the loss.
Hehe
So if the website shutdown coincides with the shutdown of a company or a division within the company, that might be why. And since a website will usually not shut down if it's turning a considerable profit to begin with, just deleting everything can often look like the best option.
it's because even this cost outweighs how much they care about the content, which is 0... the people who make decisions like this aren't scrimping pennies or interested in preserving effort... they're looking for the simplest way to get millions of dollars into their pockets
doing the bare minimum to maintain a library of content indefinitely isn't it
these are the kind of people who would happily set fire to a library if they could get away with insurance fraud
OMG I haven't thought about the site in a number of years.
Used to be such a common thing to see in google search results.
There's no way they'd give up the ads dollars they get from he existing stuff... That makes no sense
Way to jump out of the boiling pot and immediately place your head in the lion's jaws...
Google "ad networks"
It won't be enough to run a big media company but more than enough to keep old content around
Within a few months someone else had snapped it up, created a site, used the same content (I assume from the Internet archive), and was using it to link farm.
At least that's what I thought when I saw the new site with a few links that were not there before.
I only discovered it because they also left the Gmail address that I was using for the site. So I got a couple of emails from folks wanting to update info on the site.
Feels a bit weird, tbh.
No way to determine who is behind it, and what would I do anyway?
https://jalopnik.com/maryland-license-plate-url-directs-gamb...
I guess using subdomains of a main site isn't that bad an alternative.
>It wouldn't even be a significant cost and ads would recoup it anyway
Just so you know... investors would probably think it's just a waste of time, which means a waste of money as they (or their financial handlers) would have to keep an eye on this website. Remember, if they think this website is going to make them under $XX an hour they will just nuke it, they are not attached emotionally or otherwise to this "product", exception being shareholder founders but e.g. Steve Jobs.
I came across this thread on DataHoarders before seeing it here and was glad to see so many jump at the opportunity to help archive it on our own ends. In case anyone may be interested to jump in, here’s the link to that thread. https://www.reddit.com/r/DataHoarder/s/fnY46CuYOq
Technical people, who look at these decisions from across the Internet, aren’t exactly known for their humility, or assuming that non business people may have perfectly valid reasons for their decisions.
A lot of the cool things we were doing at the outline were on the CMS side of things, its a shame we were never able to share it
I was involved with some sites recently that stopped publishing new content. The plan is to keep most of it around for now. But I have no illusions that if it becomes a "project" for some reason or another in a few years, it will just be turned off.
If I stop posting new content, have I stopped publishing? Or is it if I stop serving, then I've stopped publishing?
Saying “we will no longer publish content” could very well mean they are taking down the site; thus it is no longer published
I think it’s very likely that this is what’s going to happen because even a static archive site is going to cost money and dilute their new social brand
Perhaps they’ll repost their old stuff on their new channels but it will nevertheless suck for old timers like me that refuse to use social media
I don’t know, but I suspect it’s a similar answer.
The deal to work with the media outlets likely dictates they cannot have their own distribution channel even for historical content.
No idea about this case specifically, but in general this is an assumption that only hurts you in most cases. People have motives and people want to be good. If their "good" is your good is a different question - a question that is often worth figuring out.
But more likely, too your last point, lots of people have a distorted sense of good, in that they can’t separate it from their own self-interest. What is good for them will be justified or reframed to be ’good,’ and this is often unconscious.
People are complicated, and negative interactions do tend to have an outsized influence on molding our behavior
Maybe in 2007. But you've got an ad-blocker in your browser now just the same as every other Vice visitor does.
No single raindrop thinks they were responsible for the flood.
Can't you?
I thought everyone with an account could upload. If it'd stay, is another question and they'd definitely not archive against the expressed will of copyright holder, but you could give it a try.
You can store the resulting WARC file wherever, be it on S3 or under your pillow.
As an archivist, I urge everybody here to not reinvent the wheel, please..
[2] https://en.m.wikipedia.org/wiki/WARC_(file_format)
[3] https://www.iso.org/standard/68004.html
[4] https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...
You practically cannot preserve and make available something if the copyright holders don't want you to. If the copyright situation is complicated you bear the risk.
You can say, that this is how it is supposed to be, but it is not like it works in the non-digital realm. You could argue that we'd need something like digital monument protection, where artifacts can be preserved against the copyright holders will.
You'd need a corp to deal with the money, people to handle things like takedown requests and expired copyrights, maybe some IT functions to keep things up when suppliers change stuff. It wouldn't be that straightforward.
(I've seen this issue raised on previous site shutdowns, quite probably Gawker.)
Keeping your own personal clippings file preserves words but not provenance.
I wanted to say, that Dr. Dobb's did that. https://drdobbs.com/ But some time in the last few years something mus have gone broken. You can't open the article links anymore.
Which tends to be what happens. Something breaks. No one can be bothered to fix it. And at some point they don't renew a domain and take the content offline.
So do you believe this is an action taken out of spite, or that there may be circumstances that a bunch of hackers aren't taking into account?
I'm genuinely curious, because I don't believe it's spite, but can't think of the reason. Outside of some long-tail potential liability for which the value of maintaining the site doesn't overcome.
* Potentially expensive licensing fees for the CMS, depending what it is, and migration costs to something “free” if not. * Ongoing costs to maintain the site (security updates etc). * Of course, hosting costs.
I agree in this case it would be smarter to keep it running in a stripped-down form since there’s still value there, then work to find buyers or someone who has an interest in keeping it running.
You start off by saying: it costs nothing to keep this site up. No downside.
But there may be some downside from an SEO perspective, which the minuscule ad revenue would not offset.
Even brands with good SEO, with pages which are making money, take some of them down occasionally. They idea is that you want to refocus all your SEO on the pages that matter most - which means pulling others.
Here, you may want all the SEO “juice” to go to your videos, to push the maximum number of users there. If some of them are getting diverted to the website, and that pays less per click, you may in fact be losing money on it vs. having no website.
https://thedissolve.com/ is still up, and it's been eight years, and I'm really glad they didn't take it down. It's still a useful resource.
It wouldn't even be a significant cost and ads would recoup it anyway
There should be a way to donate a website to the Internet Archive so that they run an online archive on it, basically keeping the site frozen forever (rather than relying on the Wayback Machine which has worse UX)