News for hackers. Stuff that matters.
Re:Avoid Git LFS if possible by goodcjw2 on Wednesday May 12, 2021 @05:46PM (#27135548)
A side topic: is there a concrete reason why github's LFS solution has to be so expensive?

IIRC, it's $5 per 50GB per month? That's really a deal breaker to me and wondering whether people actually use LFS at volume will avoid LFS-over-GitHub.

Re:Avoid Git LFS if possible by tux1968 on Wednesday May 12, 2021 @05:53PM (#27135610)
$60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me. What price would you expect?
Re:Avoid Git LFS if possible by Dylan16807 on Wednesday May 12, 2021 @09:15PM (#27137327)
> $60 / year for a decent fraction of a hard disk and the associated backup resources, seems pretty fair to me.

What you describe sounds fair to me. The problem is that 50GB is not a decent fraction of a hard disk.

Re:Avoid Git LFS if possible by 0xbadcafebee on Wednesday May 12, 2021 @11:32PM (#27138095)
I don't know if you remember back when service providers would put customers' entire dataset on a single set of spinning platters. The drive would of course die and the customer would get super pissed off when the provider would say "You were supposed to keep a backup..." So now providers like GitHub and GitLab are basically super-redundant storage, network, and application providers, who also happen to run Git.

If you store 50GB in AWS S3 (US-East-2), download 1000GB, do 100 PUT operations and 1000 GET operations, the cost is $89.68 per month.

Considering that GitHub isn't just providing you with storage, but a complete Git LFS solution plus storage, plus traffic that you can just use and not think about, I think it's worth the expense. But then again I probably wouldn't store binary blobs in Git.

Re:Avoid Git LFS if possible by Dylan16807 on Thursday May 13, 2021 @05:39AM (#27140294)
The number you're citing is basically entirely bandwidth, which has two main problems.

One is that amazon has an enormous markup on bandwidth, compared to their other products.

The other is that GitHub does not actually let you download each file 20x in a month and "not think about" it. 50GB of space for a month only gets you 50GB of bandwidth.

If Amazon didn't explicitly ban people from using Lightsail bandwidth with other services, you could put together an all-AWS package that has 150GB of high quality S3 storage and enough bandwidth to download it 2-3x per $5 (minimum order quantity 2). For a service like B2 you could store 250GB twice (each copy having its own cross-server RAID) and download it once for $5. At digitalocean $5 will get you 250GB of probably-redundant data with 1TB of bandwidth, though it eventually tapers off toward 167GB/$5.

Re:Avoid Git LFS if possible by Danieru on Wednesday May 12, 2021 @09:48PM (#27137541)
Just be thankful you are not on Gitlab. We pay $60/year for 10GB.

I need the data for my team, so we pay, but I use it as an excuse to force the team to clean up data every so often. As a game company we need large repos. It was either Gitlab or Azure DevOps.

Re:Avoid Git LFS if possible by acdha on Wednesday May 12, 2021 @10:17PM (#27137698)
$60/year is $5/month. You said “game” and “company” — how is that not a tiny fraction of your total costs?
Re:Avoid Git LFS if possible by goodcjw2 on Wednesday May 12, 2021 @10:58PM (#27137947)
Nowadays games assets are just crazily huge. AAA games tends to ship at 50G. Now think about the raw assets size and saving all the intermediate revisions.

It's quite easy to burn 1000G of storage on GitHub/GitLab (again, don't forget all the revisions). That puts just the storage cost at $6000/year. At this price point, it's really worth hosting on your own.

As Danieru mentioned they are forcing ppl to manually do cleanups, that probably indicates the storage costs are even higher which worth manual interventions.

Re:Avoid Git LFS if possible by Danieru on Friday May 14, 2021 @11:51AM (#27155560)
Can I afford it? Yes. Do I need it? Yes.

An I happy paying 6 dollars a year for a single gigabyte: I'd rather pay less.

Edit: I pay attention to costs, and repo size is the sort of thing which has a. Habit of growing. All else being equal I'd prefer my team receive the money. A dollar I can give to my team/employees feels good, a dollar paying for over priced storage feels bad.

Re:Avoid Git LFS if possible by acdha on Friday May 14, 2021 @12:28PM (#27156113)
Doesn't that also apply to, say, GitLab paying their team? The pricing for a highly-available service isn't as cheap as it could be but it doesn't seem terribly far off of S3's pricing.

I know that a small game studio must have a tight budget — and yours looks like a really interesting project (just subscribed) — but it seemed like an awfully strong objection to what I would have assumed would be a small fraction of your total expenses.

Re:Avoid Git LFS if possible by Dylan16807 on Friday May 14, 2021 @03:33PM (#27158770)
It's good for GitLab to pay their teams, but that stops being a justification to pay them more once the profit margin for a service gets high enough.

> it doesn't seem terribly far off of S3's pricing

GitHub's offering is close to S3 but only because AWS charges so much for bandwidth. The storage portion is less than a quarter of the equivalent bill.

And then GitLab is charging 5x as much as GitHub.

Re:Avoid Git LFS if possible by Dylan16807 on Wednesday May 12, 2021 @10:58PM (#27137948)
The problem is that it's per ten gi-- actually, looking it up, I found a page saying there is a 10GB cap!
Re:Avoid Git LFS if possible by moshmosh on Wednesday May 12, 2021 @06:30PM (#27135950)
It's a very large markup on the small-user retail cost of the basic thing they're providing (web-accessible, access-controlled file storage—see, for example, BackBlaze B2) but that's utterly typical of services that can get away with charging you a "convenience fee" for that sort of thing once you're on their SaaS. 2-3x markup isn't unusual, and that's about what this is, and that's above typical retail—even if GH's not managing the storage and such themselves, they're likely getting an even better (bulk) rate.
Re:Avoid Git LFS if possible by klodolph on Wednesday May 12, 2021 @07:13PM (#27136352)
Yes, I would estimate that the markup is more like 5-10x for GitHub LFS.
Re:Avoid Git LFS if possible by goodcjw2 on Wednesday May 12, 2021 @07:33PM (#27136528)
Exactly. For us we are dealing with lots of large gaming assets and those are burning through those $50 data packs like butters :)
Re:Avoid Git LFS if possible by adkadskhj on Wednesday May 12, 2021 @06:17PM (#27135837)
Yea i actually wrote my own file chunking and general git-lfs-like backend for this exact reason. I liked Git LFS, but Github's pricing felt insane for my indie dev. For my needs i could backup onto a local server, network drive, or w/e at an insanely cheaper price.

Hell even uploading to an S3 compatible API was insanely cheaper than Github.

That and i really hated the feeling that Git LFS was being designed for a server architecture. I didn't have an easy way to locally dump the written files without running an HTTP server.

There are a couple Git LFS servers that upload to, say, S3 - but i really just wanted a dumb FS or SSH dump of my LFS files. Running a localhost server feels so.. anti-Git to me.

Re:Avoid Git LFS if possible by elcritch on Thursday May 13, 2021 @01:01AM (#27138588)
Do you have a repo for it?
Re:Avoid Git LFS if possible by adkadskhj on Thursday May 13, 2021 @10:18AM (#27142351)
I do, but i'm not comfortable sharing it. It's an experimental project, very new, and not polished enough for general use. A POC if you will. Plus this is an anonymous account :P

While i was writing it i found the basic process of basic large file storage to be insanely simple with Git. I debated doing the same thing but backed by a seasoned backup solution, like Borg/Bup/etc.

Re:Avoid Git LFS if possible by toomuchtodo on Wednesday May 12, 2021 @06:02PM (#27135694)
Consider S3 storage and egress costs. You’re paying a flat rate to store and then pull that 50GB data (edit: removed an incorrect statement here).
Re:Avoid Git LFS if possible by Someone1234 on Wednesday May 12, 2021 @06:07PM (#27135736)
Just for clarity, it is 50 GB of storage and 50 GB of bandwidth.

So definitely not "as much as you want." If you pull it too many times you may get charged another $5.

Re:Avoid Git LFS if possible by remram on Wednesday May 12, 2021 @08:46PM (#27137103)
For S3, that's $1.15 storage ($0.023/GB) + $4.50 transfer ($0.09/GB) so it's actually very comparable.
Re:Avoid Git LFS if possible by Dylan16807 on Wednesday May 12, 2021 @09:18PM (#27137348)
Yikes. Being comparable to amazon in price, purely because of how much amazon overcharges for bandwidth, is not exactly a flattering comparison.

If you were to add the $1.15 storage cost onto a reasonable bandwidth number like $0.01/GB you'd just about reach a third of $5.

Re:Avoid Git LFS if possible by remram on Wednesday May 12, 2021 @09:23PM (#27137387)
I can't argue with that. I didn't expect the prices to align when I researched it.
Re:Avoid Git LFS if possible by toomuchtodo on Wednesday May 12, 2021 @06:33PM (#27135976)
Thanks for the correction!