News for hackers. Stuff that matters.
Re:GitHub Incident by freedomben on Tuesday March 22, 2022 @12:17PM (#30767962)
Mind sharing some details on your install? Which database are you using (especially wondering about sqlite with gitea)? Are you using the docker install, or from package, or in k8s?

No pressure, but I'm seriously considering standing one up but it would need to be production ready.

Re:GitHub Incident by 5e92cb50239222b on Tuesday March 22, 2022 @12:40PM (#30768311)
It's a really boring setup, I'm unlikely to share anything of much value here.

No k8s or anything like that. An Ubuntu LTS virtual machine on top of I don't know what hypervisor (probably Hyper-V). Gitea requires very few resources.

Data is stored in PostgreSQL 11. Probably should be upgraded at some point, but it's working fine for now.

A Drone CI instance runs on that same machine. It controls CI workers on a dozen other VMs.

Everything is in Docker containers under docker-compose for two reasons: easy upgrades, and the ability to shove all data into a single directory.

caddy for HTTPS.

Sonatype Nexus for package repositories (caching upstream npm, nuget, maven, composer, and our own internal repos). This one is pretty heavy and will probably be moved to a separate machine at some point.

Never had any issues with any of these in about three years we've been running this setup.

I was a bit facetious with the "zero downtime" statement. It has about 30 seconds of downtime per month, but it is always planned and outside of working hours deep into the night.

There's been zero seconds of unplanned downtime.

Basically you do:

    $ docker-compose pull
    $ docker-compose up -d
and half a minute later it is running new versions. If anything is broken (it hasn't been yet), call the sysadmin guy, and a few minutes later he restores a VM snapshot.

Backups are being done on the hypervisor's level. I don't know about that much.

Re:GitHub Incident by freedomben on Tuesday March 22, 2022 @12:44PM (#30768369)
cool, thank you!
Re:GitHub Incident by raxxorrax on Tuesday March 22, 2022 @12:24PM (#30768070)
Gitea is absolutely production ready, we use an older version by now and don't even think about it much because it just works so well. Exactly what you want in production (production = software should work more often).

Actually not "vendoring" dependencies from github is very much not production ready in many cases.

Re:GitHub Incident by qbasic_forever on Tuesday March 22, 2022 @12:38PM (#30768271)
IMHO storage is the big issue to figure out with a production ready gitea installation. Gitea only supports the local filesystem for storing bare repo data and not anything like S3 or other distributed or highly available storage systems (it supports S3 only for git LFS data right now, it is a long awaited open issue to get the bare repos there too).

So the big issue for any non-trivial or highly available setup is going to be how you get high availability for the local storage volume. There are tons of options with various tradeoffs and levels of complexity here--simple local disk RAID, distributed filesystems like gluster or ceph, etc. I think this is the real crux of getting a good gitea instance going.

No matter what figure out and test a good backup and recovery solution!

Re:GitHub Incident by freedomben on Tuesday March 22, 2022 @01:15PM (#30768779)
Ah interesting! I read somewhere that it uses the database for that, but I think they were mistaken.

Thanks that's quite helpful. If we do it in prod we'll have to accept single-instance.

Re:GitHub Incident by qbasic_forever on Tuesday March 22, 2022 @02:25PM (#30769603)
Yeah I forgot to mention too a lot of complexity goes away if you host it on a good IaaS platform. On AWS for example you could just mount an EBS volume as the bare repo root and use RDS for a database. Then you just have to deploy and manage an EC2 instance to host gitea's main process. It's more expensive to use those managed platforms but might be well worth the cost vs. doing it all yourself with a bare metal deployment of everything.