News for hackers. Stuff that matters.

aarondl0

Re:Concourse CI on Tuesday July 18, 2017 @02:54AM (#14794726)
UI freezes are completely client side and related to the elm implementation and your browser's execution of the code. The size of your VM doesn't matter at all.

ATC is a more of a dependency scheduler. The code [1] shows that it basically gets all pending jobs, and then runs them. There's no concept of queueing or maximum number of jobs, you just have to hope your limits are high enough (max containers, max tasks in systemd, max fds in the same) and that your machine doesn't fall over in the attempts.

The "massive" scheduling system also has no idea what nodes need work and which do not [2] so the idea is to heavily overprovision until it doesn't fall over (on top of already beefy requirements which others have alluded to in this thread).

You can not serialize multiple pipelines. Only within pipelines. If I have 10 pipelines, they will all run independently and there's nothing you can do about it other than attempt serialization with the pool resource (which we've recently had problems with - it also appears to be buggy and we're looking at submitting patches).

I've got a tremendous amount of experience with this system and I believe it's everything I made it out to be. The rebuttals you've provided to my issues are simply a lack of understanding of our context, not every user will have the same experience with any given product.

[1] https://github.com/concourse/atc/blob/master/scheduler/sched...

[2] https://github.com/concourse/concourse/issues/675

Re:Concourse CI on Monday July 17, 2017 @04:20PM (#14791976)
Excellently designed system. Very poorly implemented.

Our team's been using it since it's initial releases. It's been nothing short of disastrous for all but the smallest pipelines.

The design is great. Keeping configs in yaml instead of little white boxes in a Jenkins database is much better. Pipelines as a first class concept. It feels inspired by a functional programming language. You get great build reproducibility since there are no workers that get dirtier over time if you forget to clean up. The resource model is awesome. Very cool stuff. I'm hoping every CI system learns from what's here. Second to none in design.

However it performs like a dud. No scheduling to speak of, just runs everything as soon as it can. We've run into nodes dying under load (-not- underprovisioned, could run all these jobs manually at once on these monsters). We've run into problems with volume reaping, fork bombs, ui freezes, everything under the sun.

I really like Concourse and will hopefully one day be able to come back to it when its implementation is as solid as its paradigms are.

But I'd avoid use for now.

An incredibly rude post in reply to something I think is reasonable. My opinion is as I stated, NOT as you very loosely interpreted thank you very much. I don't think of myself of a gatekeeper of others, just my own things.

I have a very high bar for open source personally and I tend to view it "more hardcore". I think that if you're putting open source out there to a wide audience such as this that you have a contract with the people you're putting it out to that you're going to maintain what's there and ensure it's of high quality. And if you're going to go to that level of effort why not contribute to something that's already mountains of work ahead of you and effectively the same code anyway (upperdb/gorp/gorm/xorm in this case)?

I have many Go repositories on my Github account that I've used for learning - but I've never put any of them out there to wide audiences that were solved by a more popular project in the same way - the approach has always been radically different. When I couldn't find logging libraries I liked I tried using others and contributed to them. I've also never put anything out there that I don't intend on maintaining basically forever. So my behavior is consistent with my viewpoint for whatever that's worth.

My colleague suggests that other people use open source for different reasons and view it much differently than me, and that's possibly the misunderstanding here. I shouldn't try to reflect my opinion in that sense on others and I do apologize for that. I suppose "if it were me" I wouldn't have done what he has, and I was commenting as if people knew and understood that as a commonality - which isn't probably true.

Also - I think you hugely overstated my bias for my own library. I'd be just as happy if he had not posted this and worked on gorp instead. /shrug.

Also I've been waiting for a long time for Go's libraries to be converged on - hasn't happened yet :)

I want to see the Go community and it's libraries flourish. But having libraries like this around which provide extra confusing and probably poorly maintained choices for users is detrimental to it's overall health. There's already great examples of maintenance rot found even amongst the very popular libraries like go-yaml and httprouter.

One thing you can say about Go is it has 1000 Micro-frameworks, ORMs, Routers, and Logging libraries. Most of them are within a percent of giving the exact same feature set with the exact same approach and and ever so slightly different API. Lots of them share the same mistakes as they are a user's first project or a learning arena (though routing has been a pretty benchmarked area so that's been better than the others).

As someone who has open source libraries, and who uses Go in their day job and at home my personal opinion is that Go needs fewer, stronger libraries. Well supported, well maintained pieces rather than this illusion of choice that's presented today. I'm sorry if I came off as crass in my reply to the OP but that's simply the state I see in the community today.

Ideally there'd be 2-4 ORMs in Go that would be the goto. And half would use a code-generation approach, half would use the reflection based approach.

https://github.com/astaxie/beego/tree/master/orm https://github.com/cosiner/gomodel https://github.com/jinzhu/gorm https://github.com/go-gorp/gorp https://github.com/coocood/qbs https://github.com/go-reform/reform https://github.com/vattle/sqlboiler https://github.com/upper/db https://github.com/astaxie/beedb https://github.com/naoina/genmai https://github.com/monochromegane/argen https://github.com/square/squalor https://github.com/theduke/go-dukedb https://github.com/dropbox/godropbox/tree/master/database/sq... (...and more) And now: https://github.com/samonzeweb/godb

Do we truly need this? This one in particular feels a lot like upperdb. And if it's for learning only, don't publish it to things like hackers news which makes people seem like it's something you should use as opposed to the more mature libraries out there. This is what I meant by fragmentation, it's an illusion of choice. There are actually very few logging/routing/orm/micro-framework libraries you should really use if you're serious about a project.

Well, now that you're done learning I hope you'll become an sqlboiler user and maybe contribute to it to fit your use case too. Fragmentation in the sql world in Go is pretty bad right now, hence my reaction to seeing your lib, sorry for that :)
I'm the author of a code-generation ORM in Go (real orm, does relationships fairly robustly) https://github.com/vattle/sqlboiler and I'm curious why you needed to make another one despite something like this existing? Granted we don't support sqlite/mssql yet, but surely a PR to support it is easier than writing your own :(