Pre-commit is optional, you can just not install them into .git/ ... Although I'd indeed prefer just before push, or just make display warnings

One thing that's really annoying these days are CI/CD that can't be replicated locally, generating quite annoying delays in the development. Jenkins seems particularly problematic in this regard: the steps get encoded in some cryptic pet Jenkins server, and then you have to wait minutes until an agent picks it up and reaches the step you actually care about. Other tools are a little quicker, but still...

So, I think at the very least pre-commit hooks help with this "over-reliance" on the CI/CD server. It's so much better DX when you can run parts of the pipeline instantaneously.

> One thing that's really annoying these days are CI/CD that can't be replicated locally

pre-commit just runs your linter, formatter and tests. Surely you can fully replicate this step locally. Just run make lint both locally and on CI.

Anything else that's hard to replicate has to do with the distributed systems that you are probably working on because these systems are all probably proprietary stuff that live on the cloud.

> Although I'd indeed prefer just before push, or just make display warnings

Wholehearted agree. This is a compromise I can live with.

My point is that using these tools at least makes people put these scripts within the repo instead of just in the CI server.

There's no "instead of" to speak of. Either you run the same linting step both locally and on CI, or just on CI, you can't skip linting on CI. Every properly set up CI pipeline has some combination of Makefile/shell script/Dockerfile that runs exactly the same way locally and on CI, which the script checked into the repo itself. If your CI scripts don't exist in your repo, you are doing it wrong.

Our CI pipelines invoke pre-commit. That way it's trivial to run the exact same tools locally as would be run in CI.

Running the tools locally is basically about tightening the development loop. Many of the commonly used tools (e.g. black, isort etc.) actually make the changes to the files so you'll never even commit failing versions. Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check? There's nothing at all stopping you from doing that. Pre-commit is completely optional for each developer. I would just recommend it for sanity reasons.

> Do you really want to push changes to some remote CI system only to be told it's failed some boring QA check?

Yes, that's what CI/CD is for.

> Pre-commit is completely optional for each developer.

No it is not, it's installed at your git pre-commit hook, and it gets run every fucking time I commit, even if I've set up my editor correctly to auto-format and lint everything at I develop, meaning 99.9999% of the time sure the code I commit will pass all these linter checks. I can use git commit --no-verify to bypass pre-commit, but then again, what's the point of using pre-commit in the first place if you need to bypass it? There is absolutely no point of linting twice locally and thrice in total just to hit the first stage of your deployment.

  rm ./.git/hooks/pre-commit

Or just do not install the hook in the first place. Choice is still the developer's unless your team is doing something else here that limits your ability to delete/rename files locally.

That does not alleviate the problem with difficulty having your editor to use the correct version and settings for the linters and formatters. This approach also does not address the issue of having 2 separate sets of configuration when you are trying address problem number 1 above.

> Yes, that's what CI/CD is for.

Not really. CI/CD is a methodology where a team immediately integrates new changes into a trunk/release. The CI/CD pipeline is there to give the team the confidence to merge your stuff. When I see people constantly pushing breaking code into a CI pipeline I see an incredible amount of wasted time and shared computing resource. Especially if it's some trivial formatting check that you should have already done locally. You do what works for you, but tools like pre-commit were invented to save time and effort and they work well.

> I've set up my editor correctly to auto-format and lint everything at I develop, meaning 99.9999% of the time sure the code I commit will pass all these linter checks.

Then what is the problem? Have your team installed hooks that take a long time to run? Even on larger codebases pre-commit adds a negligible amount of time to each commit, unless perhaps you've touched every file in the codebase or something. Honestly your gripe with pre-commit seems mostly irrational.

> The CI/CD pipeline is there to give the team the confidence to merge your stuff.

That's why linting is a part of every CI stage. Linters check your code for bugs.

> Have your team installed hooks that take a long time to run?

Yes, they are called tests.

Wait, you're not supposed to run tests with pre-commit. I can see why that would be frustrating. If that's the case, your team is doing it wrong.

Perhaps, but it's not just one team, and if you are doing code quality checks on CI anyway, I don't see the point of pre-commit (for most people).

How is it mandatory? You can just not install the hooks

1. Because it defeats the purpose of pre-commit.

2. pre-commit install installs the hook by default.

3. If you happen to fail linting a few times a year, there's always an anal coworker telling your boss in a 1-on-1 you are not following some BS ways of working.

> 1. Because it defeats the purpose of pre-commit.

No, it doesn't. Pre-commits are just a DX bonus so you won't have to wait for the CI/CD server.

2. pre-commit install installs the hook by default.

Am I missing something? If you run "$ pre-commit install" of course it will install the hooks. Do you mean that poetry install, setup.py themselves are making the magic of installing the hooks themselves? If so, it's indeed a bad practice.

3. If you happen to fail linting a few times a year, there's always an anal coworker telling your boss in a 1-on-1 you are not following some BS ways of working.

I don't get why anyone would care, unless you're polluting the CI/CD history with tons of tiny commits all day along just to run linting.

Filling the CI/CD pipeline with tiny commits to fix linting when it should be done before commit or push is why pre-commit exists. It's not BS, clogging the job queue shits in everyone else.

Of course I don't know GPs specific context so they may have other details, but generally it's

Except this problem rarely comes up. Everyone has their favorite way to set up their editors to lint and format automatically using the project's config.

Over my career, I see that every company has some version of:

1. New guy joins a team, wants to push code ASAP

2. He sets up his favorite editor and punt configuring linters and formatters

3. Code pushed, CI fails at linting

4. Some well-meaning coworker or new guy suggests some variations of husky/pre-commit/fancy git hook scripts

5. Team agrees and put that into the repo

6. 6 months to 1 year later the entire team realizes its benefit does not out-weight the cost, and unanimously agree to rip it out unceremoniously.

Well I have different experiences where no one would ever go back to living without pre-commit, black, autoflake, etc. It moves the fixes to before they get to CI. There's literally no downside.

Maybe you have experience with some weird tool? Or running black and autoflake messed around with emacs and it needs to reload all the buffers?

I initially had some problems, then I've written an emacs package[1] to fix all of them. And it's due to the course of writing this package made me realize just how bad pre-commit is from its UX, design to its entire premise.

[1]: https://github.com/wyuenho/emacs-python-exec-find

Ok then I think the issue is not so much pre-commit but that python tooling in general should not be written in python. It's a minority opinion but the fact that there is a tooling python and an application python can mess with things as you're experiencing.

Yeah, I agree. It's just that they can just run pre-commit manually if the hooks bother them that much

Tell me you use Emacs without telling me you use Emacs.

Curious, how does using Emacs affect that much the workflow to the point of pre-commit being so annoying?

Not sure why, though? I've used emacs for 15 years and pre-commit for probably 5 years at this point.

Are you using git inside Emacs or from the command line? It seems GP is doing everything in Emacs and application python resolution is tripping up with tooling python.

If you have it all sorted out from Emacs maybe check the repo they posted which tries to do all this.

Guilty

Does it matter if it’s run twice? Usually it’s so fast you hardly even notice it.

If your pre-commit takes more than 3 seconds to run it’s set up incorrectly IMO, and should belong to a more manually (and CI of course) invoked test suite instead.

Pre-commit running tools in it's own virtual environment is a feature, not a bug, in my book -- it means that the dependencies for my linter tools aren't mixed in with the dependencies for the code I'm writing.

And, keeping things separate from setup.cfg or pyproject.toml is optional: The tools still look for configuration in their usual places, so it's still possible have your black options in pyproject.toml and just a bare-bones entry to call black in your .pre-commit file if you prefer.

Except your usual configuration don't necessarily work for your pre-commit hooks. A prime example is mypy.

Sure it can. The pre-commit tool is just a framework for running multiple hooks at commit time against just the files which are modified in that commit. You can configure those hooks however you want.

You also don't have to run the hooks at pre-commit time. Just don't hook pre-commit into your checkout. The pre-commit tool can also be configured to run its checks at a different stages than, well, the pre-commit stage:

https://pre-commit.com/#top_level-default_stages

Do you have any tutorials for setting up CI/CD? My impression was that's all stuff that runs in the cloud but if it's something I can use on my own personal projects I'd play with it. Frankly a lot of these things become unintelligible. I've used pre-commit for things like black and autopep8 and that's all pretty understandable to me. The CI/CD things I've read all seem like everyone already understands some giant Rube Goldberg contraption that they're strapping on things for some reason that probably matters to giant dev teams.

Good example of CI/CD that's not a rube goldberg machine per se:

https://github.blog/2022-02-02-build-ci-cd-pipeline-github-a...

Thank you. So does CI/CD require becoming dependent on a cloud service such as github?

Not necessarily, it just depends on how invested you are in the CI/CD pipeline for any given project, your preferences regarding self-hosting vs. cloud, and the amount of time you have to dedicate to the subject.

Strictly speaking, any tool or set of tools that allow you to trigger building & deploying/publishing artifacts in response to source control commits can be used to build a CI/CD pipeline. One could write bash scripts linked to a cron job that pulls a remote repository every n minutes and then performs some scripted actions to integrate changes between branches before building & publishing the artifact to a local SFTP server.

If you prefer a more mature solution with better documentation however, there is a (non-exhaustive) list of CI/CD tools on this awesome-devops list:

https://github.com/wmariuss/awesome-devops#continuous-integr...

---

edit: I saw gitlab on the list and realized it is probably the closest self-hosted equivalent to the github option I mentioned previously

https://about.gitlab.com/features/continuous-integration/

It's convenient to run the lint step faster/sooner than at CI/CD time. Depending on your setup, the separate linter deps handled by pre-commit can be more convenient than hassle both locally and in your CI/CD pipeline (re the makefile script you mention).

Having done it both ways several times I lean pre-commit for now

You've already run them at edit time.

Pre-commit can also be configured to use the project's dependencies, which is what I do with my repos when there is overlap. You don't have to use its built-in integrations. Indeed, I find its integrations most useful for tools that aren't specific to my project: things like the white-space checks, yaml checks, etc.

You can also set things up the other way around, having the CI/CD system install and and run pre-commit's checks as a build step. Pre-commit provides a nice framework, I find, for running these checks.

The advantage of pre-commit is that it catches mistakes before they are committed. Most CI/CD systems are setup to only validate the tip of the branch (i.e. the last commit in the PR), not all the commits along the way. Yes, you can configure a CI/CD system to test each commit, but it's usually swimming upstream to do so. And yes, you can squash all of a PR's commits into a single commit, but there are good reasons NOT to do that. So assuming you have multiple commits in a change, it's nice to know they have likely all been validated in a project using pre-commmit.

I'll make an appeal to authority here:

I've been a professional developer for decades. I've worked with a variety of VCS's and build systems. I've written plenty of Makefiles. Nonetheless, I still find pre-commit useful. I use it even in combination with a Makefile sometimes. You'd be horrified, I guess, to know some of my Makefiles have a rule which runs `pre-commit run --all`.

As an example, earlier this week I setup a new repo which build packages an AWS Lambda written in javascript, and deploys it using the AWS "sam" CLI via a CI/CD system. So the deployed code is Javascript. The repo contains shell-scripts to assist with deployment. And there are multiple yaml files. There's a yaml file to configure the CI/CD system, and there's the CloudFormation template files.

Here's what I configured pre-commit to do:

1. Check for whitespace nits.

2. Check for syntax errors in the yaml files.

3. Validate the CloudFormation templates.

https://aws.amazon.com/blogs/infrastructure-and-automation/u...

4. Run shellcheck against the shell scripts.

5. Run eslint and prettifier against the javascript.

6. Run "npm test" as a local step.

Normally I'd leave (6) out because in most projects its too time-consuming, but in this project the tests run quickly enough that I just made it a pre-commit check. The "npm test" step runs jest, which is installed as a dev dependency in the project's package.json. The other tools are all installed by pre-commit itself.

> I really don't see the point of any pre-commit hooks unless you are the one guy that doesn't use a modern CI/CD platform.

I've tried to make an argument for why to use pre-commit above. Nevertheless, you don't have to install pre-commit's hooks in your checkout. They are ideally there to save you time having to correct mistakes after the fact. It sounds like there's an impedance mismatch between your personal workflow and how you've seen pre-commit set up. Perhaps by resolving that mismatch by adjusting the pre-commit configuration, you can enjoy pre-commits benefits w/o experiencing the issues you've run into.

As another dev with decades of experience, I've got to ask, why would you "validate" every commit in a PR lol. The reason every CI pipeline is setup to only validate the tip is because your combined outgoing changes that is about to be merged is all that matters. Nobody cares if you have an experimental branch full of commits failing lint and tests as long as the PR doesn't.

However, in my experience, I've seen plenty of pipelines setup to run lint on every push on a PR branch, which is effectively only checking outgoing changes before merge, it's just in this case it's merging to your feature branch. My point still stands - as long as linting is done on CI, and you've set up your editor to lint as you edit, you don't need pre-commit.

I'm not entirely sure what point you are trying to make. It sounds like all you've done is moved all of the tools you'd call anyway from a Makefile to a YAML file.

Here's an example from one of my repos that mostly uses the repo's own installed dependencies so that I don't need to manage those in more than one place:

https://pastebin.com/raw/0SP3NvBS