> In most other languages you can get away with just running tasks in the background for a really long time before you need spin up a distributed task queue

In Python too:

    # or ThreadPoolExecutor depending of the type of work
    from concurrent.futures import ProcessPoolExecutor, as_completed

    with ProcessPoolExecutor(max_workers=2) as executor:

        a = executor.submit(any_function)
        b = executor.submit(any_function)

        for future in as_completed((a, b)):
            print(future.result())

Any task you would put in celery would be a good candidate for being first passed to a process pool executor.

No, the real reason we use celery is that it solves many problems at once:

  - it features configurable task queues with priorities
 
  - it comes as an independent service accessible from multiple processes

  - its activity can be inspected and monitored with stuff like flower

  - it has the ability to create persistent queues that survive restart

  - error handling is backed in, they are logged and won't break your process
 
  - it offers an optional result backend for persisting results and errors
 
  - many distribution and error handling strategies can be configured

  - tasks are composable, can depend on each others or be grouped

  - celery also does recurring tasks, and better than cron

  - you can start tasks from other languages than Python

Now some People use celery when they should use a ProcessPool, because they don't know it exists. But that's hardly because of the language: you didn't seem to know about it either.

It is true that Python concurrency story is not comparable to something like Go, Erlang or Rust, but very common use cases are solved.

In the same way, we can perfectly use shelves instead of redis.

We use redis beacuse it solves many problems at once:

    - it's very performant on a single machine, and can be load balanced if needed

    - it has expiration baked in

    - it offers many powerful data structures 

    - it embeds numerous strategies to deal with persistence and resilience

    - it's accessible from multiple processes, and multiple languages

    - its ecosystem is great, and it's hugely versatile

It's almost never a bad choice to add redis for your website. It's easy to setup, cheap to run, and and it shines to manage sessions and caching for any service with up to a few million users a day.

Once you are at it, why not later use it for other stuff, like storing the result of background tasks, log stream, geographical pin pointing, hyperlolog stats, etc. ? There are so many things it does better than your DBMS, faster, easier or for less resources.

It's such fantastic software really.

But no, nothing prevents you in Python to create a queue manually and serialize it manually. It's just more work for less features.