alexrustic

Jinbase is designed to be thread-safe, ensuring it can be reliably used in a multithreaded context.

To interact with SQLite, Jinbase uses LiteDBC [1] an SQL interface compliant with the DB-API 2.0 specification described by PEP 249 [2], itself wrapping Python's sqlite3 [3] module for a more intuitive interface and multithreading support by default. I wrote for LiteDBC a stress test [4] involving concurrency with Asyncpal [5].

[1] https://github.com/pyrustic/litedbc

[2] https://peps.python.org/pep-0249/

[3] https://docs.python.org/3/library/sqlite3.html

[4] https://github.com/pyrustic/litedbc/blob/master/tests/test_s...

[5] https://news.ycombinator.com/item?id=41404020

Thank you for your comment ! Indeed, for wide adoption across languages, we will need to port at least Paradict as that is the format in which BLOBs are serialized.

Protobuf relies heavily on predefined schemas and this rigidity goes against the flexibility of Jinbase's schema-less philosophy.

MessagePack (or CBOR) seems more convincing but Paradict has some subtleties that I don't find there. For example, Paradict preserves UTC offsets [1], handles integer bases, allowing for the representation of integers in decimal, binary, octal, and hexadecimal formats, has an extension mechanism that I find more interesting, etc.

Soon I will be adding a command line interface to Jinbase. From the CLI one will be able to read and write any type of data and this will only be possible because Paradict has a twin text format. MessagePack, from what I know, started with 1:1 compatibility with JSON and then over time it added things that are not present in JSON, thus breaking the 1:1 compatibility with JSON.

If I understand Peter Naur's take on programming [2] correctly, Jinbase is a software idea that I'm trying to implement (bring to life) one iteration at a time, and for that I need to have some level of control over the components (like the serialization format) so that I can adjust things accordingly. For example, the Paradict binary format is originally intended to serialize and deserialize only dictionaries (P...dict), but I changed that detail so that Jinbase users can freely store other things than dictionaries.

Once the core idea is fully implemented, we will see how to reproduce it elsewhere, one contribution/compilation after another...

[1] https://codeblog.jonskeet.uk/2019/03/27/storing-utc-is-not-a...

[2] https://news.ycombinator.com/item?id=26027448

If I understand correctly, goroutines = asyncio (non-invasive) + asyncio.to_thread (when needed, but automatic). So, it is still cooperative concurrency that feels preemptive due to its design. This Go capability is interesting, and it 'seems' that it cannot be replicated without some integration with the runtime of the target language, i.e., a 'baked-in solution'. For now, I'm fine with true preemptive concurrency, but this might change in the future.

Thank you !

Thanks for your comment !

When you are doing parallelism, don't forget to protect the 'entry point' of the program by using "if __name__ == '__main__'", and also avoid the __main__.py file [1]

  # this file isn't `__main__.py` !
  from asyncpal import ProcessPool

  def square(x):
      return x**2

  if __name__ == "__main__":  # very important !
      with ProcessPool(4) as pool:
          numbers = range(1000)
          # note that 'map_all' isn't lazy
          iterator = pool.map(square, numbers)  # map is lazy
          result = tuple(iterator)
          assert result == tuple(map(square, numbers))

[1] https://discuss.python.org/t/why-does-multiprocessing-not-wo...

If I'm given a chance to develop, I would start with the last part of the sentence that might sound subjective. For example, the 'Future' class in 'concurrent.futures' exposes a method named 'result' to collect the result of a task. In contrast, the 'Future' class in Asyncpal exposes a 'collect' method and a 'result' property.

The stdlib's pools only grow in size and do not shrink, making them non-elastic. Therefore, I would not want to keep them alive in the background for sporadic workloads [1].

Discussions on 'concurrent.futures' vs 'multiprocessing.pool.Pool' highlight that each has unique features. While 'concurrent.futures' is the modern package, it omits some niceties found in 'multiprocessing.pool.Pool'. For example, 'concurrent.futures' has only one 'map' [2] method, which works eagerly and therefore not suitable for very long iterables [3][4]. However, I acknowledge the superiority of 'concurrent.futures.Future' [5] over 'multiprocessing.pool.AsyncResult' [6] because tasks cannot be cancelled with the latter (among other things).

[1] https://www.cloudcomputingpatterns.org/unpredictable_workloa... (related)

[2] https://docs.python.org/3/library/concurrent.futures.html#co...

[3] https://docs.python.org/3/library/multiprocessing.html#multi...

[4] https://docs.python.org/3/library/multiprocessing.html#multi...

[5] https://docs.python.org/3/library/concurrent.futures.html#fu...

[6] https://docs.python.org/3/library/multiprocessing.html#multi...