rogerbinns

Your feedback is appreciated. I am busy working on a release which should be available later today. Your request has now been addressed and is part of the release. Thanks for trying APSW.

Thank you for the kind words. I was especially delighted when noticing that cysqlite also supports table valued functions in Python. I'm hoping they become a standard thing for Python developers using SQLite, because it makes bridging SQLite and Python even more productive.

It has been going since 2004. There is going to be a new release today. What year do you suggest :)

(I am the APSW author)

I originally started out doing exactly what you said - dumping Python dictionaries into a file and reading/writing that. It worked well. Where it really hurt was I wanted to add undo functionality, and that means needing to store the before and after view of the data. And I wanted it more sophisticated than that - be able to view what the data looked like at any point in time.

That forced me to use a database, because now I could use triggers to automatically save off old values, and views to use them nicely.

There were many Python SQLite wrappers at the time, but I didn't like them. So I wrote another one :)

https://www.rogerbinns.com/blog/20-years-of-apsw.html

(I am the APSW maintainer)

If you ask via the contacts such as Github discussions or email then I can help find a perfect solution for your needs - https://rogerbinns.github.io/apsw/about.html#mailing-lists-c...

As you found out this happens because the cursor is immediately exhausted as there are no rows, and then is cleaned up and so column data names are not available.

You can use apsw.ext.query_info to get details about a query without running it - see https://rogerbinns.github.io/apsw/ext.html#apsw.ext.query_in... which includes an example.

Behind the scenes you can use an execution tracer to extract the information. That is what query_info does. although it aborts execution after recording it.

(I am the APSW maintainer)

The Linux and BSD flavours have APSW - see https://rogerbinns.github.io/apsw/install.html#linux-bsd-pro...

There is even a website that tracks around 150 different Linux and BSD flavours (even termux) and shows package name, version, etc for each it is present on - see https://repology.org/project/python:apsw/versions

(I am the APSW maintainer)

That is documented behaviour - think of it as making a best effort, and not losing the value.

As of January 2006 you could add CHECK constraints using the TYPEOF function to reject that at the SQL level. And it is your own code - there is no server - doing the insertions. As was common back then, protecting you from your own bugs was not a high priority for APIs!

SQLite was originally started as a local database library for use during development for times when the main networked database was not available. It used dbm as the underlying storage mechanism, with the dbm API roughly being string keys with string values. ie all underlying values were actually stored as strings. The SQLite code would automatically do conversions - eg the plus operator would convert the strings to int or float, add them, and generate a stringified number as a result. The vast majority of implementation code did not have to care about types, and very local decisions could be made such as in the addition example.

TCL was used as a dev wrapper language at the time, and it functioned the same way.

It was only in mid-2004 that SQLite 3 was released which used its own storage backend, and that allowed for the 5 supported storage types (int64, string, bytes, float, null). It was API compatible (with minor adjustments) with the earlier SQLite 2, so the lack of static typing continued, otherwise everyone would have to rewrite their code. You do get dynamic typing, which hasn't been a problem for the vast majority of SQLite users.

Do remember that SQLite is competition for fopen, not Oracle / Postgres etc. It is trying to make things as effective as possible in that scenario. If you don't want numbers in your string column, then don't do that!

SQLite has code to work around anti-virus scanning, by doing retries while the anti-virus tool is processing an operation. I've not seen anti-virus tools result in corruption on the SQLite forum.

Check out wastrel https://codeberg.org/andywingo/wastrel

Of note WASM 3 has garbage collection (GC), multi-memory, exception handling, tail calls and more which can be challenging to implement.

> ... why it was simpler.

In the early 90s I implemented a gateway between Novell email and X.400. What amused me the most was X.400 specified an exclusive enumerated list of reasons why email couldn't be delivered, including "recipient is dead". At the X.400 protocol level this was a binary number. SMTP uses a 3 digit number for general category, followed by a free form line of text. Many other Internet standards including HTTP use the same pattern.

It was already obvious at the time that the X.400 field was insufficient, yet also impractical for mail administrators to ensure was complete and correct.

That was the underlying problem with the X.400 and similar where they covered everything in advance as part of the spec, while Internet standards were more pragmatic.

That is something you can do in cahoots with a regular cashier and the reason places like Costco check your receipt. The cashier just has to fake scan an item, and nobody would notice. Receipt checking makes it possible to get caught.

Related, there is also sqlite3_rsync that lets you copy a live database to another (optionally) live database, where either can be on the network, accessed via ssh. A snapshot of the origin is used so writes can continue happening while the sqlite3_rsync is running. Only the differences are copied. The documentation is thorough:

https://sqlite.org/rsync.html

I remembered Weitek as making math co-processors but it turns out they did an 80287 equivalent, and nobody appears to have done an 8087 equivalent. Wikipedia claims the later co-processors used I/O so this complicated monitoring the bus approach seems to have only been used by one generation of architecture.

I was one of those students saving up the large sum for the book, when Linux was announced. There were other tensions at the time - the biggest was that Minix on 8086 was 16 bit real mode only. Someone had developed patches to run in 32 bit protected mode, but they were invasive and large, and the Minix maintainers would not integrate them as the increased complexity would not help the mission of Minix being easy to learn and tinker with. The filesystem code was also single threaded, essentially doing one request at a time. IIRC there were patches to address that too, also not integrated for the same reason. (Note that the books included print outs of the source so keeping it short did matter.)

This explains the final 2 sentences of the original Linux announcement:

> PS. Yes - it's free of any minix code, and it has a multi-threaded fs. It is NOT portable (uses 386 task switching etc), and it probably never will support anything other than AT-harddisks, as that's all I have :-(.

The book publisher is blamed for preventing Minix from being freely distributed: https://en.wikipedia.org/wiki/Minix#Licensing

Do you know what other prior systems did for co-processor instructions? The 8086 and 8087 must have been designed together for this approach to work, so presumably there is a reason they didn't choose what other systems did.

It is notable that ARM designed explicit co-processor instructions, allowing for 16 co-processors. They must have taken the 8086/8087 approach into account when doing that.

Did it make things simpler or more complex for the byte order they picked? It is notable that new RISC designs not much later all started big endian, implying that is simpler. Can you even tell the endianess from the dies?

Pelican is what I use, and it works well.

I used to use Nikola, but gave up on that for two reasons. One was it was adding every possible feature which meant an ever growing number of dependencies which gets scary. And that is hard to use - eg how do you embed a youtube video becomes a trawl through documentation, plugins, arbitrary new syntax etc.

But the biggest problem and one that can affect all the SSG is that they try to do incremental builds by default. Nikola of the time was especially bad at that - it didn't realise some changes mattered such as changes to config files, themes, templates etc, and was woolly on timestamps of source files versus output files.

This meant it committed the cardinal sin: clean builds produce different output from incremental builds

Pelican has kept it simple.

In 1994 at the second WWW conference we presented "An API to Mosaic". It was TCL embedded inside the (only![1]) browser at the time - Mosaic. The functionality available was substantially similar to what Javascript ended up providing. We used it in our products especially for integrating help and preferences - for example HTML text could be describing color settings, you could click on one, select a colour from the chooser and the page and setting in our products would immediately update. In another demo we were able to print multiple pages of content from the start page, and got a standing ovation! There is an alternate universe where TCL could have become the browser language.

For those not familiar with TCL, the C API is flavoured like main. Callbacks take a list of strings argv style and an argc count. TCL is stringly typed which sounds bad, but the data comes from strings in the HTML and script blocks, and the page HTML is also text, so it fits nicely and the C callbacks are easy to write.

[1] Mosaic Netscape 0.9 was released the week before

Another excellent GUI is gitg. You can select specific lines for staging, but also for discarding. The latter is especially useful for temporary debug only changes that you want to throw away.

Heap allocation does also allow other things. For example stack frames (PyFrame) are also heap allocated. When there is an exception, the C stack gets unwound but the heap allocated frames are retained forming the traceback. You can then examine the values of the local variables at each level of the traceback.

And it also allows async functions, since state is held off the C stack, so frames can be easily switched when returning to the event loop.

The other thing made easy is C extension authoring. You compile CPython without free lists and an address sanitizer, and getting reference counting wrong shows up.

You aren't forced to use a GIL as I keep stating. You can set an environment variable or a command line flag to Python and the GIL will remain disabled. My package will work just fine if you do that, unless you provide it with data you concurrently modify in which case you can get corruption and crashes.

I provide 3 options:

1) Use regular GIL Python and you get the highest levels of integrity and correctness of operation of my package

2) Use a free threaded Python, the GIL will be enabled at load time, and you get the highest levels of integrity and correctness

3) Use a free threaded Python, and set $PYTHON_GIL=0 or start with -Xgil=0 to keep the GIL disabled, and providing you do not do concurrent mutation of data provided to my package, you get the highest levels of integrity and correctness

BTW I did not randomly choose to provide the free threaded builds. I specifically asked the setuptools maintainers (under the Python Packaging Authority) how to prevent free threaded builds for PyPI. They encouraged me to do the free threaded builds so that a user doesn't to have maintain parallel regular and free threaded Python installations. And it allows option 3 above.

What is better:

1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes

2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment variable PYTHON_GIL=0 or start Python with -Xgil=0 which will keep the GIL disabled, and they will be fine if they avoid concurrent mutation.

I chose 2. The stdlib json package (along with many others) picked 1. Heck I'll guarantee that most that picked 1 aren't 100% safe either, because doing the changes is hard work, *every* case has to be covered, and tools like thread sanitizers don't work.

The reason I chose 2 is because I care about data integrity. I will eventually reach 1, but only once I can be certain the code is correct.

Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded.

Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.

Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.

But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.

As I stated:

> so that you don't have to keep separate GIL full and free threaded interpreters around

It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.

It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.

Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter.

This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other.

Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated.

You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311

It is multiple fine grained locking versus a single global lock. The latter lets you do less locking, but only have a single thread of execution at a time. The former requires more locking but allows multiple concurrent threads of execution. There is no free lunch. But hardware has become parallel so something has to be done to take advantage of that. The default Python remains the GIL version.

The locking is all about reading and writing Python objects. It is not applicable to outside things like external libraries. Python objects are implemented in C code, but Python users do not need to know or care about that.

As a Python user you cannot corrupt or crash things by code you write no matter how hard you try with mutation and concurrency. The locking ensures that. Another way of looking at Python is that it is a friendly syntax for calling code written in C, and that is why people use it - the C code can be where all the performance is, while retaining the ergonomic access.

C code has to opt in to free threading - see my response to this comment

https://news.ycombinator.com/item?id=45706331

It is true that more fine grained locking can end up being done than is strictly necessary, but user's code is loaded at runtime, so you don't know in advance what could be omitted. And this is the beginning of the project - things will get better.

Aside: Yes you can use ctypes to crash things, other compiled languages can be used, concurrency is hard

Some examples of what it could do when using the C Python APIs:

* Point out using APIs that return borrowed references

* Suggest assertions that critical sections are held when operating on objects

* Suggest alternate APIs

* Recognise code patterns that are similar to those done during the stdlib auditing work

The compiler thread sanitizers didn't work the last time I checked - so get them working.

Edit: A good example of what can be done is Coccinelle used in the Linux kernel which can detect problematic code (locking is way more complex!) as well as apply source transformations. https://www.kernel.org/doc/html/v6.17/dev-tools/coccinelle.h...

The core Python data structures are atomic to Python developers. eg there is no way you can corrupt a list or dictionary no matter how much concurrency you try to use. This was traditionally done under the protection of the global interpreter lock which ensured that only one piece of C code at a time was operating with the internals of those objects. C code can also release the GIL eg during I/O, or operations in other libraries that aren't interacting with Python objects, allowing concurrency.

The free threaded implementation adds what amounts to individual object locks at the C level (critical sections). This still means developers writing Python code can do whatever they want, and they will not experience corruption or crashes. The base objects have all been updated.

Python is popular because of many extensions written in C, including many in the standard library. Every single piece of that code must be updated to operate correctly in free threaded mode. That is a lot of work and is still in progress in the standard library. But in order to make the free threaded interpreter useful at this point, some have been marked as free thread safe, when that is not the case.