News for hackers. Stuff that matters.

edsac_xyzw

Re:Attempts to make Python fast on Saturday October 24, 2020 @01:40PM (#24880713)
This duct type is the Python native extension api (not python ctypes) which allows creating native code modules (aka libraries) in C or C++ and creating wrappers to existing C or C++ libraries. This escape hatch that enables offloading cpu-intensive computations to high performance libraries written in C, C++ or Fortran. Another benefit of python modules written in C or C++ is that they are not affected by the GIL (Global Interpreter Lock) problem, thus they can take advantage of multi-core and SIMD instructions and achieve higher performance.
Hardly it will fully replace C++ due to the pervasiveness of the language on many domains and also due to the C++ ecosystem that encompasses system programming; embedded systems; games and so on. For instance, the C++ ecosystem still outmatches rust regarding high performance libraries for several scientific computing domains; GUI graphical user interface libraries, such as Qt, MFC, wxWidget; Game engines and so on. In addition, Rust lacks openMP feature which allows parallelizing for-loops, which are widespread on scientific computing, for taking advantage of multi-core CPUs and from SIMD.
Re:The Accelerating Adoption of Julia on Wednesday October 21, 2020 @04:21PM (#24851702)
That is not too hard if one uses Pybind11 C++ library, which allows creating native code Python modules (shared libraries) in a simple and declarative manner. Other library that can be used with Pybind11 is Eigen for linear algebra that can perform loop fusion in numerical computation, for instance, it can add multiple vectors of same size in a single loop.
Re:The Accelerating Adoption of Julia on Wednesday October 21, 2020 @04:01PM (#24851466)
Matlab and its Simulink extension, which is widely used for designing control system has a feature for generating C code that can be deployed without Matlab runtime. Someone with knowledge of code generation and compilers could take this Matlab feature as an inspiration and implement a C code generator for Julia functions.

> Right now, it appears like a niche interpreted language for numeric geeks only.

Yes, it was designed for numerical and scientific computing. In Julia, you can at least prototype the numerical algorithm before implementing in C++ or Fortran.

Unlike Rust and Golang. C++ has long history and baggage, thus there is no standard tooling for C++ such as building systems and package managers. In the past, every operating system had its own set of preferred tooling for C and C++, on Linux and Unix-based system, lots of projects still use GNU autotools or Makefiles; on Windows, many projects still use the XML-based MSBUild from Visual Studio IDE. Many IDEs also have their own XML based building systems as well. Nowadays, CMake major IDEs support CMake and allows using CMakeLists.txt as a project file by just opening the directory containing this file. Some of those IDEs are Visual Studio IDE, Visual Studio Code, CLion, Eclipse (Via plugin), KDevelop and so on.

Regarding libraries dependencies, CMake has a FetchContent feature which allows downloading source code directly from http, ftp or git servers and adding this code as project dependency. This approach can avoid wasting time installing and configuring libraries. However, this technique is only feasible for lightweight libraries when the compile-time and library object-code size are not significant. For large libraries and frameworks where the compile-time is significant, such as Boost Libraries, Gtk or Qt, Conan or Vcpkg are the best solutions as they can cache and reuse library object-code with many other projects which reduces the compile-time and disk space usage.

C++ is not best solution for Web applications, in most cases you will not gain anything using C++ for this case, unless the web application is running in an embedded system such as router or a network printer. C++ is most suitable for cases where you need: high performance; access the operating system low level features; implement an operating system; implement an embedded system or a triple-A game.

If you use QEMU plus KVM (Kernel Virtual Machine), you can say goodbye to dual boot as the virtual machine is run with near native performance. For playing windows-only games, the solution is GPU passthrough, which allows allocating a GPU for the virtual machine providing the maximum gaming performance.
Re:Implementing UFCS for C++ in Clang on Monday August 03, 2020 @03:20PM (#24042126)
C++ could adopt the UFCS universal-function-call feature of D-language that allows any ordinary free function to be called as if they were methods, without any special kind of type annotation. One of major advantages of UFCS is that it enhances the discoverability of free functions in IDEs. An user could write 'Object' followed by dot '.' and the IDE could show all possible functions applicable to that type. In D-language, the UFCS also works for primitive types such as int, double and so on.

D-Language UFCS: http://ddili.org/ders/d.en/ufcs.html and https://tour.dlang.org/tour/en/gems/uniform-function-call-sy...

Re:GTK 3.99 on Sunday August 02, 2020 @02:14AM (#24026730)
Most embedded graphical Linux applications don't use X (X Windows System), instead they use framebuffer directly or any library such as Qt Widgets or QtQuick/Qml that draws on top of the framebuffer device file. So, there is no need for developing a new Window system that can work on both Desktop or embedded systems.
Re:GTK 3.99 on Sunday August 02, 2020 @01:51AM (#24026654)
Well, most of QT libraries are LGPL, therefore you don't need a commercial license for a non open source project as long as you dynamic link against the Qt library and don't static link against it. On Windows it is easy to pack an application with all shared library dependencies, all you need is copying the shared libraries (*.dll files) to the executable directory. Unlike on Unixes, on Windows, the loader searches first the dependencies at the executable current directory or in directories listed in PATH variable.

Other alternatives are: WxWidgets; MFC - Microsft Foundation Classes that has GUI builder in Visual Studio IDE; Borland VCL - Visual Component Library; Sciter library which allows creating the GUI in html and has a C API. Another way is to build the application on top of Win32 GUI API that is well documented and has lots of examples, unlike X11-Xorg (X Windows System) used on Unixies, such as Linux distributions or BSD variants.

Re:GTK 3.99 on Sunday August 02, 2020 @01:36AM (#24026610)
The lack of standard ABI makes harder to call C++ from foreign-function interfaces or from dlopen/LoadLibrary dynamic loading functions. It lack of ABI also does not allow linking against object-codes built by the other compilers or even older versions of the same compiler.

In C, a symbol name matches the function name, if you have a function called 'void my_function()', the symbol will be 'my_function' and you will be able to load this function from a foreign function interface as libffi.load('my_function', ... types ...). The trouble with C++ bindings is that symbols are mangled, the function names are encoded, a function named 'Workaround1::get_callback()::callback' can be encoded (mangled) as _ZZN11Workaround112get_callbackEvE8callback - this encoding is specific to a given compiler ABI, in addition FFIs cannot load non-C types.

Tensorflow uses the hourglass design pattern, this library is implemented internally in C++, but does not exposes a C++ API, instead it exposes a C wrapper built with 'extern "C" and opaque pointers (void* pointers or incomplete types). This approach allows the library to be used in the same way as a C library and makes easier to call the library via FFI. Another benefit is the better binary compatibility as object-codes built with other compilers canlink against Tensorflow without any issues. Another library that uses this approach is the Squirrel embedded scripting language, that is implemented in C++, but only exposes a C API. So, this technique allows a C++ library to be used as C library and makes possible calling it through FFI.

I guess that in the case of LLVM it may also use this pattern of using C++ internally and only exposing a C API via extern "C". The disadvantage of this pattern is the need of lots cumbersome extern "C" wrappers for every member function of a class and for all exposed classes. LibClang LLVM may be helpful for automating the generation of this C wrapper code.

Re:GTK 3.99 on Friday July 31, 2020 @10:27PM (#24016843)
Those large amount bindings are possible due to the GTK be written in C, which allows creating binding to GTK via FFI - foreign function interface or native interface (borrowing the term from java JNI).

Creating bindings to QT is harder because Qt is written in C++ and C++ compilers does not have a standard ABI and FFIs foreign function interfaces can only load C-compatible functions and types. Another challenge is that older Qt (< Qt5) versions relied too much on MOC - Meta Object Compiler for reflection and callbacks.

The workaround for calling a C++ libraries from other languages is to create a C-API for the C++ library via 'extern "C"' wrappers with opaque pointers. Qt could be used by many other programming languages if it was shipped with an official C-API or interface as previously mentioned.

I a have a proof-of-concept code of a C-API for Qt5 Widgets and three client codes in C, D-language and Julia language which call the C wrapper library and build Qt5-Widgets user interfaces. This code is available at: https://caiorss.github.io/C-Cpp-Notes/CwrapperToQtLibrary.ht...

Another way to generate bindings to C++ libraries is by using SWIG that parses the C++ code generates native interface bindings.

Well, besides ARM, there is also MIPS core or ISA that IC makes can also buy a license and embed into their products in the same fashion as ARM.
Re:Overview of Python dependency management tools on Tuesday June 02, 2020 @07:46AM (#23389991)
You don't need to teach docker. All you need is providing a docker image with everything pre-installed such as Julia, R language, Python, numpy, pandas, Tensorflow and maybe Vscode. And also any Linux distribution, then one can just type "$ docker --rm -it -v $PWD:/cwd -w /cwd my-image ipython" For better convenience, it is better creating a command line wrapper or shell script that saves one from typing that such as $ ./run-my-image ipython. I don't prepare anyone, but I guess that if I knew anything about docker and was given a docker image with everything ready and pre-configured and also a shell script or command line encapsulating all docker command line switches, I would find it more convenient than installing everything myself or fighting some dependency conflict or dependency hell. So, docker can be used as a portable environment development. VScode, aka visual studio code, also supports remote development within docker containers with extensions installed per container. I am a mechanical engineer by training, but I found docker pretty convenient for getting Julia, Octave, R language, Python, Jupyter Notebook server without installing anything or fighting with package manager of my Linux distribution when attempting to install a different version of R, Julia or Python. This approach makes easier for getting bleeding edge development tools without breaking anything that is already installed. I even created a command line wrapper tool for using docker in this way that simplifies all those case: $ mytool bash jupyter-image; $ mytool daemon jupyter-notebook ...
Re:Overview of Python dependency management tools on Monday June 01, 2020 @02:56PM (#23382277)
Besides the dependency management, another major problem of Python is the deployment. Although Docker is not a dependency management tool, it can be used as a deployment tool which encapsulates the application to be deployed, python runtime and all other shared libraries dependencies alongside the configuration. Another deployment tool that is worth mentioning is Pyinstaller. It can pack the python runtime, the application and all dependencies into a single native executable. Pyinstaller is better for Desktop applications and for building single file executable applications.
On Windows, MSVC does not provide guarantees that it will not keep the ABI on new releases. The drawback of using the Qt provided by the Linux distribution, is the reproducibility of the development environment as it is not possible to control the version of the Qt installed and also to have multiple Qt installations with different versions. As a result, the compilation of a Qt application may fail on different Linux distribution which uses a different and incompatible version of Qt. Docker can solve the reproduciblity of the development environment problem, but it still does not integrate well with development tools such as IDEs, building systems and so on.

The advantage of a package manager is the reuse of binary artifact on many projects and also the control over the library version. Conan package manager could be used for providing pre-compiled binary artifacts of Qt library. For instance, provides pre-compiled boost library and poco library for lots of different compilers, compiler versions and operating systems which saves one from building and installing those libraries.

Re:Overview of Python dependency management tools on Monday June 01, 2020 @01:51PM (#23381443)
Python dependency management of packages using C or C++ behind the scenes is really problematic and sometimes, the installation may fail. In this case, a solution is to use Conda or mini conda which provide many pre-compiled packages and also Clang C++ compiler.

An alternative way to allow people without software engineering background to play with Python data science and machine learning tool may be providing pre built Docker images with everything pre-installed which may save one from configuration trouble.

Docker is also useful for learning about new programming languages without installing anything. With just one command $ docker "run --rm -it julia-image", one can get a Docker image containing a GOLang compiler; a Julia language installation; a Rust development environment and everything else. Docker is really a wonderful tool.

The trouble of distributing Qt in binary format is the C++ non standard ABI problem. Qt has to be compiled for every possible compiler and many different versions of the same compiler, in order to avoid linking error. On Unices, most compilers, specially GCC and Clang, are using the Itanium ABI, but on Windows, compilers still have different ABIs, so linking against Qt shared libraries built with older versions of MSVC compiler may result in linking error. On Linux, there still the problem of implicit dependency on GlibC (GNU C Library), most applications built using newer versions of GlibC will not work on distributions using older versions of GlibC. It would be nice for Linux Desktop, if GlibC did not break the binary compatibility.

That said, the most suitable way to get Qt binaries for development may be using a C++ package manager such as Conan or Vcpkg which can cache the library compilation output and reuse it on many different projects without recompilation, unless the current compiler has an incompatible ABI.

Conan C++ package manager recipes can also provide pre-compiled binaries which reduces the compile time pain. Another interesting tool is the conan server which allow fetching conan recipes and pre-compiled binaries on the local network or across the internet.

In the case of SpaceX, it seems that they use embedded Linux and C++ with x86 processors, same as standard PC processors. A reason for using embedded linux is that, it allows using standard C++ or even scripting languages for controlling the hardware from the user-space by just reading and writing files. Linux device drivers (aka kernel modules) maps the hardware to special files on /sys or /dev. For instance, it makes possible to control a GPIO (General Purpose IO) which the device driver maps to sysfs special file system, by just writing 1 or 0 to the file a like /sys/class/gpio/gpio4/value which would enable GPIO 4 and turn on a LED attached to it. Another practical example about this feature is that, on Linux, it is possible to turn on or turn off the keyboard capslock LED by writing to some /sys/class/leds file such as "$ echo 1 > /sys/class/leds/input7\:\:capslock/brightness" which turns on the Capslock LED. By writing 0 to it, the LED is turned off.

This feature of Unix and Linux allows controlling the hardware with any programming language capable of reading and writing files, including Python and standard C++. I guess that they may be using standard PC hardware with industrial IO card. They may also use a single-board computer or custom board with x86 low-power variants processors built for embedded applications as SOC system-on-chip. One example of low power x86-SOC based processor is: https://www.cnx-software.com/2015/04/09/vortex86dx3-is-a-new...