Did you forget to run the OpenGL teapot?
BeOS was magical. I wish I could use Haiku in my desktop today.
Worse with cooperative multitasking any running application could mess things up.
For its era, System 7 was good, but later, it was worse than even Windows 98.
Be tried to do things right, it was ~1995 after all.
I was taken aback by how slow Windows 95 felt after I finally jumped the wagon and retired my beloved Amiga in favor of a PC, circa 1995. Just moving the cursor was incredibly jerky in comparison and things would randomly freeze for seconds for no apparent reason. Windows NT 4 was much smoother though.
Even today, in Visual Studio, I regularly have multi-second pauses between a key press and the character showing-up on the screen, though this probably has more to do with Resharper than either Visual Studio or Windows.
On Arch Linux this is available as linux-zen
There is also realtime-linux - https://wiki.archlinux.org/index.php/Realtime_kernel_patchse...
This looks super fun to play with, thanks!
You can tell the execs that approved that were probably impressed watching someone else demo it but probably never used it themselves.
Jobs would have spotted it was rubbish immediately.
It has a dial on the front to choose settings. Potentially a great idea to skip quick to the one I want - BUT the damn thing can only detect 1 step of change about every half second. So if you fast-turn it 4 clicks, it still only moves one step. So you have to stand there, slowly turning it, click, click, click, click....
The dial is the biggest design feature on there. Massive. Right in the middle. Bright lights all around it. But they couldn't even be bothered to make it solve the one problem it was there to do.
It's the kind of thing that'd make you reject purchasing it if only you'd thought to have tested that specific bit of it before buying it.
Honestly, I wish that they would make appliances with physical controls as opposed to digital. At least those are easier to fix/mod on your own.
I think if someone did a kickstarter for a very simple physical buttoned digital camera it would do very well.
Caveat: said Arduino may need to sit between the rest of the buttons as well, in case inputs must be given in sequence.
Possible benefit: switch the Arduino for something from the ESP family, and you could feed instructions to the machine over Wi-Fi or Bluetooth. (Which could take the form of a dedicated wireless button panel above the machine with macros for frequently-used options.)
(Hmm. I can tell there's some over-optimization in here, but I'm not sure _where_.)
Finder browsing files is much faster in macOS, Windows scans the entire directory with defender every time you click on it before it allows you to sort it or interact with it. That can take several minutes on some of my directories.
The mouse tracking model in Windows is somewhat different, and a bit more responsive, but there are multiple third party extensions to macOS the make its model exactly the same as the Windows model.
But I also remember not having background processes. It's a trade of.
I’d hope this is much better now given things like Wayland and the increased use of parallelism.
Considering how FireWire worked in comparison to USB that's not particularly surprising. The host machine was more or less out of the way while the FW interfaces did their thing instead of requiring constant hand-holding.
Citation needed. Do you have any numbers on that?
I'm skeptical that BeOS has better latency than modern OSes on modern hardware.
IIUC input is still driven at 60Hz but changing this is in discussion [2].
[1] https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1285 [2] https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/168
Why are inputs synchronized to the frame in Wayland? What the f---? That's just... wow.
The logic isn't perfect and the implementation is flawed but I see where the decision originated.
Since the compositor is already using vsync, it's generally pointless to run graphical applications under a compositor with vsync as well, since that only adds latency. [2] Hypothetically that can result in tearing, however in practice that doesn't matter, because the scanout of the compositor is basically a copy in GPU memory, so you'd have to be rather unlucky to see tearing, let alone consistent tearing. [1]
Of course, OpenGL doesn't support triple buffering (since it has fixed definitions of exactly one GL_FRONT and exactly one GL_BACK buffer per framebuffer/output surface), and it seems like at least nVidia on Linux doesn't support it for Vulkan, either (no mailbox present mode, only FIFO).
In any case, on X11 there is a huge difference between running a 3D UI on a compositor with (A) VSync enabled, (B) running without VSync (which introduces unncessarily high GPU loads, since UIs are generally super-quick to render and if your app architecture is correct, you should be seeing thousands of frames per second), (C) running with a frame limiter (careful choice of limiting FPS to avoid common multiples of 30), (D) rendering a frame as soon as at least one event has been posted, which requires some explicit synchronization work and thought (worker threads have to post events to wake the UI, animations need a timer). (D) is what QML does and what I'm implementing currently. It works well, latency seems to be indistinguishable from B) and C), without the unnecessary power use. X11 only seems to deliver events at a rate of somewhere around 250-400 per second, so an explicit FPS limiter is unnecessary.
[1] If you are rendering to a monitor, scanout happens approx 95 % of the time, so tearing is very likely. Let's say your app's window is 1451x1400 pixels big, then its framebuffer will be about 8 MB. The "scanout" will take something like 15 µs, so at 60 Hz scanout happens about 0.1 % of the time.
[2] My angle here is primarily applications that happen to use <insert 3D API here> as their UI rendering backend. Games have slightly different considerations and should generally just run full-tilt with triple buffering.
https://en.wikipedia.org/wiki/Mutter_(window_manager)
Edit: I missed the part where you were asking specifically about GNOME. Removed everything unrelated to that.
When I scroll in MS Word, there is this insane black tearing across the white surface when I scroll, but only on the laptop monitor.
It's A WORD PROCESSOR! It's just a word processor... It's.... weeps
Or the graphics card isn't being used for some reason? E.g., if your laptop has NVidia's Optimus[1], the graphics card may be inactive while Word's running. There's usually a way to force applications to use the GPU with Optimus.
[1] https://www.nvidia.com/en-us/geforce/technologies/optimus/
People put up with slow responses today because many programmers are simply unaware that instantaneous response architectures are possible.
(And a real-time scheduler isn't enough by itself, the many layers of architecture around handling input events can still be bad even if you're meeting every deadline there.)
The human is still faster than the computer!
"""In an interactive software application, any user action SHOULD result in a noticeable change within 16ms, actionable information within 32ms, and at least one full screen of content within 64ms."""
click About - Not Found
to be fair, the empty white not found page did open probably under 64ms...
</subjective_opinion>
Sounds interesting enough to try. Which operating system is this ?
One really neat demo was 250 windows running the size of poststamps with a little bouncing line demo inside them. All still received their 'fair share' of time, all could still be moved around and expanded as though the machine was otherwise idle.
That's the scheduler. There are a lot of moving parts to running this code, I may make a little project out of restoring it to life under a VM at some point.
Happy reading.
Edit: reading it myself, wow, my English was cringe worthy back then. Moving to Canada certainly fixed that.
Mouse, keyboard, screen. Those are the things that should run at high priority. Everything else can - quite literally - wait.
I might add audio to this list.
Listening to music, on the other hand, does not require low latency except in response to user input.
Please expand on your reasoning here. Maybe hard realtime is optimal for a kiosk or appliance, but a general purpose OS treating "user input" as a hard interrupt seems problematic. I see merit in your position if we had a sub-category of "interactive OS" for general purpose end-user devices.
There used to be a one-floppy installer with the desktop on it, I'm sure you could get that to work in a VM.
Lots of screenshots:
https://www.operating-system.org/betriebssystem/_english/bs-...
Docs:
https://www.mikecramer.com/qnx/qnx_6.1_docs/sysadmin/intro.h...
Unfortunately I can't seem to locate any disk images, rumor has it there was a VMWare image floating around.
Unfortunately QNX 6.5 is pretty old and not too well-supported (QNX is fully POSIX-compliant so you can sometimes get things working on it without too much trouble however) but it's damn fantastic to play around with in a VM. It's actually so fun.
Grr.
In this model, your compositor, input pipeline and (at least the UI latency sensitive part of) your applications would have to be ported to the RTOS, which makes this pretty infeasible. But it works really well if you have a some hard-realtime control loop talking to a bunch of non-realtime network IO or UI code. Would be a fun way to build a wifi controlled quadcopter.
These days you probably want to use Xenomai, or possibly RTAI.
Thanks for the great writeup correcting me!
Is Fuchsia hard real time?
edit: looks like Fuchsia is based on the Wikipedia Overview https://en.wikipedia.org/wiki/Google_Fuchsia#Overview
If the processor is rendering a frame and a network packet comes in, now it needs to respond to the network first, and render the frame after processing the packet. That is a very simplified example on one processor, but there are trade-offs depending on workload/inputs/context-switching is what I'm saying, and measuring latency is step one in trying to optimize for it.
And does the mouse being a ps/2 input vs USB make a difference?
On the developer side, documentation was also amazing. There are tens of full, working example applications. BB10 used QT behind the scenes (Cascades UI). It had also a Python (2.x) binary on the device among other typical UNIX programs.
[0]: https://developer.blackberry.com/devzone/files/design/bb10/i...
[1]: https://developer.blackberry.com/devzone/design/bb10/10_3_vi...
[2]: https://developer.blackberry.com/devzone/design/bb10/keyboar...
The BB10 UI was built on something like QML (the language and runtime) from Qt 4 with their own UI elements and an OpenGL based backend from the acquired company The Astonishing Tribe. They had animations e.g. for slider switches running in the render thread, perfect 60 fps. Qt Quick (the UI framework based on QML, colloquially called "QML") only got an OpenGL backend in Qt 5.
Another very good Qt-based phone OS (after the Nokia N9, got one of these at a conference as well) that failed :(
By the way, the Ford Sync 3 IVI ("in-vehicle infotainment system") is also based on QNX and Qt and it received fairly good reviews. I think I made some tiny contribution to it, if only helping a coworker with something.
This would be one way in which the Rust crowd could really make a difference.
The difference used to be very noticeable. Nowadays my PC doesn't have a PS/2 port.
- USB works by polling for changes at fixed intervals
- ps/2 works with interrupts, so the OS will know immediately when hw does something.
A kid who asks you 1000 times a second whether you are there yet is going to know within 1ms of when you get there, but I imagine we’d all like being the driver much more if there were no “are we there yet”s and all the kid needed was one “We’re here” exactly when you arrive.
That all being true, having a publish/subscribe model is still not the same as having a true interrupt (and I’d have to do some digging to figure out whether there wasn’t still polling somewhere in the pub/sub pipeline)
The difference is real time control of stepper motors with crazy speeds when microstepping at 300K steps / second or more interpolating across 5 axis vs moving them at a snails pace of maybe 10K steps / second (on really nice and otherwise idle hardware, miss a single step and your goose is cooked, now you have a cumulative error).
This intuitively makes sense especially with SSDs and especially with NVMe SSDs: you're not going to have to wait that long, especially compared to the overhead of an interrupt. Also, if you're reading sequentially, once you get one I/O completion you're going to get a lot more in a short timeframe—it makes sense to just poll for those instead of handling every one as an interrupt. Interrupts are pretty expensive (I honestly don't have a good grasp of how Linux interrupt handling works but it can easily be on the order of microseconds.)
It also has some side benefits like preventing the CPU from going into a lower power state for only a short period of time.
If it’s possible to interrupt on a serial port, are there existing examples of how to configure a mouse or keyboard to interrupt over a serial port?
Typically a serial port would contain a small buffer which would fill up, upon completion of the first byte an interrupt would be generated and you'd respond to that and read out the register freeing up room for more bytes to be received. Transmit the same but reversed, as soon as a byte had left the shifter in the chip it would generate an interrupt so you could re-use that space for more bytes to send.
This works quite well. Hardware flow control can help in case the OS doesn't respond fast enough to the interrupts, so you don't lose characters.
RTOSes are often found in autopilot and vehicle management systems controlling critical peripherals or safety critical software. More mundanely there are also sensors that will get upset if you don't give them undivided attention for fixed periods of time (or if you interrupt them, they will terminate the transfer). Image sensors are particularly picky about this.
It's a good lesson to play with polling versus interrupts on a micro. Susprising just how many cycles it takes to jump into the ISR.
I've run a plasma cutter under Linux on a run-of-the-mill VIA SBC, it was actually pretty easy: just make a program setuid root, use that to disable interrupts after you've read all your data in and then just go wild on polled and precision timed IO until you're done. Re-enable interrupts and return as if nothing ever happened.
More to read here: https://stackoverflow.com/questions/5452371/why-isnt-every-o...
Was this a QNX variant?
I'm wondering how you'd rate iOS, especiallly with the ProMotion displays on the iPad Pro. Do they come close in terms of responsiveness?
Which one?
They should start measuring the time it takes for monitors to recognize the signal when you press the "Source" button.
Even on recent monitors, it's often 5 seconds or more.
Basically, the device has to sweep through several different EQ settings until it achieves the combination with the best signal eye.
You'd think this would be lightning fast, but as it turns out, this is often handled in firmware (with hardware support for measuring the eye). I've seen this process take upwards of two seconds.
A lot of these displays probably only have one RX core, so it's not like it can remain locked to all inputs at the same time.
They have a lot of popular models and super in depth reviews.
The latency in the hardware before the signal makes it to the usb (for most keyboards, even specialised gaming keyboards this is like 30ms, which is 2 frames).
There is also the latency from when the monitor gets a signal to when the pixels have perceptually finished transitioning. This can be 10s of ms too.
So even if the OS has 0 latency, and the monitor measures that, you could still easily observe a latency of say 60ms.
The video mentions that they get 15 ms end-to-end latency on a 360 Hz monitor in Fortnite. So that's what it takes for a modern system to finally beat the latency that you used to get on, say, an NES connected to a CRT. Still an order of magnitude above the limits of perception though [2].
I often wonder at what cost, could we get it down to Sub Ms. Where everything from Input, Video and Audio are sync to the precision of 1ms. There used to be very little incentive to do so. Now that Gaming has taken over I hope it will develop some market perception and value so we could finally move into that direction.
The age of Instantaneous Computing.
Worth noting that unlike some of the other sources being discussed, this source of latency is more due to physical constraints than bad engineering or bloated software. Keyboards are limited by the physical travel time & de-bounce of the mechanical switches themselves.
So this is more a tradeoff of using mechanical keyboards at all rather than "gah bloated electron" or whatever.
The other common peripheral, mice, don't have this mechanical constraint. They can even achieve sub-10ms for clicks due to the difference in switch expectations ( https://www.rtings.com/mouse/tests/control/latency )
EDIT: Also 30+ms seems to be quite far off. Cherry MX's specs are around 5ms, and I'm getting 8ms playing around with this tool: http://blog.seethis.link/scan-rate-estimator/
I mean, yes but not really? There's very real speed limits to the human finger, after all.
But the times given above are also quite far off - the actual keyboard numbers seem to be more in the 5-15ms range, not 30+ms. At least, my mechanical gaming keyboard is hitting 8ms on this test: http://blog.seethis.link/scan-rate-estimator/
If you only have to worry about false deactivation and there's no line noise, and your keys always stay down long enough to be caught by your scan interval (or they trigger an interrupt), then probably yes. If you have to worry about false activation, then probably no.
You certainly don't want to low-pass filter the whole thing, or otherwise wait for the contact to settle before reporting the keystroke or mouse click.
In general, you can make a denouncer with zero latency: the receiver still triggers on the first logic change for idle to active, but then gates any further changes for a while.
(At least, that’s how I design all my debouncers.)
There may be aspects that are specific to a matrix keyboard configuration that prevent this...
It seemed a reasonable number for a goodish keyboard on that list
An interesting set of numbers, but doesn't tell you all that much about the hardware quality itself. The actuation point and travel distance of mechanical switches is itself an endless source of user opinions & discussions.
I will say that I am excited about 8k video for computational photography. I am studying computer vision for robotics and it is quite clear to me that a simple spherical lens with a very high resolution sensor would be a very good imaging system for a real world robot.
I recently got a Panasonic GH5 and it shoots 18 megapixel “6k” 30fps video, encoded in h265 (important for higher res). I am experimenting with photogrammetry using this photo mode. The initial results are very promising. In four minutes I took a scan of a short street block in Oakland, and after processing the photogrammetry solution I have a very good dense 3D full color map of the street. The model has holes but I am slowly learning how to plan a photogrammetry capture. Currently computation takes hours but I am seeing more and more research on ways that machine learning can improve compute times and data density.
See how low the visual acuity is on 5.6k spherical video here: https://youtu.be/nASvIYq3VkE
However all this is to say that very high resolution sensors are a very good thing for robotics.
So, all free/open multi-view stereo and structure-from-motion software I know of is incapable of handling rolling shutter artifacts.
The problem seems to be that electronic global shutter sensors lack (some) dynamic range compared to otherwise-equal rolling shutter cameras.
If you'd be interested in talking more about this, contact me/let me know (I'll get in touch if requested).
My photogrammetry experiments typically encompass low-effort bulk data collection, though the search for a light-weight camera to use with a prosumer-class drone and some revelations about reconstruction quality issues inherent to older, easily-optimized algorithms for both multi-view stereo and mesh reconstruction stalled progress somewhat.
In general, machine learning doesn't seem to be as much of a benefit as one might guess, when compared to applying the resources in non-ML ways to the data.
Mind teasing some numbers from your street capture?
I did see in this paper that it seems like rolling shutter compensation is possible: https://research.fb.com/wp-content/uploads/2019/09/An-Integr...
I wonder if a library could be produced for this?
As far as my street capture I am currently using 830 frames from my 4 minute video (2 fps), images are 4992 x 3744 shot with a 12mm lens. The reconstruction is still texturing so I don't have the final result. What kind of numbers are you interested in?
Please contact me yes! My email is in my profile. See also my four wheel drive robot with four 4k cameras, and feel free to actually initiate discussion on that site where others can see it too (or emails is good). Thanks!
I'm aware that global shutter cameras are more expensive, though I haven't even found a self-contained one that's notably below 500 g.
I'd like to know the polygon count, both vertices and faces, if possible.
I'll check the site and write more tomorrow, with the sun coming up shortly I should catch some sleep.
Also, yeah, rolling shutter can be compensated, but the impact goes through all the pipeline steps. All of them.
I don't want to re-write the whole pipeline. Worst case I'll settle for an Apertus Axiom Beta.
Also, a global shutter works nicely with a strobo, and a NIR-enhanced sensor works well with e.g. 808 nm light sources. The reason would be around power conservation when the robot has to bring it's own light source.
My last experiment had about 5k or so depth maps getting reconstructed (15 fps 4k Mavic Pro footage, iirc 100Mbit/s), from barely over 20k captured frames that I did full SfM on. The reconstruction density was adaptive, manually choosen to ensure sufficient scene coverage without spending too long on the depth map reconstruction.
The issue is that you have to assume somewhere between 1/60 and 1/500 exposure time for motion blur, if operating without an electronic global shutter. And I like detail, so I'm more looking for something like a 50 mm lens in front of an academy-sized sensor, yielding about 70~90 mm 35 mm-equivalent for the typical 4k by 2~3k sensor.
Many older (and now inexpensive) Blackmagic models have a global shutter, and shoot RAW at 25 fps / 4k or higher.
Would you reccomend a wide angle lens for this purpose? Intuition tells me a telephoto would capture more detail, though with the caveat of requiring more movement over the surface area to be captured.
But some people are capturing individual rocks to be used for video game assets, or surveying historical sites or something else. There are times when a telephoto would be desired, especially if you cannot get close to the target scene.
4k is 8.5 megapixels so it’s relatively low detail compared to my 18 megapixel video from my Panasonic. I personally do not really know the math behind any of this so I am not sure where the trade offs are between high resolution rolling shutter or lower resolution global shutter.
Because I am doing scene capture for robot simulation, I do not need a perfectly accurate model. I would rather have a machine learning algorithm that places simple polygons where real world objects are in the video and gives them a reasonable texture. That is, I want to go from video directly to a low poly textured 3D model. My problem is complicated by the fact that I am developing an off road robot, and photogrammetry of forests is very hard.
This is why I mentioned machine learning. I care a great deal about the semantics of my scene and less about metric accuracy. What kills me is compute time. Finding feature points in every image and matching them all together is extremely compute intensive. I believe shortcuts can be discovered for problems like mine with creative neural networks. That said, I still need to learn more about existing algorithms.
Those black magic cameras are severely bandwidth bound, compare to e.g. the Pocket Cinema 4k (USB 3 and able to fully load a USB-C SSD), which unfortunately only offers a rolling shutter.
The lens is only slightly relevant, in that an extremely long focal length will cause issues in estimating said focal length, and typically also in depth-of-field if that's relevant for your scene. I suggest no ultra-wide-angle lens and no more than 500 mm (35mm-equivalent). 30-150 mm focal length (35mm-eu) would be the range you want to look at.
Because of how it was implemented, you could drag VLC around while the video was playing and the video would stay "behind" everything, with the VLC window acting as a "hole" through which you could see it. (So you could move the window to the left and see half a black square on the left, and the left-most half of the video on the right)
Nowadays with desktop composition AKA DWM, Windows just makes sure to black out DRM content from any frames/video it sends to an app requesting to capture the screen, making sure to send the video-including composed desktop only to the display. (And if you have some GPU recording software like NVIDIA ShadowPlay, it switches off when DRM content starts playing) You can see it in action with the Netflix UWP app. Of course, a bunch of DRM-respecting software -- like Netflix in Google Chrome -- doesn't really follow that spec and can still be screenshot/video captured like any app.
Which is the same as Netflix effectively, Netflix just provides higher to PCs when they can block screen recording entirely. Disney might eventually I guess.
The problem—and it is purely a firmware problem—is that all such monitors make this kind of support "dumb", with hardcoded geometry (i.e. split-screen as exactly halving or quartering the screen; PiP as putting one input in an box that composites over exactly the upper-left quarter of the lower-right quadrant of the screen.)
There's nothing in the scaler ASIC that particularly benefits from these numbers being hardcoded, such that its job would be any more complex if they were controllable via machine-registers POKEable using HDMI-CEC commands.
HDCP runs over DP and Thunderbolt. But derefr's discussion above is accurate.
Related, "Your mouse is a terrible webcam", 2014: https://hackaday.com/2014/01/14/your-mouse-is-a-terrible-web...
Yes. The industry has been optimising for throughput in the past decades. It is time to bring latency back to the table. I want super low latency Computing. Street Fighter or The King of Fighters in CRT Arcade Era just felt so much more responsive.
(Melee on a CRT has 3 frames of input lag)
This is just a matter of opinion. Wide gamut is more important to me, latency is more important to you.
Thankfully, we can just have all three :)
Not at this time. There are some hypothetical options like the LG 27GN950, but it has poor contrast and uses Displayport 1.4 compression, which is only supported in the latest graphics cards. VESA and friends really made DisplayPort 1.4 as close as they possibly could to false advertising without actually committing it (because the only relevant new thing in DP 1.4 -- DSC, display stream compression -- is technically optional and no one claimed they'd support DSC while saying they support DP 1.4, which is pretty much the same as supporting DP 1.3, since nothing of note changed).
And now we're looking at DisplayPort 2.0 which can already barely support uncompressed 8K at 60 Hz and is basically maxed out by 5K/6K 120/144 Hz. And it's unclear if the presently introduced generation of GPUs even supports it, or if we're going to use effectively-2013-Displayport until ~2022.
Note how the marketing material only talks about HDMI 2.1; DisplayPort isn't mentioned once.
OLED is clearly the way forward - accurate colors, excellent contrast, no bleeding, no uniformity issues, proper HDR, excellent response time. Except OLED doesn't come to PCs.
(there is a guy on Youtube who installed an OLED TV as his main monitor: https://www.youtube.com/watch?v=AXKhYH3BaIA&t=36s)
MicroLED is another promising tech that has a lot of OLED upsides, but no burn-in issues.
Unfortunately my desk's dear centerpiece, https://iiyama.com/gl_en/products/prolite-x4071uhsu-b1/ , has been EOL'd about 2 years ago, because I have been unable to find a replacement that's not worse, while staying in the 40~50" range. Any concrete suggestion would be greatly appreciated.
It's long overdue to have widely available metrics for latency in consumer tech. Many hardware/software setups have absurdly long latency for no good reason other than that it's difficult to measure. People underestimate the subconscious effects it has on your usage of technology.
I couldn't be happier that phone and monitor manufacturers are finally starting to compete on refresh rates >60 Hz. It's far more important than wide gamut or 8K.