kristian1109
Yes, especially for local things like navaids, cities, etc. Our expectation is that we'll need to fine-tune for different deployments.
> Do you have any plans to make the data searchable/discoverable in real time? Something like being able to search for any audio transmissions regarding a specific flight by number, etc., available on ADS-B exchange, Flightradar, FlightAware, etc., both in text and audio form.
Definitely! This actually already exists and we're selling it to airports today.
Your general point is 100% correct. It's not enough to only do the command parsing; you have to also compare that to radar + trajectory planning to get the full picture. That's exactly what we're working on now. Fortunately, the radar piece is much more deterministic.
One of the best parts of this job is how awesome the labeling community is. So many super qualified folks like yourself. Thank you!
The bird problem is a whole other issue. Mostly handled by PIREP today if birds are hanging out around an approach/departure path.
Computer vision here is definitely going to be useful long term.
On a clear day though, it's probably just people looking out the window of the tower (maybe with binoculars) and realizing in horror that there's been a mistake. The "noticing" is just intuiting the trajectory of the two moving objects...
Of course, this still keeps the pilot in the loop and ideally they will notice if something seems weird.
In a critical function like control, you don't want to split a pilot's attention. You wouldn't want them to sometimes be monitoring a datalink system, but then also sometimes be listening to the radio for deviations. Even if it's less efficient 70% of the time, you reduce cognitive load by training a pilot to ALWAYS go to the radio for clearance and command.
Of course, there are edge cases these days where pilots use datalink for some clearance delivery before taxi and enroute, but you can see how these phases of flight (before you push back and after the auto pilot is on) are selected for very low competing load. In a live terminal environment, you want a pilot focused in one place for instructions.
Furthermore, you're correct that most pilot-controller communication falls largely within tight set of procedures, but deviations are common enough (weather, traffic, emergencies, hold patterns, taxi route, etc) that you find yourself on the radio regularly to sort it.
Last thing: pilots are allowed to say "unable" if they deem an instruction unsafe. I've personally had to do that many times (most common case for me is trying to comply with a vector instruction under VFR with a cloud in my way). VFR may seem like an edge case that commercial planes don't deal with, but again that's not always true in a terminal environment. Plenty of these planes fly visual approaches all the time. And if ATC is talking directly to the computer and not through the pilot, you lose the opportunity for the pilot to quickly and clearly decline an instruction.
> We've accelerated past our capabilities and need to slow down.
This is a super interesting meditation. As much work as there is to be done now, the demand for air traffic is growing and power laws are concentrating it into tight airspace bubbles. It would behoove us to figure out how to make airspace more dense without compromising safety. There's lots of good economic incentive for this.
What were your main takeaways from this paper?
Pitching air traffic management is going to be a years long process.
Getting certified for on-board avionics is similarly challenging.
In the meantime, we'll get better and better at monitoring the airspace system and deploy that technology into unregulated applications like post-operational roles.
I believe scaling laws will hold as we start to feed all of this context data into an integrated model. You could imagine a deep-q style reinforcement learning model that ingests layers of structured and visual data and outputs alerts and eventually commands. The main challenge I foresee here will be observability... it's easy enough to shove a ton of data into a black box and get a good answer 98% of the time. But regulation is likely to require such a system to be highly observable/explainable so the human can keep up with what's going on and step in as needed.
Looking further into the future, it's plausible the concrete structures of today with humans looking out windows will be replaced with sensor packages atop a long flagpole that stream high-res optical/ir camera data, surface radar, weather information, etc into a control room with VR layers that help controllers stay on top of busier and busier airspace.
One problem for sure is that when you're close to the ground you have to be careful about buildings, cell towers, etc. Terrain is one thing, but when you're a few hundred feet AGL, you could quickly be in the way of tall structures if a TCAS alert goes off.
There are lots of small steps on this ladder.
The first is post-operational. You trigger an alert async and someone reviews it after the fact. Tools like this help bring awareness to hot spots or patterns of error that can be applied later in real time by the human controller.
A step up from that is real-time alerting, but not to the main station controller. There's always a manager in the tower that's looking over everyone's shoulder and triaging anything that comes up. That person is not as focused on any single area as the main controllers. There's precedence for tools surfacing alerts to the manager, and then they decide whether it's worth stepping in. This will probably be where our product sits for a while.
The bar to get in front of an active station controller is extremely high. But it's also not necessary for a safety net product like this to be helpful in real time.
2. Agree
[edit] (oops, sorry, seeing your edit)
2. The regulation allows for escalating assurance levels. We'll start with low assurance (advisory) and climb that ladder. We're definitely not naive about it; this will be hard and annoying. But it's inconceivable that someone won't do this in the next 10 years. Too important.
ADS-B helps with this as it's self-reporting. And systems like ASDE-X are useful to track objects once they hit the ground. But low-altitude deconfliction is a big problem.
Military aircraft are typically equipped with UHF radios (in addition to civilian VHF). Many of the same systems apply, just a different RF band. And we're in the process of adding UHF capabilities to our product as a lot of these military aircraft land at civilian airports for training exercises.
I can't imagine what would've happened if we adopted block signaling for ATC ...
Kidding aside, yes, you're exactly right. We're already doing this to a large degree and getting better. Lots of our own data labeling and model training to make this good.
In the very short term, we're deploying this tech more in a post-operation/training role. Imagine being a student pilot, getting in from your solo cross country, and pulling up the debrief will all your comms laid out and transcribed. In this setting, it's helpful for the student to have immediate feedback such as "your readback here missed this detail...", etc. Controllers also have phraseology and QA reviews every 30 days where this is helpful. This will make human pilots and controllers better.
Next, we'll step up to active advisory (mapping to low assurance levels in the certification requirements). There's always a human in the loop that can respond to rare errors and override the system with their own judgement. We're designing with observability as a first-class consideration.
Looking out 5-10 years, it's conceivable that the error rates on a lot of these systems will be super-human (non-zero, but better than a human). It's also conceivable that you could actually respond "Say Again" to a speech-to-speech model that can correct and repeat any mistakes as they're happening.
Of course, that's a long ways from now. And there will always be a human in the loop to make a final judgement as needed.