The problem is that most companies using OSM are using Mapbox. This leads to a few problems, including attribution, which I decided not to touch on, but is a serious problem that Mapbox doesn't address sufficiently.
If they could use OSM directly, they'd be engaged with OSM directly. They'd care about OSM. Right now OSM shoves these users away and makes them into customers of some third party (maybe Mapbox, maybe someone else) instead of caring about OSM itself.
> Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.
How about letting users experiment and seeing what cool stuff they can come up with?
Is that even true? And is number of companies even a good metric? Based on revenue/market cap, the bulk of companies use OSM directly.
The world's biggest company (Apple) uses OSM directly. Telenav uses it directly. Don't Tesla and Uber use OSM directly too? And we use Mapbox at Gaia GPS, but we also use OSM data directly.
> How about letting users experiment and seeing what cool stuff they can come up with?
In my experience building a company, if you focus on too many things, you get everything wrong. This smells to me of being not on target for making OSM as strong as possible, even though it has alluring upside.
https://electrek.co/2017/07/03/tesla-map-navigation-open-sou...
Actually it does. Especially since you can have 1 billion in revenue by selling $10 dollar bills at $1 dollar.
>b) having the biggest market cap is something thats good for their shareholder, and terrible for anyone else
Whatever.
>But yeah, do continue to praise your beloved highness.
Irrational argument ad hominem.
It's an awful measure for "biggest company", since private companies have no market cap. This was discussed a while ago in relation to Saudi Aramco's potential partial IPO. Saudi Aramco is huge, much bigger than Apple, with no market cap.
They absolutely do. They're just updated less frequently and called "valuations". Closely-held private companies also have valuations, they're just even less frequently made visible.
In the way the Fahrenheit and Celsius scales "are entirely different things." Both "valuation" and "market cap" are measures on the aggregated estimated present value of future cash flows, i.e. net asset value, from/of a security.
can't there be an open source set of service
APIs that's self hosted
Unfortunately, the developer experience is much more painful than using Google Maps.
With Google Maps, if you can add Javascript to a web page you're basically good to go. Data updates, software updates, server redundancy, server support and security, all taken care of. Even if you're exclusively a front-end dev, you're making a static site, and you've never heard of Linux.
Compared to that, self-hosting OSM tile generation requires a bunch of extra knowledge, equipment and maintenance.
With image tiles it's a hurdle. You have to build 18 zoom levels of tiles. The most detailed layers have an astronomic number if tiles. You also need a huge amount if storage.
http://tools.geofabrik.de/calc/#type=geofabrik_standard&bbox...
Did I mention the data being huge? What's the point of having 15Tb container, which takes forever to deploy? You can certainly containerize the services, but there's no benefit containerizing the data, since you have to replace it all at once rather than synchronizing the differences.
Even if you could make the general setup and maintenance incredibly easy somehow, just syncing the map data from osm regularly would require decent hardware: the extra cost of each dev doing this themselves just doesn't compare to using a central tile host.
On top of that, if every dev was regularly updating their own OSM data, OSM would have a lot of extra strain on their data download servers to deal with. Not sure if this could be comparable to the hardware overhead of providing a tile hosting service though.
We need servers with at least 24 CPU threads and 256 GB RAM and moving the data into PostGIS, region by region takes a week or two. Plus you manually have to fix the process when it just spits out an error and aborts after 15 hours of churning.
You can get country level extracts, which can be updated every day. Nearly any server which can import OSM data is able to keep it up to date.
I have a hobby site about railways [0] I'm picking up again after a decade of inactivity. I wanted to display a route map of each railway, with clickable stations and the route highlighted.
I can not work out how to do that. OSM has the route and station data (there and in Open Railway Map [1]). I expected to find a layer called 'Railways' with sub-layers, one for each railway, and I could export the data and highlight it on another map.
Instead I have raster tiles, and no way to highlight the railway I'm talking about. I know the data must be there somewhere but how on earth do I get it?
[0] https://www.narrow-gauge.co.uk [1] https://www.openrailwaymap.org/
https://catalog.data.gov/dataset/railroad-kmz
A brief google search came back with this; I haven't inspected the data or even tried to use it to see what it contains, but the page description indicates it may not be exact routes you are looking for-but it's a start. Implementing the data is a bit more work in order to display route maps, something like the previously mentioned MapBox does this very interactively.
My email address is in the profile, feel free to shoot over an email if you'd like some help with it. Depending on the complexity, I can probably devote a few cycles this weekend to show you a one-off map with railway routes, and show you how it's done.
I'm only a hobbyist with GIS, so I can't really say or speak to how practical of a format KML/KMZ are in production.
But that's a really good question.
KML/KMZ is decently useful as an interchange format and simple rendering but almost all web GIS software has better support for GeoJSON. It’s richer and easier to work with.
I’ve never worked on a GIS system that used KML/KMZ in prod for rendering. It’s usually shapefiles, PostGIS Geometry, ESRI’s proprietary GDB feature, WKB/WKT or GeoJSON on the backend being served up to clients.
I have used it as a lightweight way to deliver simple data in the past though for use cases where there was no proper client renderer (we would just tell the end user to download google earth and open up the file)
https://wiki.openstreetmap.org/wiki/Railways
In the sense of being easy to retrieve I guess it isn't a layer. Global railway data will be pretty huge, so Overpass-API may not work out if that is what you want. In that case, the strategy is to grab a planet file (a snapshot of the db) and filter it locally:
http://osmcode.org/osmium-tool/manual.html#filtering-by-tags
And then you probably have to translate the resulting data into a format that you would use for your map.
As to why you have to do that work instead of just pulling layers, it comes down to the core data model being focused on editing and there being many different end use cases (making it difficult to anticipate and serve each one).
And really, any strict layering system in OSM would quickly just fall apart into debates of exactly what belonged in each layer. The range of detail captured by OpenStreetMap users in different areas is really far too nuanced to work crammed into coarse, crude "layer" classifications.
Absolutely not. Not only is the selection of types huge and full of shades of grey, but an element can be tagged as things from multiple different categories at once (and this is a wonderful feature). Take a browse of the OpenStreetMap wiki sometime. OpenStreetMap data is far richer than a layer model could cope with.
However, let's not confuse this with an editor having a layers-like interface feature to help the user focus on a particular subset of the data. JOSM for instance does this very successfully with the "filters" tool, which I use heavily.
e.g. the railway nerds have very detailed ways to tag railway infastructure: https://wiki.openstreetmap.org/wiki/OpenRailwayMap/Tagging
My current solution is based on OpenMapTiles and uses their stack for creating, and subsequently rendering, vector tiles. First, I obtain an extract of OSM data via the Overpass API. Then I convert it from XML to protobuf, as it's a much more efficient format. Afterwards I import it into a PostGIS database using imposm3 and define some SQL functions that transform that data. Then I generate some vector tile definition from the OMT format into tm2source format. After that I finally generate the vector tiles in mbtiles format using tilelive, server them as GeoJSON using tileserver-gl and style them using maputnik and mapbox-gl styles.
There seems to be some bug with imposm3 and relations, because I can't get them to import. Or I'm just not doing it right.
It sounds like Overpass is the way to get the data I need - once I figure out how to reference it in Overpass (network/relation details).
It talks about the errors in Open Street Map data, and the consequences of that for railway routing.
The main point against a service seems to be that’s it’s an expensive thing to do, however I don’t see how we can draw any conclusion from that without weighing the costs against the upsides.
I don’t see how the expense argument can be made with confidence without first establishing some basic data points. For example:
1) How expensive is expensive?
2) Have the benefits and advantages been throughly explored ie what’s the value proposition?
3) Given this value proposition, what level of fund raising success would be plausible?
As an aside, the ineffectiveness of leadership and how it is seemingly blocking progress is quite striking.
Off the top of my head, I can’t think of any actively developed software with a non-deprecated api that hasn’t changed in so many years.
Sure, these aren't provided by what's formally known as the "OSM API". They are provided by OSRM/Graphhopper, Mapbox GL/Tangram, Leaflet/OpenLayers 3, and Overpass respectively. Because that's, deliberately, how OSM works: it's the centre of an ecosystem, not a single service provider.
The OSM API is for the use of map editing programs. It's an implementation detail for the authors of iD, JOSM, Vespucci and a few others. It isn't something that anyone who isn't writing an editor needs to concern themselves about; and I honestly haven't heard many comments from editor maintainers that they want the API to do things it doesn't at present (though, Bryan, Dirk, Simon, do jump in and correct me if I'm wrong).
Mentioning OSM has gained functionality is irrelevant, the article’s point is the API which serves a different purpose. What other companies offer is irrelevant that’s not OSM.
Your other points seem to attempt to diminish the importance of the API, yet the state of modern software has become such that almost any significant software project can benefit from a rich, well implemented and maintained API.
Not to mention maintaining a high quality API often in and of itself leads to improvements in the overall architecture of software, due to how it encourages careful thought about the purpose and organization of the system.
The OSM ecosystem has gained some new, much more powerful APIs, which Doctor_Fegg mentions. OSRM's API, GraphHopper's API, Overpass's API, these are all better than the APIs that were available in 2009
What other significant, widely used software, depends only on the “ecosystem” for APIs development, while having no active API efforts in the core project?
I can think of none. Maybe some examples exist somewhere, but at the least it’s unusual, especially given the overall context of the article.
HTTP itself is fundamental and a different level of abstraction. We would not want monthly updates.
A closer analogy is Wikipedia (see wikimedia api). It hasn’t had the fastest evolution, but it makes OSM look like the stone age. You could look at the github api as a project that’s invested more frequently.
And it’s not just because these are projects with vast riches (even though WP’s multi-million budget is actually tight). There are countless smaller projects that keep a well maintained API. I just list these because they are well known.
The role and importance of APIs in software has changed dramatically over the last 20 years. It’s true at one point it was more of a luxury and just wasn’t as central to many projects.
Those days are far enough in the past and so anachronistic that a ~10 year stale API at this point should be considered architectural malpractice.
And then you could lay out why those things should be done centrally rather than on separate servers. For example, there is a sophisticated data query api https://wiki.openstreetmap.org/wiki/Overpass_API that keeps up with edits within a minute or two but is free to lay out the backing database in a way that is optimized for queries and people can run their own instance and so on.
Then how is this productive you ask? Because for now I’m only drawing much smaller conclusions, which also generalize to most actively developed software. Rich, well maintained APIs tend to be useful. As of yet I see no real explanation of why that doesn’t apply to OSM.
To start, not sure why you mention a central server vs. “separate servers”. SaaS users shouldn’t care, or ideally even need to know whether something runs on 10 or 100 servers.
Maybe instead of separate servers, you meant different companies? Knowing which organization is the provider of each service is paramount. For each provider it’s also important to know something about their business model, reputation, roadmap, and the pros/cons of using multiple providers vs. one stop shopping. It’s important enough that companies like Gartner are paid a truckload of money by SaaS users to analyze these details.
So back to my original point: Most significant and active projects benefit from investing in rich and well maintained APIs.
Take that fact, and add in a domain expert taking the time to write an article asserting it’s indeed a problem relevant to OSM. One person’s opinion is not always a smoking gun. But combined with the fact a 10 year old stale API is so unusual, and the fact that what other companies do is a red herring argument being made, at minimum it sure seems like a bright red flag that warrants better understanding of what’s going on over there.
There's a particular service running on some OSM servers that provides data to editing software and receives changes from editing software and applies those changes to the master database. The primary version number on that api is 10 years old. It has still evolved over that time. But it is not the entirety of the OSM API, it is the OSM editing API, which probably benefits from being at least somewhat stable.
I mean, I guess the basic data model could be changed all the time so that all the tooling had to chase after it, but the only big proposal along those lines is to add an explicit "area" type which would simplify some things (it wouldn't really create room for any new capabilities).
Take a look at the vector tile stack Mapbox has put together if you want to see a new way of working with OSM data that has come about over that same 10 year period. Mapzen and OpenMapTiles have also worked on vector tile stacks.
I had to implemented a specialized geocoder/administrative boundaries referential last year. Mapzen was really compelling, the documentation about how they handled hierarchical data was clear and interesting but execution was lacking. I am French, handling European data was important and I quickly hit several pain points:
- Their github repository organization was a mess. A lot of small ones without a clear map of which did what. - The vagrant setup to experiment locally was unmaintained - I found a basic unicode normalization issue in the geocoder several hours after playing with it. - Manipulating their dataset via git-lfs was a massive pain - Their geocoder was nicely presented but relations to administrative levels was done with neightbourhood search to administrative area centroids, instead of (even approximated) geometries. - The data was not as comprehensive and up to date as OSM one, at least in France. For instance, France second level administrative areas (Régions) were redesigned at the beginning of 2016. They were defined in OSM but progress seemed fairly slow in Mapzen.
Now, I appreciate Mapzen was a very ambitious project and handling cartographic data at this scope and scale is a difficult job, but as pure data consumer, it was not comparable to OSM.
I eventually used OSM data and while their documentation/tools are quite a mess too, wiki style, it is workable. Datasets are large but there are a lot of way to retrieve them efficiently. Their non-protobuf binary format hits a sweet spot between size efficiency and ease of processing. And at least in Europe, the data is fairly good, at least for my requirements.
Maybe I'm misunderstanding the whole domain, but if OSM would get layers, that would lead to better composability, then wouldn't the question of transient/frivolous data kinda become moot? The way I see it, in that scenario I could make and host my own whatever pokemon layer and OSM proper could provide the base layer, and then they could be easily merged at usage point; whereas based on the discussion now without layers you'd need to essentially make hard fork of the OSM data to make your own pokemon map?
I don't really agree. I have seen so many wrong imports and I have seen imports demotivating individual mappers.
With a good data source an import can be a good start, but it's not a silver bullet.
I agree to your other items, though.
That's strange. What were you trying to do? If you convert your data to OSM's relatively simple XML format, you can open that file in JOSM, and just press the upload button. That's what most people do with imports. It's possible to communicate with the OSM API directly and make API calls, but that's essentially what JOSM does. JOSM also has a validator, which can spot some problems with your data.
Isn't all of the data in OSM transient to some degree?
What's the threshold for when data is too fleeting to be worth storing? Days, months, years, decades?
Wrong that OSM should be a service provider and not a database - I think it should be a database, and corporations are good for OSM. The post calls for OSM to essentially become Mapbox. It's insanely expensive to be Mapbox, and Mapzen just failed trying to be a less corporate version of Mapbox. And then the usage policy part of the article is moot or at least not that consequential if you don't believe OSM should be a service provider.
Right: Moderation tools need to be created/fixed. New mappers need to be encouraged/onboarded better. Vandalism is a problem. OSMF culture is toxic. Hidden gatekeepers are toxic. Imports need to be supported. Bots need to be enabled, and we need to consider people who contribute code/tech/imports first class OSM contributors and not just glorify the individual mapper.
Probably right: OSM needs layers, or even a more sweeping re-architecture to allow better versioning, moderation, better tooling, and easier understanding of the data.
Probably wrong: Expanding the scope of OpenStreetMap to include transient data. Too much other work to do, maybe later for this.