emw
> "it is important to note that Creative Commons licenses allow for free reproduction and reuse, so AI programs like ChatGPT might copy text from a Wikipedia article or an image from Wikimedia Commons. However, it is not clear yet whether massively copying content from these sources may result in a violation of the Creative Commons license if attribution is not granted. Overall, it is more likely than not if current precedent holds that training systems on copyrighted data will be covered by fair use in the United States, but there is significant uncertainty at time of writing."
The new Wikimedia Enterprise APIs facilitate attribution. For example, the "api.enterprise.wikimedia.com/v2/structured-contents/{name}" response [2] includes an "editor" object in a "version" object. So the Wikipedia editor who most recently edited the article seems quite feasible to attribute. ML apps could incorporate such attribution in their offering, and help satisfy the "BY" clause in the underlying CC-BY-SA 4.0 license for Wikipedia content.
---
1. https://meta.wikimedia.org/wiki/Wikilegal/Copyright_Analysis...
2. https://enterprise.wikimedia.com/docs/on-demand/#article-str...
As someone who avidly edited Wikipedia for 6-8 years, I am happy to see my volunteer work used for LLM training. I also agree some other editors likely aren't.
Wikidata is also worth considering for that task. It is:
* Directly linked from Wikipedia [1]
* The data source for many infoboxes [2]
* Seeded with data from Wikipedia
* More active and integrated in community
* Larger in total number of concepts
Wikidata also has initiatives in lexicographic data [3] and images [4, 5].
On the subject of Cyc: the CycL "generalization" (#$genls) predicate inspired Wikidata's "subclass of" property [6], which now links together Wikidata's tree of knowledge.
---
1. See "Wikidata" link at left in all articles, e.g. https://en.wikipedia.org/wiki/Knowledge_base
2. https://en.wikipedia.org/wiki/Category:Infobox_templates_usi...
3. https://www.wikidata.org/wiki/Wikidata:Lexicographical_data/...
4. https://www.wikidata.org/wiki/Wikidata:Wikimedia_Commons/Dev...
5. See "Structured data" tab in image details on Wikimedia Commons, e.g. https://commons.wikimedia.org/wiki/File:Mona_Lisa,_by_Leonar...
6. https://www.wikidata.org/wiki/Property_talk:P279#Archived_cr...
I would be surprised if Gage and his parents have spent more than $10,000 of their personal money on this hobby. Again, cost near that range is certainly significant, but not monetarily immense for a middle class kid with a consuming hobby and supportive parents over the course of 7+ years.
Gage has been frugal in his choice of college, and gets funding from GoFundMe campaigns. He also seems to have had side jobs. Simply choosing to attend a community college and then an in-state public university as Gage has done -- rather than a private university for 4 years -- is probably enough to defray a huge portion of his hobby's cost.
I suspect Gage is also frugal in his means of travel and lodging. A sibling comment mentions the possibility of photographing candidates that come within a day or two trip of home. I imagine that accounts for most of Gage's photography.
Consider this note from [2]: "Skidmore is a 19-year-old student at Glendale Community College in Phoenix and a freelance graphic designer. A Ron Paul supporter, he began photographing politicians when he was living in Terre Haute, Indiana, attending events held by Rand Paul during his successful 2010 Senate run in Kentucky." The drive from Terre Haute, IN to Lexington, KY is about 4 hours. That's completely doable in a day trip. I've driven 4 hours each way in day trips for similar free culture pursuits. It costs about $80 for gas and food.
> "between states to more than 40 speaking engagements."
Travel among multiple US states to attend 40 speaking engagements over the course of 7 years is not necessarily a major financial burden, even for someone Gage's age.
> "I traveled to nearly every part of the country to cover his political events"
"Part" can be pretty general. One could have covered events in Arizona, Iowa, New Hampshire and, say, Virginia and say one has traveled to nearly every part of the country -- the American West, Midwest, Northeast and South.
> "Skidmore is hot on the campaign trail again, toggling his time between New Hampshire, Iowa, and Arizona"
I think it's much more likely that Gage has been to New Hampshire and Iowa each once or twice in the 2016 campaign season, rather than flying out every weekend or so like a high-level political operative or corporate executive from his Arizona State University dorm room.
---
[1] http://priceonomics.com/how-a-college-student-used-creative-...
[2] http://www.niemanlab.org/2012/09/how-a-19-year-old-student-b...
The bigger cost in Gage's photography is almost certainly time. He's uploaded almost 40,000 photos to Flickr since December 2007 -- roughly 5,700 photos a year. Even with the one-person-many-angles style seen in his Flickr stream at https://www.flickr.com/photos/gageskidmore/with/24744613302/, that's a lot. He also captions his photos, e.g. https://www.flickr.com/photos/gageskidmore/24744613302/.
The vast reach of his photography is likely a sufficient incentive for Gage to invest all that time and significant-but-not-immense money. I hope Gage continues his excellent work.
* San Francisco (Saturday): https://en.wikipedia.org/wiki/Wikipedia:Meetup/San_Francisco...
* New York City (Saturday): https://en.wikipedia.org/wiki/Wikipedia:Meetup/NYC/Wikipedia...
* Boston (Saturday): https://meta.wikimedia.org/wiki/Wikipedia_15/Events/Boston
* Bangalore (Sunday): https://meta.wikimedia.org/wiki/Wikipedia_15/Events/Bangalor...
* London (Sunday): https://meta.wikimedia.org/wiki/Meetup/London/101
* Portland, Seattle, Vancouver (Saturday, meet Ward Cunnigham!): https://meta.wikimedia.org/wiki/Wikipedia_15/Events/West_Coa...
New York will feature a talk about Wikidata, how to query it with SPARQL, and how we are integrating it with Wikipedia and pushing forward the Semantic Web. Other NYC talks include things like "Git-flow approach to collaborative editing", "Copyright and plot summaries", and "Automated prevention of spam, vandalism and abuse". We will be linking up with San Francisco and likely some other cities for a global teleconference at 4:00 - 5:00 PM ET (21:00 UTC).
If you're interested, sign up and stop by!
Regarding Wikidata and DBpedia: to my understanding the latter gets much of its content by scraping Wikipedia infoboxes. Wikidata will increasingly provide data for those infoboxes, and thus DBpedia.
Regarding your fears: I don't share them. Wikidata will greatly enhance the accessibility of knowledge for open access research.
'Instance of' and 'subclass of' provide Wikidata with a way to express the basic philosophical notion of type-token distinction [3]. For things that are a subclass of something like 'material entity', all instances are physical objects that have a unique location in space and time.
Not all instances are spatiotemporal particulars, though. For example, one might say "Homo sapiens instance of taxon", where taxon is a metaclass, i.e. a class in which the instances are classes. (Here 'taxon' would not be a subclass of 'material entity' -- i.e. taxa are information artifacts, not physical objects.) Support for this kind of "punning" via metamodeling is a major feature of OWL 2 DL [4].
If this sort of thing interests you, definitely take a look into Wikidata [5]. The project will be a sea change for several key features in Wikipedia (e.g. infoboxes), and will likely be a main hub of the Semantic Web.
---
1. https://www.wikidata.org/wiki/Property:P31
2. https://www.wikidata.org/wiki/Help:Basic_membership_properti...
3. http://plato.stanford.edu/entries/types-tokens/
There's an RDF/OWL export of the Wikidata taxonomy available at [2] as wikidata-taxonomy.nt.gz, which can be explored with Semantic Web browsers like Protege [3].
Another fundamental relation -- "part of" (P361) [4] -- expresses mereological relationships. For (oversimplified) example: "iris part of eye", "eye part of head", "head part of body", etc. Both "subclass of" and "part of" are transitive.
A separate comment of mine in this discussion [5] describes how to traverse the "subclass of" tree in the Wikidata UI and a third-party tool called Wikidata Generic Tree. The same principle applies to the "part of" tree. The latter gets less attention, but is also quite interesting.
---
1. https://www.wikidata.org/wiki/Property:P279
2. http://tools.wmflabs.org/wikidata-exports/rdf/index.php?cont...
3. http://protege.stanford.edu/
* Go to a Wikidata item, e.g. "sailboat" [1]
* Click on the "subclass of" (P279) value or, if no such value exists, the "instance of" (P31) value
* Follow the "subclass of" chain up to "entity" [2]
You can do the reverse -- i.e. go down the classification tree -- with tools like Wikidata Generic Tree [3].
---
1. https://www.wikidata.org/wiki/Q1075310
2. https://www.wikidata.org/wiki/Q35120
3. http://tools.wmflabs.org/wikidata-todo/tree.html?q=Q35120&rp...
Wikidata is only for notable data, but the notability threshold is much lower than that for Wikipedia. The criteria for notability are described at [1]. For example, we might add items for all known pathogenic genetic variants, but likely would not have an item for the fire hydrant on your street.
For things not notable enough for Wikidata, interested users could install a local instance of Wikibase [2], the software that runs Wikidata. Wikidata editors and administrators determine what is notable, and have places like [3] to discuss questionable cases.
> Can data be permanently, unrecoverably deleted, or is it more like Wikipedia
Wikidata works like Wikipedia in that regard. Previous versions of a given item or property are almost always viewable (and recoverable) through the History tab [e.g. 4]. In extraordinary cases, like a vandal posting sensitive information about a person, data can be hidden from normal view and/or actually deleted.
> Is the plan to assign a unique integer to every unit that's ever been used?
Basically yes, to my understanding. How many units do you think exist? We already have items for many units, e.g. meter, micrometer, nanometer, foot, yard, bit, byte, , gigabyte, etc. I can see how this implementation might seem naive; e.g. perhaps we could represent one standard metric (like meter or byte) and handle factor conversions of scale (kilometer, millimeter, etc.) through some mechanism we don't have in place right now. Consider also asking about this on the "Contact the Development Team" page [5].
1. https://www.wikidata.org/wiki/Wikidata:Notability
3. https://www.wikidata.org/wiki/Wikidata:Requests_for_deletion...
4. https://www.wikidata.org/w/index.php?title=Q1&action=history
5. https://www.wikidata.org/wiki/Wikidata:Contact_the_developme...
Example queries:
* Politicians who died of cancer (of any type): https://query.wikidata.org/#PREFIX%20wikibase%3A%20%3Chttp%3...
* Who discovered the most planets? https://query.wikidata.org/#PREFIX%20wikibase%3A%20%3Chttp%3...
* Largest cities with a female mayor: http://query.wikidata.org/#PREFIX%20wikibase%3A%20%3Chttp%3A...
More Wikidata SPARQL query examples: https://www.mediawiki.org/wiki/Wikibase/Indexing/SPARQL_Quer....
I suspect larger applications of Wikidata in AI will follow. For example, as of 2010, IBM Watson acquired at least some of its content from DBpedia and YAGO [5], which ultimately derive much of their content from scraping Wikipedia's infoboxes and category system. Now, come 2015, Wikidata is supplying data for some Wikidata infoboxes, and the proportion of infoboxes that pull from structured data in Wikidata will increase over time. And I expect Wikipedia's category system will gradually be supplanted by Wikidata's more expressive property system over time.
Thus, I imagine Wikidata will form a semantic backbone for Q&A systems like Watson in the future.
The Wikidata development team's work is funded through donations by the Allen Institute of Artificial Intelligence, Google, the Gordon and Betty Moore Foundation, and Yandex [6]. So organizations with an interest in AI see potential in Wikidata.
1. https://github.com/Ladsgroup/Kian
2. http://ultimategerardm.blogspot.com/2015/09/wikidata-ten-que...
3. https://github.com/shilad/wikibrain
4. http://conservancy.umn.edu/bitstream/handle/11299/163269/Und...
5. http://www.aaai.org/Magazine/Watson/watson.php
6. http://cacm.acm.org/magazines/2014/10/178785-wikidata/fullte...
If we're talking all philosophy articles in all 291 Wikipedias [2], and (generously) assume that the average Wikipedia has 10% as many philosophy-related articles as English Wikipedia, that's 19,339 * 0.10 * 291 = 562,764 philosophy articles on all Wikipedias. That's 3 orders of magnitude more than SEP's 1,500 articles -- although that's not really comparing apples to apples as we're comparing many languages to one.
1. See Total x Total cell at lower right in matrix at https://en.wikipedia.org/wiki/Category:Philosophy_articles_b...
We're getting there! There are currently 793 philosophy articles [1] better than C-class in Wikipedia.
By quality:
* Featured articles: 41 [2]
* Featured lists: 4 [3]
* Good articles: 113 [4]
* B-class articles: 635 [5]
1. Summary table of philosophy articles by quality and importance: https://en.wikipedia.org/wiki/Category:Philosophy_articles_b...
2. https://tools.wmflabs.org/enwp10/cgi-bin/list2.fcgi?run=yes&...
3. https://tools.wmflabs.org/enwp10/cgi-bin/list2.fcgi?run=yes&...
4. https://tools.wmflabs.org/enwp10/cgi-bin/list2.fcgi?run=yes&...
5. https://tools.wmflabs.org/enwp10/cgi-bin/list2.fcgi?run=yes&...
The implementation [1] has various components including:
* Kartotherian [2]: a server capable of providing map tiles in vector (pbf) or raster (png) formats, as well as static map snapshots of any size for a given location.
* Tilerator [3]: a distributed backend tile generation service with a jobque
* A flexible sources [4] system to set up the needed storage and processing pipeline
1. https://www.mediawiki.org/wiki/Maps/Tile_server_implementati...
2. https://github.com/kartotherian/kartotherian/
There are actually already ways to browse Commons images on a map, but they need significant work. For example, the tile at [1] depicts an area with easily 20 geotagged Commons images, but, inexplicably, none of them are shown until you zoom in another level. Or, zoom out to see the entire state, only to see Massachusetts shown as completely lacking any geotagged Commons images [2].
There is a ton of potential for awesome applications involving geotagged images and geographic maps. I'm glad to see the Wikimedia Foundation stepping up its investment here.
1. https://tools.wmflabs.org/wiwosm/osm-on-ol/commons-on-osm.ph...
2. https://tools.wmflabs.org/wiwosm/osm-on-ol/commons-on-osm.ph...
This includes English! http://en.wikipedia.org/wiki/English-language_spelling_refor... covers several major successful (and unsuccessful) English spelling reform efforts since the 16th century.
The Atlantic article discusses the shenanigans of Belgian printers inserting unnecessary letters into words, changing e.g. "frend" to "friend". Spelling Reform 1 (SR1), a failed effort from the late '60s and '70s, would have addressed that and more.
Noah Webster -- of Merriam-Webster Dictionary fame -- spearheaded early American spelling reform. This is why American English writes "favorite" instead of "favourite", "center" instead of "centre", "catalog" instead of "catalogue", etc.
Encouraging contributions from females has probably been the Wikimedia Foundation's largest policy effort. Increasing Wikipedia's coverage of notable women is a common theme in edit-a-thons, events where Wikipedians get together at a library or cafe and edit Wikipedia together for a few hours. There is more to do, but these are effective and useful efforts.
I am not familiar enough with the editorial conflicts surrounding Gamergate to make an informed comment on Mark's post, but thought I would offer some broader context.
That said, while the San Francisco Wikidata page may currently be uglier than its Freebase counterpart, it is not slower. webpagetest.org has the Wikidata page fully loaded at 8.8 s and the Freebase page 11.2 s [2, 3]. And while Reasonator is certainly dog slow (21.2 s to fully load! [4]), its San Francisco page is much more polished than the Freebase's.
[1] http://www.wikidata.org/wiki/Wikidata:UI_redesign_input
[2] http://www.webpagetest.org/result/141218_DR_9W4/
- Total statements: 50,457,200
- Items with referenced statements: 8,188,516 (49.41%)
- Statements referenced to Wikipedia: 18,614,138 (36.89%)
- Statements referenced to other sources: 7,466,240 (14.80%)
"References" to Wikipedia are obviously frowned about, so the relevant datum here is that 14.80% of Wikidata's 50,457,200 statements have references.
I have no idea what that figure is for Freebase. Anyone know how to find what proportion of statements in Freebase are referenced?
The best way to contact me about this is to leave a note at https://www.wikidata.org/wiki/User_talk:Emw.
Wikidata RDF exports are made every two months or so from those dumps and are available at [2]. I imagine that frequency will pick up. You can generate your own RDF exports using the Wikidata Toolkit [3, 4].
[1] http://dumps.wikimedia.org/other/wikidata/
[2] http://tools.wmflabs.org/wikidata-exports/rdf/
From Denny Vrandecic, current Google researcher working on the Google Knowledge Graph, former project director of Wikidata [1]:
"Freebase has seen a huge amount of effort go into it since it went public in 2007. It makes a lot of sense to make the results of this work available to Wikidata. But knowing Wikidata and its community a bit, it is obvious that we can not and should not simply upload Freebase data to Wikidata: Wikidata would prefer the data to be referenced to external, primary sources.
"In order to do so, Google will soon start to work on an Open Source tool which will run on Wikimedia labs and which will allow Wikidata contributors to find references for a statement and then upload the statement and the reference to Wikidata. We will release several sets of Freebase data ready for consumption by this tool under a CC0 license. This tool should also work for statements already in Wikidata without sufficient references, or for other datasets, like DBpedia and other machine extraction efforts, etc.
"To make sure we get it right, we invite you to participate in the design and development of this tool here:
https://www.wikidata.org/wiki/Wikidata:Primary_sources_tool "
[1] https://lists.wikimedia.org/pipermail/wikidata-l/2014-Decemb...
Topics / items:
- Freebase: 46,476,860 [1]
- Wikidata: 12,921,731 [2]
Facts / claims:
- Freebase: 2,696,141,481 [1]
- Wikidata: 50,457,200 as of 2014-11-10 [3]
Instances of person / human:
- Freebase: 3,391,533 [4]
- Wikidata: 2,638,614 [5]
License for data
- Freebase: CC-BY [6]
- Wikidata: CC0 [7]
Data on Paul Graham:
- Freebase: http://www.freebase.com/m/017cm9
- Wikidata: https://www.wikidata.org/wiki/Q92650
Data on San Francisco:
- Freebase: http://www.freebase.com/m/0d6lp
- Wikidata: https://www.wikidata.org/wiki/Q62
Data on Python:
- Freebase: http://www.freebase.com/m/05z1_
- Wikidata: https://www.wikidata.org/wiki/Q28865
Data on APOE / Apolipoprotein E:
- Freebase: http://www.freebase.com/m/0byv2v
- Wikidata: https://www.wikidata.org/wiki/Q14890468 (APOE), https://www.wikidata.org/wiki/Q424728 (Apolipoprotein E)
See [8] and [9] for an introduction to Wikidata. I have no notable experience with Freebase, but I've been contributing to Wikidata for about 2 years and would be happy to answer any questions I can.
[3] http://tools.wmflabs.org/wikidata-todo/stats.php
[4] http://www.freebase.com/people/person?instances
[5] http://tools.wmflabs.org/autolist /autolist1.html?q=claim[31:5]
[6] http://www.freebase.com/policies/tos
[7] See bottom of [2]
[8] Up and running with Wikidata: http://www.slideshare.net/_emw/up-and-running-with-wikidata
[9] Introducing Wikidata to the Linked Data Web: http://korrekt.org/papers/Wikidata-RDF-export-2014.pdf
Sociological issues among established Wikipedia editors do contribute do lower retention of desirable new editors, but like Animats I suspect Wikipedia's comprehensive coverage is a significant factor (perhaps more so). It's simply less appealing to do curatorial work than to make large additions to major articles like "Genetics" or "France".
Which factor contributes more to Wikipedia's active editors drop-off, sociological issues or encyclopedic completeness? Has anyone looked at that? I'd be interested to see an analysis of how the trends in the notorious graph at http://commons.wikimedia.org/wiki/File:Enwp_retention_vs_act... correlate with word count and reference count in important articles.
Wikipedia has a set of 1000 "vital articles" that would be a major asset in rebuilding civilization.[1] Of course, the internal details of the encyclopedia's organization, its Talk pages, esoteric editorial conflicts, Pokemon pages, etc. are irrelevant and would be omitted for a version of the encyclopedia put onto truly durable, readable storage media.
A few years ago, while talking with a Wikipedian about the Long Now Foundation,[2] he said something that has stuck with me:
"Who says it's not in the Wikimedia Foundation's mandate to commission certain articles to be etched in stone?"
I would love to see that goal advertized in one of the WMF's annual fundraising banners. It would be so much more awesome than investing in improved default typography (not that the exquisite new typography [3] isn't nice).
Naturally, there would need to be several language versions and many copies of the stone-etched Wikipedia articles. A more durable storage medium than stone would also be a good idea. And having the articles assumes people in the post-apocalyptic world could read an existing major language to a reasonable degree. Having a gradated table of contents, such that more fundamental and simple concepts came before sophisticated ones (e.g. 'sanitation' before 'vaccine', 'arithmetic' before 'calculus') would also obviously be helpful.
Most of the vital articles that would help in rebuilding civilization [1] are in bad condition. Maybe this discussion will inspire someone to push one of those articles to a better state.
[1] https://en.wikipedia.org/wiki/Wikipedia:Vital_articles. The vital articles are divided into 11 categories. Of these, the following 300 or so articles from 6 of those categories would probably be the most helpful in rebuilding civilization:
- Everyday life: clothing, cooking, job, writing, and various major languages.
- Geography: city, Beijing, Cairo, Delhi, Jakarta, London, Mexico City, Moscow, New York City, Rome, Sao Paulo, Singapore, country, Africa, Antarctica, Asia, Europe, Oceania, North America, South America, Arctic, Middle East, sea, Atlantic Ocean, Indian Ocean, Pacific Ocean, lake, river, land, desert, forest, glacier, mountain, Alps, Andes, Himalayas, Rocky Mountains
- Health and medicine: common cold, influenza, pneumonia, burn, tuberculosis, ageing, antibiotic, vaccine, sanitation
- Science: scientific method, nature, astronomy, asteroid, comet, orbit, Solar System, Sun, Earth, Moon, Universe, biology, life, death, extinction, anatomy, brain, blood, heart, eye, immune system, liver, lung, muscle, skin, skeleton, reproduction, male, female, pregnancy, sex, botany, cell, ecology, evolution, genetics, heredity, organism and the 21 articles within (e.g. bird, insect, animal, human, dog, horse, plant, seed), virus, chemistry, alloy, bronze, steel, atom, chemical bond, salt, water, chemical element, carbon, iron, oxygen, silicon, chemical reaction, metal, mineral, molecule, periodic table, history of the Earth, geology, earthquake, rock, volcano, climate, cloud, flood, global warming, rain, season, weather, wind, map, soil, physics, electron, proton, photon, classical mechanics, energy, force, gravitation, heat, temperature, light, magnet, mass, matter, measurement, sound, thermodynamics, time, day, year
- Technology: all 109 articles
- Mathematics: arithmetic, addition, multiplication, fraction, logarithm, combinatorics, geometry, angle, trigonometry, area, line, point, circle, triangle, sphere, volume, function, calculus, number, 0, pi, e, integer, probability, statistics
[2] https://en.wikipedia.org/wiki/Long_Now_Foundation
[3] https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...
There are also many, many gems among the 20+ million other free images at https://commons.wikimedia.org.
The site has an extensive category system that complements keyword-based search, e.g. https://commons.wikimedia.org/wiki/Category:Fruit.
There's a fair amount of debate over that property. Are those current high level types (person, place, work, event, organization, term) a good fit for a knowledgebase that aims to structure all knowledge and not just library holdings? Does classifying subjects like inertia, DNA, Alzheimer's disease, dog, etc. as simply "terms" make sense?
More reading related to Wikidata, ontology and types: https://blog.wikimedia.de/2013/02/22/restricting-the-world/.