Wikidata:Requests for permissions/Bot/WikiGrok
- The following discussion is closed. Please do not modify it. Subsequent comments should be made in a new section. A summary of the conclusions reached follows.
- Approved. We need to do smth with this request (about 1k items), and it looks like the majority of commenters agree that the testing is acceptable even though may be not optimal. Ryan, please take the comments into account, since for the large scale deployment we need to find a way to comply with them. --Ymblanter (talk) 08:16, 14 April 2015 (UTC)[reply]
WikiGrok (talk • contribs • new items • new lexemes • SUL • Block log • User rights log • User rights • xtools)
Operator: Bmansurov (WMF) (talk • contribs • logs) or Ryan Kaldari (WMF) (talk • contribs • logs)
Task/s: Add aggregated claim data from WikiGrok
Code: Bot code is here. Extension code is here.
Function details: As I've mentioned on Wikidata project chat, the WMF mobile web team has been experimenting with micro-contribution interfaces for adding metadata to Wikidata from within Wikipedia articles. These experimental interfaces are very similar to Magnus's Wikidata game, but instead of posting the results to Wikidata immediately, we have been collecting the results in a database so that they can be aggregated for better accuracy. Now that we have collected a large number of responses, we would like to try posting some of the aggregated data to Wikidata (less than 1000 edits) and get community feedback on the quality and usefulness of the data. This will help us to tune the feature in preparation for continuous larger-scale use in the future (which is not part of this bot request). You can see some of the results from our tests so far at meta:Research:WikiGrok. You can also view a sample of the first 100 edits that would be made by this bot. These edits will be to add claims of the following types:
- occupation: author
- occupation: film actor
- occupation: television actor
- instance of: live album
- instance of: studio album
In addition, in the cases where we are adding "instance of: live album" or "instance of: studio album", we will delete any existing "instance of: album" claims (as discussed at Project Chat). We will also avoid creating any duplicates of existing claims. --Ryan Kaldari (WMF) (talk) 23:28, 16 March 2015 (UTC)[reply]
For reference:
- Wikidata:Project_chat/Archive/2014/12#Experimental Wikidata feature coming soon to English Wikipedia Mobile site
- meta:Research:WikiGrok with a "read more" link to nowhere/selfredirect
- meta:Research:WikiGrok/Test3
- mw:Extension:MobileFrontend/WikiGrok#Claim suggestions
- mw:Extension:MobileFrontend/WikiGrok/Claim suggestions
- mw:Extension:MobileFrontend/WikiGrok#Pushing_responses_to_Wikidata
- example: https://en.m.wikipedia.org/wiki/Sabrina%20Sabrok?wikigrokversion=B
Some comments:
- Scale? "Now that we have collected a large number of responses" - You give no number of claims to be pushed to wikidata except "100 claims (as a test run)". Are the 100 test edits the selected best-of a "large number of responses"? Or a random sample? How many would follow after the test? ([1] 7199 unique pages had at least one version (a) played on them / This number is 9013 for version (b)...?)
- We currently have ~36,000 responses. The vast majority of those are not yet usable since they have not been corroborated by multiple people. Depending on how we do the aggregation, we could be looking at anywhere from about ~200 edits to ~1000 edits total for the current data set. That number depends on what values we set for the two threshold variables (number of responses for the item, and percentage agreement), which in turn depends on the feedback we get regarding the first 100 edits. The first 100 edits will be a random selection using a relatively conservative set of threshold values: >=5 responses, >=80% agreement. Those threshold values give us 602 vetted claims total. If the community was satisfied that the first 100 edits are high quality, we would then ask to post the remaining 502. If the community was not satisfied that the first 100 edits were high quality, we would ask to try again with a more conservative set of threshold values. We can create a separate bot request for potential higher-volume use in the future (which may or may not ever happen). Ryan Kaldari (WMF) (talk) 23:42, 18 March 2015 (UTC)[reply]
- References? No references afaics. Will the "continuous larger-scale use in the future" also push data without any source into Wikidata? Help:Sources: "The majority of statements on Wikidata should be verifiable insofar as they are supported by referenceable sources of information such as a book, scientific publication, or newspaper article."
- It is correct that none of the edits will include references. We evaluated methods for supporting the addition of references from the interface, but we were not able to come up with any practical solutions for a small mobile interface (other than using "XX Wikipedia" which isn't a real reference). I know this isn't ideal and we are interested in hearing the community's feedback on this issue. Ryan Kaldari (WMF) (talk) 18:36, 17 March 2015 (UTC)[reply]
- Claims generation? Wikipedia readers are asked to decide Yes/No/Don't know to proposed claims (for wikidata) on the Wikipedia article page. The proposed claims seem to come from wdq.wmflabs.org. Just like the Wikidata Game https://tools.wmflabs.org/wikidata-game/#mode=occupation (where users need to log in)?
- The initial prototype of WikiGrok used the same engine as Wikidata Game for generating potential claims, but we now have our own engine that is completely separate and does not rely on Tool Labs. Ryan Kaldari (WMF) (talk) 18:36, 17 March 2015 (UTC)[reply]
- Data quality? mw:Extension:MobileFrontend/WikiGrok#Pushing_responses_to_Wikidata: "Before we start pushing response data to Wikidata, we will be analyzing the quality of the responses along different axes: logged-in vs. anon, high edit count vs. low edit count, version A interface vs. version B interface, etc. We will then use that data to figure out how to maximize the quality of the data we push to Wikidata via a scoring system. For example, we could implement an algorithm that selects responses to push to Wikidata as so:(...) If number of responses > 1, and composite response score > 65%, then push response to Wikidata." Where is this analysis? Is your "aggregated data" from WikiGrok indeed from >1 responses?
- I haven't been that involved in the analytics aspect of the project, but here's the gist of what I know so far... First, we don't have a ton of data yet, so our conclusions are not bullet-proof. The results of this bot run will be an important part of the over-all analysis of how or if we move forward with the feature. Our existing analytics data is a mix of comparison with existing Wikidata claims and hand-checking (neither of which are perfect). So far, we have been pleasantly surprised by the overall quality of the results. Especially surprising is that we haven't seen any real difference in response quality from logged-in vs. anon users. I'll ping our analytics person and ask them to make sure that all our results so far are posted on the test pages. Ryan Kaldari (WMF) (talk) 18:36, 17 March 2015 (UTC)[reply]
mw:Extension:MobileFrontend/WikiGrok#Response_Analysis gives curious definitions:
- "Quality: For the questions we do not know the ground truth an aggregated response is considered high quality if the submitted response to WikiData is not reverted two weeks after the data is written to WikiData. The aggregated response is considered low quality otherwise."
- So, someone, supposedly a human someone, is supposed to check all the WikiGrok bot edits for accuracy in two weeks, but if nobody cares to clean up after the bot, then the WikiGrok data is considered "high quality"... uhm, totally quality, sure ^^. But there is still the "ground truth", wow, what is that?
- "Ground truth" is based on comparison with existing Wikidata claims. We fully realize this is an imperfect measure, which is why we are also doing analysis based on hand-coding of results (at a small scale), and analysis based on test submissions to Wikidata. We will be asking the Wikidata community on Project Chat to scrutinize all the edits from this bot, so we're hoping that at least a significant percentage of the edits will be looked at within 2 weeks. If you feel like that isn't realistic, let us know. Ryan Kaldari (WMF) (talk) 18:36, 17 March 2015 (UTC)[reply]
- So, someone, supposedly a human someone, is supposed to check all the WikiGrok bot edits for accuracy in two weeks, but if nobody cares to clean up after the bot, then the WikiGrok data is considered "high quality"... uhm, totally quality, sure ^^. But there is still the "ground truth", wow, what is that?
- "Percentage agreement with WikiData: For each question, percentage agreement with WikiData is the fraction of responses for that question that match the ground turth as available in WikiData. This measure is defined only when the ground truth exists."
- So, Wikidata = ground truth... (or "turth" ;-) uhm. Do you check where this "wikidata turth" came from? Via Widar from the same Wikidata Game? Or via bot imported from person data at (italian?) Wikipedia? Or manual edits by registered users? Or: Doesn't mattter because if it's on wikidata it's true?^^
- No, we do not check where the existing claims come from and you're right that there's a chance our claims will actually be more reliable than the existing claims. We kind of have a chicken-and-egg problem in this regard. We've discussed the idea of just using WikiGrok to verify existing Wikidata claims, rather than creating new claims, and I would be interested to hear your thoughts on this idea. Ryan Kaldari (WMF) (talk) 18:36, 17 March 2015 (UTC)[reply]
- So, Wikidata = ground truth... (or "turth" ;-) uhm. Do you check where this "wikidata turth" came from? Via Widar from the same Wikidata Game? Or via bot imported from person data at (italian?) Wikipedia? Or manual edits by registered users? Or: Doesn't mattter because if it's on wikidata it's true?^^
I'm starting to wonder if it wouldn't be better to look at WikiGrok data disagreeing with "wikidata turth". Is it available? --Atlasowa (talk) 15:07, 17 March 2015 (UTC)[reply]
- The existing WikiGrok data isn't currently publicly available, but I don't see why it couldn't be. I'll ask the project manager if it would be feasible to post the data somewhere. Thanks for your detailed response to the bot request. We are very interested in community feedback on this feature and want to make sure that it is something that the Wikidata community actually thinks is useful to the project. I know there are mixed feeling about the impact of tools like Widar and Wikidata Game, but we are hoping to improve on those models and strike a good balance of volume and quality. One advantage that we have over those tools is that we have a huge audience (all visitors to mobile Wikipedia), so we can potentially have both higher quality (through aggregation) and higher volume (in the long run). The feature is still considered an experiment, however, and will be killed if it doesn't prove to be useful. Ryan Kaldari (WMF) (talk) 18:36, 17 March 2015 (UTC)[reply]
- @Atlasowa: I've posted an anonymized dump of the existing test data on github. Ryan Kaldari (WMF) (talk) 22:14, 30 March 2015 (UTC)[reply]
Thanks for your response, Ryan Kaldari (WMF). And thank you especially for being so upfront about potential problems. BTW, i primarily selfidentify as a Wikipedian. Where do i start?
- "We've discussed the idea of just using WikiGrok to verify existing Wikidata claims, rather than creating new claims, and I would be interested to hear your thoughts on this idea." Yes, sounds good, but using WikiGrok to "verify"? If we have no sources supporting the WikiGrok correction, we just have 2 contradicting claims: 1 old wikidata "truth", 1 new WikiGrok claim. To decide for the one or the other, someone has to investigate the claim. And if no reference is added, this investigation will have to start anew everytime: when this claim is changed next (vandalised?), or appears in a list of mismatches from cross-checking databases. And this investigation again and again and again.
- @Atlasowa: That's true, and one of the reasons we've been reluctant to pursue changing or deleting existing claims. We mainly just want to help with filling in the low-hanging fruit of missing claims that can get unanimous or near-unanimous support for being added. Like the early days of Wikipedia, most of the current work on Wikidata is simply around filling in "common knowledge" (which, of course, is imperfect). Once there is a broad coverage of "common knowledge", I'm sure there will be more emphasis on adding references, qualifiers, ranks, and settling disputed information. As a Wikipedian myself it took me a while to get used to this difference, but the Wikidata community as a whole seems to be OK with it. I've added over 1000 claims myself with my Wikidata community account, and so far none of them have been reverted for lacking references. Ryan Kaldari (WMF) (talk) 23:24, 19 March 2015 (UTC)[reply]
- "We will be asking the Wikidata community on Project Chat to scrutinize all the edits from this bot, so we're hoping that at least a significant percentage of the edits will be looked at within 2 weeks. If you feel like that isn't realistic, let us know." Asking at a prominent place for handcoding a list of 100 edits is a completely different thing than having a bot do 100 live edits and expecting "someone" to check this in 2 weeks. The last feels to me like throwing a handful of sand in the ocean. The first seems realistic, for a couple of times, at that scale (not "potential higher-volume use in the future").
- That sounds like a decent idea, although a bit of a departure from the normal bot approval process. Is there really that much of a difference between asking people to look at a list of 100 possible changes and 100 actual changes? The advantage of using actual changes is that we can automatically measure how many of those 100 claims still exist after 2 weeks (or whatever time-frame). Analyzing a haphazard discussion of an on-wiki list is much less conducive to generating quantifiable results. Also, keep in mind that all of the edit summaries will link to an FAQ page where people can give more detailed feedback. FWIW, we've done some limited hand-coding of the results ourselves and are pretty happy with the results. If anything, they should at least be more reliable than the Wikidata game. Ryan Kaldari (WMF) (talk) 23:24, 19 March 2015 (UTC)[reply]
- I went ahead and posted the first 100 potential edits: Wikidata:Requests for permissions/Bot/WikiGrok/First edits. We ended up deciding on a slightly more conservative set of threshold values to start with, specifically 10 responses instead of 5. If those look good, we can always lower it for future test pushes. Ryan Kaldari (WMF) (talk) 22:23, 20 March 2015 (UTC)[reply]
- "We evaluated methods for supporting the addition of references from the interface, but we were not able to come up with any practical solutions for a small mobile interface (other than using "XX Wikipedia" which isn't a real reference). I know this isn't ideal and we are interested in hearing the community's feedback on this issue." This is really the most important issue with wikidata imho. Unless easy referencing can be fixed, wikidata quality will only become worse. See my practical comments at Wikidata:Referencing improvements input and Wikidata talk:Primary sources tool. In terms of interface, i highly recommend to have a look at the tool CitationHunt, which was built and presented a week ago at en:Wikipedia:Village_pump_(idea_lab)#CitationHunt, very concise and with topic selection for users by category.
--Atlasowa (talk) 09:42, 19 March 2015 (UTC)[reply]
- I took a look at CitationHunt. It looks like a great tool for desktop users, but not really that useful on mobile. The main challenges with adding references on mobile are that you usually need to have multiple tabs or windows open at once and creating references in the correct syntax typically requires presenting the user with a large complicated form to fill out. One of the ideas that we discussed for a future version of WikiGrok is an interface to browse the existing references in an article and choose one that applies. The main concerns with this type of feature were that (1) the added complexity to the workflow would deter people from using it, and (2) that people would just choose a random reference without checking that it actually supported the claim. We discussed this issue with the Wikidata development team as well and came to the conclusion that filling in missing claims with unreferenced information was better than having no information at all. I'm not 100% satisfied with that conclusion, but it seems to reflect the current practices on Wikidata (given that 90% of current claims have no references). BTW, I asked our analytics person to post more detailed analysis of our existing results, and she says she's going to do that soon. I also asked the PM about publicly posing our existing data (the ~36,000 responses). She said that was fine as long as the data is anonymized, so I'll post that data shortly. Ryan Kaldari (WMF) (talk) 23:24, 19 March 2015 (UTC)[reply]
- Oppose approval due to absence of references. I wouldn't let my auto mechanic use a pliers when a wrench is required. References are necessary. Any tool that cannot include references is the wrong tool for the job. Jc3s5h (talk) 17:54, 24 March 2015 (UTC)[reply]
- @Jc3s5h: Of all the bot requests this year that were for inserting claims, not a single one was even requested to use references before being approved. Why is this request being treated differently? There are no policies or guidelines stating that claims must have references (or that bots must use references) and the vast majority of current claim insertions (by both users and bots) do not include references. The purpose of this feature, like the Wikidata game, is to help fill in "common knowledge" on Wikidata. Unlike Wikidata game, however, we are actually making sure that the knowledge is "common" by requiring many people to agree on it first. Ryan Kaldari (WMF) (talk) 18:55, 24 March 2015 (UTC)[reply]
- That oppose seems to be weird giving the scope of this request and current practice.
- We need to be able to follow the flow of information. Please make some sort of edit summary that links to an information page about Wikigrok, link to the article people where were on when they were asked the question and include a link to the (two?) users who approved the claim. We'll all be SUL soon so you can probably just link to local usernames. Multichill (talk) 20:46, 24 March 2015 (UTC)[reply]
- @Multichill: I've created a task for linking to an information page from the edit summaries.[2] Linking to the users would be difficult since we're probably going to require at least 5 people to agree on any claim before submitting it. Even worse, most of those users are going to be IPv6 IP addresses. We have a plan to create a CheckUser style interface that will let you look up all the users that contributed to a specific WikiGrok revision, but we don't want to build that interface until we are sure that we want to actually make WikiGrok into a real feature (instead of just an experiment). This bot request is just for submitting the data from our previous test. It's a relatively small amount of data and quite unlikely to have been a vandalism target. (It's probably the most inefficient way you could possibly vandalize a Wikimedia project since you have to submit numerous false claims under different users.) Ryan Kaldari (WMF) (talk) 22:48, 25 March 2015 (UTC)[reply]
- If the user interface of wikigrok makes it clear that only common-knowledge information should be added, fine. Some bots insert data from reasonably reliable sources, so the indication that is also added that the information was imported from the reliable source is sufficient. Most of the bot edits I've looked at were for birth and death dates, and those were a disaster. The bots didn't know the difference between the Julian and Gregorian calendar. It was often difficult to investigate what the real birth date was because no citation was provided. Jc3s5h (talk) 12:52, 25 March 2015 (UTC)[reply]
- @Jc3s5h: Of all the bot requests this year that were for inserting claims, not a single one was even requested to use references before being approved. Why is this request being treated differently? There are no policies or guidelines stating that claims must have references (or that bots must use references) and the vast majority of current claim insertions (by both users and bots) do not include references. The purpose of this feature, like the Wikidata game, is to help fill in "common knowledge" on Wikidata. Unlike Wikidata game, however, we are actually making sure that the knowledge is "common" by requiring many people to agree on it first. Ryan Kaldari (WMF) (talk) 18:55, 24 March 2015 (UTC)[reply]
Just look at the user interface. The reader is at a Wikipedia article, WikiGrok pops up and asks "which of these tags describe <article person>?" The Wikipedia article is right there for him to read, above the task, and the proposed tags are actually harvested by Wikigrok from this same Wikipedia article. It is a suggestive question. No wonder that there is hardly any disagreement at all with "ground truth": meta:Research:WikiGrok/Test3#Number of responses vs. Percentage Agreement. And the reader is not asked to look for sources (or even look at Wikipedia refs) before he tags Paul Rand as a graphic designer. Think about it. I would really like to know: What would happen if WikiGrok would propose to readers to tag Sergej Rachmaninov as: homosexual, atheist, occupation cannibalism? Maybe some disagreements? What if you present the user a doctored Wikipedia article above this WikiGrok task, which reads "Sergej Rachmaninov was a famous russian cannibalist and a convicted homosexual and atheist." (no refs, no sources) How many will then tag Rachmaninov as: homosexual, atheist, occupation cannibalism? 80%? or less? *It must be true, it's on Wikipedia* I really wonder. --Atlasowa (talk) 15:22, 25 March 2015 (UTC)[reply]
@Ryan Kaldari (WMF), "There are no policies or guidelines stating that claims must have references" - Come on, i already quoted this to you on this very page,
- Help:Sources: "The majority of statements on Wikidata should be verifiable insofar as they are supported by referenceable sources of information such as a book, scientific publication, or newspaper article." See also
- Help:Statements#Add only verifiable information: "Wikidata is not a database that stores facts about the world, but a secondary knowledge base that collects and links to references to such knowledge. This means that Wikidata does not state what the population of Germany actually is; it simply provides the information on what the population of Germany is according to a specific source, such as the The World Factbook (Q11191) CIA World Factbook. As such, most statements should be verifiable by referenceable sources of information like a book, scientific publication, or newspaper article. In Wikidata, references are used to point to specific sources that back up the data provided in a statement."
- Re: "the Wikidata community as a whole seems to be OK with it. I've added over 1000 claims myself with my Wikidata community account, and so far none of them have been reverted for lacking references." Did it ever occur to you, that maybe there are not masses of wikidata users checking up on your edits and deciding not to revert you? But rather that almost nobody watches wikidata items? Not even the most obvious vandalism by the most vandalism-prone IP editors is patrolled at recent changes [3][4][5]. Don't even ask about patrolling widar edits. Or the bots that do 90% of wikidata editing. And did you notice that it is not easy to add references to wikidata, because there are no tools that make this easier for normal human beings. "The main challenges with adding references on mobile are that you usually need to have multiple tabs or windows open at once and creating references in the correct syntax typically requires presenting the user with a large complicated form to fill out." That would have really helped normal, non-mobile, desktop, wikitext editors, which are the majority of productive editors. --Atlasowa (talk) 16:34, 25 March 2015 (UTC)[reply]
- @Atlasowa: I'm aware that claims should be verifiable with references. I used to be an admin on Wikidata back in 2012 and helped write the guidelines. There is no requirement currently that claims must be referenced. Your suggestion that such claims should be reverted is not based in policy. If it were, I'm sure someone would have written a bot to delete all unreferenced claims from Wikidata (the vast majority of them), and tools like Widar, Autolist, and Wikidata Game would have been banned a long time ago. I agree that Wikidata needs better interfaces for adding references, but that is outside the scope of this bot permission request. This request is for inserting less than 1000 unreferenced claims (100 to start with), restricted to 5 specific types, which have achieved consensus from Wikipedia readers. It's scope is smaller than a lot of single Autolist actions (which don't require bot approval). Nothing in this bot request is related to cannibalism, homosexuality, or atheism. Yes, the accuracy of the claims will be influenced by the accuracy of Wikipedia, but that's why we are choosing relatively uncontroversial and easy to verify statements. The same can't be said for tools like Autolist which make thousands of edits based on nothing but Wikipedia category inclusion (which can often be quite subjective). Let's discuss the actual cases involved in this bot request rather than hypothetical cases. Ryan Kaldari (WMF) (talk) 18:01, 25 March 2015 (UTC)[reply]
@Multichill, Pasleim, Ricordisamoa, Ymblanter: I would like to ask for 2nd opinions on this bot request. So far the only concrete reason that has been raised for objecting to this request is that the edits will not be referenced, even though bots are not normally required to add references. Is that a valid reason to block the request? Are there other concerns that need to be addressed? Keep in mind that this bot request is to make less than 1000 edits total (the first 100 of which are shown here). Ryan Kaldari (WMF) (talk) 23:09, 30 March 2015 (UTC)[reply]
- Having references is best practice but not a requirement, and I think the bot should be approved.--Ymblanter (talk) 09:59, 31 March 2015 (UTC)[reply]
- @Ryan Kaldari (WMF): While I'm eager to support verification of sources by multiple users, I cannot abstain from raising some concerns about this particular task: it is certainly possible to have a bot import fake data by adding them to Wikipedia, but even imported from Wikimedia project (P143) can help tracking data down to their original sources; as far as I understand from the above description of the task, the bot would allow malicious manipulation of our knowledge base in an almost anonymous way; indeed references are not required, but approving a bot that does not even let users know where its statements come from would constitute a dangerous precedent. For these reasons, I'd rather have an additional level of verification which would require a (auto)confirmed user to ultimately approve every statement.
- @Ymblanter: Please do not approve controversial tasks unless all concerns have been dispelled.
- --Ricordisamoa 10:26, 31 March 2015 (UTC)[reply]
- @Ricordisamoa: Thanks for your feedback. Unfortunately, I don't think requiring an additional level of verification from (auto)confirmed users would work very well in practice. So far I haven't even been able to get people to verify the 100 claims at Wikidata:Requests for permissions/Bot/WikiGrok/First edits, despite asking on the Project Chat and a mailing list. I had to verify most of them myself and it still isn't finished. If ultimately we only trust (auto)confirmed users to vouch for these claims, we should just ask those users the questions to start with and not bother asking readers. The downside to that approach, however, is that we would lower the volume of participation and incoming data by an order of magnitude (or probably several orders of magnitude). We do have several ideas for addressing concerns with data quality and manipulation, such as:
- Creating an interface to view all the users who contributed to a particular claim being added. We would then include a link from each WikiGrok edit summary to the corresponding entry in this interface.
- Giving different users different weights. Autoconfirmed users could have a higher weight than other users. For example, it might take only 2 autoconfirmed users to verify a claim, but 10 non-autoconfirmed users.
- Testing users with claims we know are wrong. This would be quite tricky to implement (since it's always hard to know what data is accurate and what isn't), but theoretically we could present users with bogus data, and if they agree with it, they would no longer be able to use WikiGrok.
- What are your thoughts on those ideas?
- I would like to clarify, though, that this bot request is just for pushing up the existing data from our test run (less than 1000 edits). We want to push this data in order to get a better understanding of the quality of the data. That information, in turn, will help us decide how many layers of quality control make sense to put on the feature. The evaluation that we've done so far (see meta:Research:WikiGrok) leads us to believe that aggregation of responses is a powerful and effective mechanism for ensuring accuracy (at least for non-controversial questions). We now want to put some of that data into the wild and get the Wikidata community to look at it and see what the feedback is (and whether any of the claims are reverted). We will not be moving ahead with high-volume WikiGrok submissions without conducting further community consultation, adding additional safeguards, and creating a new bot permission request. Kaldari (talk) 19:39, 31 March 2015 (UTC)[reply]
- @Kaldari: Just add the last two users who approved in Wikigrok to the edit summary. That way we can hunt down vandalism or other problems. Multichill (talk) 20:24, 2 April 2015 (UTC)[reply]
- I'll definitely propose that to the team, although it isn't an option for the current bot request. The prototype version (which the existing data is from) used EventLogging for recording the data and we had to use user tokens instead of user names for the records (for privacy reasons that I won't get into here). The current WikiGrok software has a real storage backend and does record user names, so we can do that in the future. Kaldari (talk) 20:48, 2 April 2015 (UTC)[reply]
- @Kaldari: Just add the last two users who approved in Wikigrok to the edit summary. That way we can hunt down vandalism or other problems. Multichill (talk) 20:24, 2 April 2015 (UTC)[reply]
- @Ricordisamoa: Thanks for your feedback. Unfortunately, I don't think requiring an additional level of verification from (auto)confirmed users would work very well in practice. So far I haven't even been able to get people to verify the 100 claims at Wikidata:Requests for permissions/Bot/WikiGrok/First edits, despite asking on the Project Chat and a mailing list. I had to verify most of them myself and it still isn't finished. If ultimately we only trust (auto)confirmed users to vouch for these claims, we should just ask those users the questions to start with and not bother asking readers. The downside to that approach, however, is that we would lower the volume of participation and incoming data by an order of magnitude (or probably several orders of magnitude). We do have several ideas for addressing concerns with data quality and manipulation, such as:
@Atlasowa, Ricordisamoa, Ymblanter, Multichill: Any suggestions for how to move forward on this bot request? I've tried to address all of the concerns raised, even though most of them were more about WikiGrok as a feature than this specific bot request (posting the test data). In the time that I've spent on this bot request, I could have just made all the edits by hand. Should I consider the bot request failed and just throw away the test data? That seems like a waste of useful metadata to me. Ryan Kaldari (WMF) (talk) 21:21, 8 April 2015 (UTC)[reply]
- Hm. The data provided by WikiGrok is probably not much worse (and possibly better) than what bots, Widar users (including myself) and Wikidata gamers currently add daily (and on a larger scale) anyway. It introduces a new concept of aggregated contributions that can't be attributed to one specific creator or one specific source, not necessarily a problem but something untried before nonetheless. I guess though, looking at this discussion and on my contributions and watchlist, getting feedback from the community on the data quality in a useful timeframe won't work well especially when the numbers go up. --HHill (talk) 09:55, 9 April 2015 (UTC)[reply]
- Support I avoided commenting until now because the above discussion is tl;dr and a bit negative but I would hate to see the bot denied because a few people out-talked a silent majority (assuming that there is one.) It looks like the major objection is that there are no references. That never stopped anybody before. I've done most of the checking of the test edits and I can say that they are very good. Maybe there will be issues but I think that the concept is good and issues can be worked out as the work continues in kaizen fashion. --Haplology (talk) 09:58, 9 April 2015 (UTC)[reply]
- I think the bot can be trusted but its task seems not a step in the good direction: WD has to face a strong skepticism from WPs due to unreferenced data and the proposed task is just adding data without any possibility to track the origin of the data. But we can't deny the possibility to this bot adding verified data after letting some bots adding data from one unique WP article: is a WP article more reliable that a data verified by 5 persons ? This is the question. I am strongly against this kind of importation but I saw so many wild imports that at the end this one seems a little more reliable.
- IMHO this kind of data should not be used in a direct import but used in feeding a tool asking people to add the data themselves after validation. A person can always skip a data validation about a subject he doesn't know, a bot can't do that.
- Finally I conclude saying that there were a lot of freedom in WD at the beginning but now we have to know if we want to continue our lonely game of data import without reference which will lead to the reject by WP or if we accept to change our mind and to work according to good referencing practice. Freebase is died and I think some persons don't understand that if we just do what Freebase was doing we will finish like this database. Snipre (talk) 13:05, 9 April 2015 (UTC)[reply]
- Support a 1000 pilot edits without a botflag so that we can see what we're dealing with. Multichill (talk) 18:14, 9 April 2015 (UTC)[reply]
- I do not dispute the quality of data, but there ought to be no "privacy reasons" in regard to references. Neither can I find any urgency to import the data in question, so I suggest that WikiGrok propose changes to be later accepted and carried out by real users via 'flood' accounts. Without that, I would lean towards Oppose. --Ricordisamoa 03:44, 10 April 2015 (UTC)[reply]
- @Ricordisamoa: There will not be any privacy reasons in the future, but for the test data we do not have the option of posting with usernames. The reason why is complicated, but I'll try to explain it. During our prototype testing, we were not only recording user's responses, but also their behavior on pages that included WikiGrok. For example, whether or not users scrolled down enough to see the WikiGrok interface, whether they clicked the initial button to participate, etc. This allowed us to analyze user engagement with the feature. Our privacy policy, however, does not allow us to associate reading/browsing behavior with usernames or IP addresses. It has to be anonymized (thus we used randomly generated user tokens instead). We could have theoretically kept the response data separate, but because the response data includes the title of the page and a timestamp, it would have been possible to still correlate the username with the reading/browsing behavior. We are finished with engagement testing, so this will not be an issue for future data (which will have a completely separate bot request). Kaldari (talk) 17:59, 13 April 2015 (UTC)[reply]
- I think what I would like to see would be an imported from Wikimedia project (P143) Wikigrok to track these first changes. Is that planned? --Izno (talk) 14:12, 10 April 2015 (UTC)[reply]
- Yes, we can use an "imported from Wikimedia project (P143) Wikipedia" claim for the changes. Kaldari (talk) 17:59, 13 April 2015 (UTC)[reply]
- "imported from Wikimedia project (P143) WikiGrok" might be better, as in case of errors we know who to blame ;) --Pasleim (talk) 19:35, 13 April 2015 (UTC)[reply]
- That was basically my comment. WikiGrok may itself be sourced from Wikipedia, but that's not knowledge that's as useful knowing that WikiGrok itself added the item. @Snipre: you have a preference, since you usually have a feeling about such? --Izno (talk) 19:41, 13 April 2015 (UTC)[reply]
- Izno Seems to be a good compromise. But I think the creators of thisapplication should go to a new direction. Snipre (talk) 23:43, 13 April 2015 (UTC)[reply]
- That was basically my comment. WikiGrok may itself be sourced from Wikipedia, but that's not knowledge that's as useful knowing that WikiGrok itself added the item. @Snipre: you have a preference, since you usually have a feeling about such? --Izno (talk) 19:41, 13 April 2015 (UTC)[reply]
- "imported from Wikimedia project (P143) WikiGrok" might be better, as in case of errors we know who to blame ;) --Pasleim (talk) 19:35, 13 April 2015 (UTC)[reply]
- Yes, we can use an "imported from Wikimedia project (P143) Wikipedia" claim for the changes. Kaldari (talk) 17:59, 13 April 2015 (UTC)[reply]
- Support references are not a requirement and we have other tools (e.g. The Game) which are also adding statements without references. --Pasleim (talk) 19:35, 13 April 2015 (UTC)[reply]