View source for Research:Topical coverage of Edit Wars

{{Tracked|T171249}}
{{Research_project
<!-- When was the project page created? Do not edit this section, it will be autogenerated-->
 | created = 23:06, 16 September 2017 (UTC)
 
<!-- Who should be contacted about this project? e.g. {{Investigator|Aaron Halfaker|Wikimedia Foundation}} You can link to your username if you have one:  {{Investigator|[[User:EpochFail|Aaron Halfaker]]|Wikimedia Foundation}}-->
 | contact = {{Investigator|[[User:Diego (WMF)|Diego Sáez]]|Wikimedia Foundation}}
 
<!-- Who else is working on the project? You can add as many {{Investigator}} entries as needed -->
 | collaborators = {{Investigator|[[User:Dario (WMF)|Dario Taraborelli]]|Wikimedia Foundation}} {{Investigator|[[User:Jmorgan (WMF)|Jonathan Morgan]]|Wikimedia Foundation}}
 
<!-- What year/month did work start? ... or when will it start? -->
 | start_year = 2017
 | start_month = September
 
<!-- What year/month did work end? ... or when will it end? -->
 | end_year =  2017
 | end_month = December
 
<!-- Project status: draft -> proposed -> planned -> active -> completed -->
 | status = active
 
<!-- Tags represent metadata about the project.  This section provides templates for all available tags.  -->
 | tags = 
 
<!-- These tags are only used for projects formally supported by the Wikimedia Foundation under our Open Access policy. Your contact at WMF will take care of filling this out.
  {{Research_project/Tags/WMF supported
   | contact = 
   | api = 
   | data_collection = 
   | endorsement = 
   | funding = 
   | private_data = 
   | subject_recruitment =
  }}
-->

<!-- This project's manuscript is published as open-access 
  {{Research_project/Tags/Open access
   | url = 
   | label = 
   | ref = 
  }}
-->
 
<!-- This project's code is open-source
  {{Research_project/Tags/Open source
   | url = 
   | label = 
  }}
-->
 
<!-- This project's data is available for download and reuse.
  {{Research_project/Tags/Open data
   | url = 
   | label = 
  }}
-->

}} <!-- End of Research project template -->


== Introduction == 
[[en:Wikipedia:Edit_warring |Edit wars ]] in Wikipedia has been largely studied. An edit war is usually consider to be the consequence of different opinions about an specific topic, between two or more users. Naturally, users with different political views might have different opinions on many articles related with politics, and these differences can scale in a multi-article edit war. These actions can be consider toxic, but is not necessarily a stalking behavior. However, in the case (if exist) that edit wars start happening across multiple topics, this can be an indicator of a person-centered attack (instead of topic-centered), that might  be categorized as wikihounding.

Taking the advantage that edit wars can be detected in content agnostic approach (without analyzing the text), we propose to study the topical span of those wars, characterizing usual and unusual (potentially toxic) behaviors.  

The main tasks to develop such model are:

* Generate a representative dataset of edit war in Wikipedia. 
* Detect pairs or groups of users involved in more than X (define X part of the study) controversies.
* Define and implement a robust topic model.
** Define a distance metric for topics (eg: Geography is N steps far from Politics, and M steps far from Sports.)
* Apply an outlier detection mechanism to find potential cases of harassment .
* Our approach focus in obtain good [[en:Precision_and_recall| recall]] <ref> [[en:Precision_and_recall | Precision_and_recall]]</ref>.

== Methodology ==
* Define a topic model, that allows to measure topical distance between Wikipedia pages.
* Model user behavior according to the topics that she/he covers (edit on) and the amount of [[en:Wikipedia:Reverting|reverts]] that she/he does.
* Compute the probability of pair of users of co-edit a page, and the probability that this co-edition is a revert.
* Based on the aforementioned co-editions' probability, identify [[en:Anomaly_detection | anomalous behavior ]] that are potentially related with stalking or wikihounding behavior, and do '''manual evaluation'''.

=== Topic Model ===

==== Pages to Topic ====
* We use [[en:Wikipedia:WikiProject |wikiprojects]] as proxy for topic assignment.
* Mapping from pages to Wikiprojects can be obtained with this  query: https://quarry.wmflabs.org/query/23214
* Therefore, we consider each wikiproject as one topic, however, usually each page belongs two more than one topic.
* Each page can belong to 0, 1 or several Wikiprojects. For example the page about [[en:Diego_Maradona]] is the interest of [[en:Wikipedia:WikiProject_Biography/Sports_and_games]], [[en:Wikipedia:WikiProject_Football]] and [[en:Wikipedia:WikiProject_Argentina]]


==== Topic Distance ====
* [[en:Wikipedia:WikiProject |Wikiprojects]] can be represented as a graph (we use this library https://github.com/wiki-ai/drafttopic/tree/master/drafttopic/utilities).
* Then, giving that each page can belong to more than one category, we define the distance between two pages as the minimum [[en:Shortest_path_problem|shortest path]] <ref> [[en:Shortest_path_problem|shortest path]]</ref> among all pairs of nodes on the Wikiprojects graph.
** Example: Given a page X and Y, with X in Wikiprojects ''a'' and ''b'' and Y in Wikprojects ''c,'' and ''d''. We  compute the lenght of shortest path between ''(a,c), (a,d), (b,c)'' and ''(b,d)'', and return the minimum value among these results. 

=== User Behavior ===

==== Topical Coverage ====
We use two metrics to characterize the topical focus/coverage of a given user, one focused on the probability of editing on a giving topic, and other focused on the ''user topical stability'':

*'''Probability of U on editing on topic ''T'' (u<sub>p<sub>T</sub></sub> ''': The probability of user ''U'' editing topic ''T'', is the her total number of editions on ''T'' divided by total number of editions in the set. Example: in our dataset T= 40,833,278 (total number of revisions), consider a user ''U'', with 3 editions on topic Sports, his probability of editing on topic sports p<sub>sports</sub> is 3/T = 7.3e-08.

*'''Topical stability (u<sub>s</sub>)''': For each user U, we obtain the '''distance for his next edition '''. For example, given a user ''U'', doing three editions, the first one in the topic 'Sports', the second one in the same topic, and the third one in the topic biology (with distance 4 from Sports), the probability of user U to edit with topical distance 0, is 2/3, with distance 4 is 1/3, and 0 for the other distances. This metric gives an idea of user stability in terms of topics. 
 
==== ''Reverting'' Behavior ====

* For user we compute portion of reverts (within the dataset) compared with his/her total amount editions. Considering that our dataset just contains editions done by users 10 or more editions, we are just considering the reverts among these users. Later, in the results section we discuss the implications of such constrain. 

=== Probability of Reverting Other User ===

* TODO

== Dataset, Tools and Resources ==

=== Revisions ===
* We consider all the users with more than 10 revisions in the selected period (1/Jan/2017 to 16/Nov/2017)
* All the revisions from those users can be obtained from this query: https://quarry.wmflabs.org/query/23424 
* Pages to Wikiprojects: https://quarry.wmflabs.org/query/23214


=== Wikiprojects Graph ===

* Using the Wikiproject graph generated from  [https://github.com/wiki-ai/drafttopic/tree/master/drafttopic/utilities this script]<ref> [https://github.com/wiki-ai/drafttopic/tree/master/drafttopic/utilities script for parsing wikiprojects graph]</ref>

=== Code ===
* Find all the code used in this study here: 

=== Terminology / Conventions ===
* We use edition as synonym of a Wikipedia revision.
* We use Wikiproject as proxy for Topics 

== Results ==
(Work in progress... ) 

=== Characterization on Topical Distance in Multipage editwars ===
In order to have notion of how frequent are reverts and editwars across multiple topics, we consider all pair of users with U, V, where U has reverted V more than 2 times, and compute the topical distance between all the pairs of pages reverted, next we compute the [[en:Mode_(statistics) |mode]]  (most frequent value) for each pairs of users, and report the frequency of those values. As expected, most of wars (71%) focus on one page, 22% in the same topic but different page. The remaining 7% are a cross-topic reverts, reinforcing our intuition that cross-topic edir wars are rare. 

[[File:Characterization Topical Distance in Multipage Editwars.png|300px|thumb|Characterization Topical Distance in Multipage Editwars]]
{| class="wikitable"
|-
! Distance !! %
|-
| -1 || 0.7107
|-
| 0 || 0.2297
|-
| 4 || 0.0143
|-
| 5 || 0.0121
|-
| 2 || 0.0103
|-
| 3 || 0.0074
|-
| 6 || 0.0056
|-
| 7 || 0.0034
|-
| 8 || 0.0027
|-
| 1 || 0.0016
|-
| 9 || 0.0012
|-
| 10 || 0.0007
|-
| 11 || 0.0002
|-
| 12 || 0.0001
|}
== Future Work ==
* Improve mapping system from pages to Wikiprojects: The aforementioned [query https://quarry.wmflabs.org/query/23214] returns around 22% of pages matching with no wikiproject. However, manually reviewing we found cases where the Wikiproject is not correctly (?) assigned as category in the Talk pages (example [[en:Classon_Avenue_(IND_Crosstown_Line) ]] belongs to Wikiproject  Trains, but that project is not listed as category)
== Timeline ==
Q1, Q2

== References ==
<references />

== Acknowledgements  ==
* [[User:Halfak (WMF)]]
* [[User:Sumit.iitp]]
* [[User:Jmorgan (WMF)]]

[[Category:Wikimedia Research project]]