AI tools: A powerful new weapon to combat the misuse of statistics
Abstract
False information – whether mistakenly or intentionally distributed – threatens societies around the world, with dire consequences. When official statistics are misused or misrepresented, trust in this public good and the institutions that produce them are undermined. Two groups, fact checkers and statistical regulators, share common goals in combating this problem. Bolstering their work is a new offering of AI tools which help monitor vast volumes of media, identify important statements in public debate and inform users when statements they know to be wrong are repeated again. This article will cover the problem posed by false information, the common goals of fact checkers and statistical regulators, the development of AI tools and their impact.
1.Introduction
Disinformation, or the spreading of false or misleading information, is widely acknowledged as a threat to societies around the world. The 24 hour news cycle and the continued growth of social media platforms has made it easier than ever to spread disinformation quickly and to a large audience. This can have serious consequences, from causing public panic to influencing the outcomes of elections, harming health outcomes and fostering distrust in democracy.
Two groups share common goals when it comes to countering bad information: fact checkers and statistical regulators. Fact checkers evaluate the accuracy of claims made by individuals, organisations and political parties by verifying the sources and evidence behind their claims. Statistical regulators monitor and ensure the quality of statistical data used by government agencies, businesses and public figures.
Fact checkers and statistical regulators work independently from those who produce and use statistics to identify important errors and inaccuracies. Both organisations aim to ensure that information presented to the public is accurate, reliable and unbiased and regularly work in partnership to achieve this aim. Together, they play a crucial role in promoting transparency and accountability, and safeguarding the public’s trust in official statistics.
In the United Kingdom (UK), fact checking and campaigning charity Full Fact has verified statements in the public domain since 2009 [1]. In recent years they have invested in automated fact checking tools that are powered by artificial intelligence (AI). These tools allow users to identify important claims to verify at scale and alert users to repeated falsehoods. The tools have largely been used by fact checking organisations; however, it has become apparent that other users, such as statistical regulators, also stand to benefit from the advances in the space.
The Office for Statistics Regulation (OSR) is the independent regulatory arm of the United Kingdom’s Statistics Authority. It provides independent regulation of all official statistics, an essential public asset, produced in the UK. The OSR aims to enhance public confidence in statistics produced by the government by setting the standards they must meet in the Code of Practice for Statistics through the pillars of Trustworthiness, Quality and Value (TQV). Statistics should serve the public good, and the OSR challenges any misuse of statistics.
This article will provide an overview of the threats posed by disinformation and the responses by fact checking organisations and statistical regulators. Developments in AI fact checking tools will be discussed, as well as their impact on the two sectors. Examples provided by Full Fact and the OSR will illustrate the impact of the tools on combating the misuse of statistics in the UK.
2.Background
Every day in countries around the world, people make decisions both big and small. They decide where to live, which medicine to take and who to vote for. Their opinions on the state of their economies and the performance of governments are informed by public statements, and often, official statistics.
Disinformation has a destructive impact on individuals and communities, perpetuating hatred and division through deceptive or false claims. It can undermine democratic values by eroding trust in elected officials and political systems. It hinders informed decision-making by disrupting public understanding on crucial matters such as climate change, healthcare and public expenditure.
Fact checkers correct misunderstandings in public discourse and hold public figures accountable for what they say. These organisations operate in many different forms around the world. Some are independent charities or registered non-profit organisations [2], while others operate within traditional media outlets. The International Fact Checking Network (IFCN), started in 2015 at nonprofit media institute and newsroom Poynter, verifies fact checkers who uphold their code of principles [3]. However, many fact checking organisations operate without this accreditation.
The fact checking methodologies vary by organisation; however, there is a common approach. Organisations monitor public debate in a variety of sources including: online media, print media, broadcast media, social media and government publications. They identify claims – statements which can be verified as true or false – that need to be fact checked. Fact checkers will often reach out to a person who has made a claim to ask for their evidence. The next step is to review available information from a number of reliable sources. These can include academic journals, official statistics, government websites, universities and private research organisations. Fact checkers consult experts in the field to unpack the claim and the information they have gathered. Experts are often able to provide context, analysis and insight. The findings are written up, reviewed and edited before publication.
Fact checkers have been likened to “first responders” during times of information crises, as they are typically at the frontlines to identify false claims or misunderstanding [4]. National statistical regulators play a similar role.
In the UK, the OSR has a double pronged approach to building public confidence in statistics. First it focuses on how statistics are produced, independently reviewing whether official statistics comply with the Code of Practice for Statistics and show trustworthiness, quality and value. The organisation’s statistical regulators use a tailored assessment process to evaluate the extent to which statistics meet the standards set out in the Code. If, following this independent review, official statistics comply with the Code, they meet the legal requirement to be badged as National Statistics.
Second, the OSR also plays a vital responsive role in challenging the misuse of statistics. The organisation actively monitors and searches for potential misuse of statistics in the media, on social media and in places of high public influence, such as parliamentary debates. Cases of potential misuse can also be referred to the organisation by individuals who wish to raise a complaint. When a case of potential misuse is discovered, the organisation considers the appropriate response in order to ensure the accuracy and integrity of statistics in the public domain.
These two groups - fact checkers and statistical regulators - share a number of common goals. Both monitor the media to detect statistical errors and verify claims shared in the public domain. They also work to verify information and hold public figures accountable for the information they share. Together they educate the public about the misuse of statistics and encourage transparency in statistical data.
AI-powered tools have the potential to support the shared goals of fact checkers and statistical regulators. The tools can quickly and accurately analyse large volumes of data and identify potential errors, inconsistencies or misleading information.
3.AI tools
Support and funding for automated fact checking gained momentum in the mid-2010s. Fact checking organisations such as Full Fact and the Duke Reporters Lab received funding for projects from a number of donors, including Google, the Omidyar Foundation, the Open Society Foundations, the Knight Foundation, the Facebook Journalism Project and the Craig Newmark Foundation [5].
The financial support allowed them, and others, to invest in research and development of automated fact checking tools. There are two stages of the fact checking process where automated fact checking tools have shown particular potential. The impact of the tools has been shown to save users time and effort and allowing them to work at scale. These are in claim detection and claim matching.
3.1Finding claims
Claim detection – or finding statements to investigate and verify – is the first step in the fact checking process. There are two components to this stage. First the statement must be verifiable, i.e. it must be phrased in a way that allows its accuracy to be confirmed.
A statement such as “the United Kingdom is the best country in the world” cannot be verified as it is based on a personal opinion. Understandings and definitions of “best” will differ by person. In comparison the statement “the UK economy recorded the fastest growth in the G7” can be verified [6]. There are globally accepted methods to measure and compare the sizes of economies. There are multiple databases to consult and experts who can advise on the best way to compare measures.
Second, the statement must be “check-worthy”. These are statements that “the general public will be interested in learning about their veracity” [7]. As an example, Gou et al. describes the statement “over six million Americans had COVID-19 in January” as a check-worthy claim. In comparison, the statement “water is wet” is not. There will not be public interest in determining the veracity of such a statement.
Traditional approaches to fact checking claim identification include manually reviewing news media, social media, press releases and government publications. Some organisations also encourage members of the public to submit statements they believe need to be verified. “Low tech” approaches include setting keyword alerts, using free online tools such as Google Alerts. These solutions return varying results. They are human resource intensive as they require team members to be assigned to do manual monitoring. They can also be technologically blunt, as simple keyword searches will not return claims with different wording.
Automated fact checking tools assist users in this first stage of the fact checking process: finding statements to interrogate and verify. The issue of ranking or labelling claims by their check-worthiness is currently in development at Full Fact.
Full Fact AI’s tools monitor a variety of media sources including news websites, social media, live broadcasts and government publications. These sources are scraped on a daily basis and split down into individual sentences, which is the unit of analysis. These sentences then pass through a number of stages where they are labelled and enriched to make them more useful to the user.
Users are then shown sentences that are claims. These are sentences that contain an assertion that can be verified or fact checked. These claims are further labelled to assist users to monitor the debates or topics they are interested in using a schema developed by fact checkers and academics [8]. The eight claim type labels used in Full Fact AI’s claim detection tools include:
• Quantity: Claims about current values, changing quantities, comparisons of quantities and ranking of quantities. For example: “There are more people in work than there were before the pandemic” [9].
• Correlation/causation: Claims about correlation, claims about causation and claims about an absence of a link. For example: “Taking the contraceptive pill could ‘save your life’ as fewer women attempt suicide” [10].
• Prediction: Hypothetical statements and claims about the future. These claims are generally not verifiable.
• Personal experience: Claims about personal experiences, which are generally not verifiable with publicly available information. For example: “I can’t afford to save for my retirement.”
• Rules: Claims about public institutional procedures and rule changes. For example: “You don’t have to follow the government’s rules after being vaccinated” [11].
• Voting record: Claims about how a person, group or political party voted on a topic. For example: “Nine in 10 of NASUWT’s members voted in favour of strike action” [12].
• Support: Claims about stances on certain topics. For example: “There is no desire in Scotland to have membership of the EU” [13].
• Opinion: Claims about public opinion or surveys on an issue. For example: “One in ten young people plan never to get a job” [14].
Users are also able to filter the sentences by defining keywords, date ranges, topics and publications. The tools also allow users to look for claims made by entities, such as people, organisations and political parties. This labelling and filtering system allows a user to reduce hundreds of thousands of claims to a manageable amount for consideration.
The tools have radically scaled the amount of media sources that fact checkers are able to monitor. At Full Fact, claim detection capacity stood at around 100 claims each day. After the introduction of the tool the organisation is now able to consider 100,000 claims each day. In a recent partnership with Nigerian fact checkers ahead of the country’s 2023 presidential election, the Full Fact AI tools were able to provide an average of 40,000 claims for review each day from 80 media sources [15].
3.2Claim matching
The second area of development in AI fact checking tools is “claim matching”. Fact checkers, and other users, need to be informed when something they know to be false is repeated again. This allows them to consider the repeated falsehood and decide what action they should take. These actions could include publishing a new fact check, contacting the person who repeated the falsehood or filing a complaint with a regulator.
Traditional approaches to claim matching include manually monitoring and reviewing media, which is time consuming. Tools like Google Alerts can search daily for certain keywords. However, this will miss a significant number of claim matches as language is dynamic and there are many different ways to make the same claim. Lastly, submissions from the public may be useful but they are not reliable or consistent. AI-powered claim matching tools reduce the time, effort and constraints of these approaches.
Claims which are scrapped and labelled during the previous step of claim detection are compared to a dataset of previously fact checked or monitored claims. Full Fact AI’s alerts tool uses a BERT-style model to predict a match/no-match for sentences. This is combined with searching for keywords, estimates of semantic similarity and entity matching. Claims which meet a minimum matching threshold are shown to the user for review. Users decide how to act on these alerts.
For example, there was public confusion about changes to the UK’s Highway Code when it was updated in 2022 [16]. Full Fact’s fact checkers wrote about several of the changes, including reporting by a newspaper that “cyclists on either side of a vehicle have priority when cars are turning” [17]. Another newspaper reporting on the same section wrote: “Cyclists can pass you on the left as well as the right when you’re in a jam: Motorists need to keep their wits about them on congested routes, as the Highway Code update now says a cyclist is allowed to pass them when in slow-moving or stationary traffic both on the right and the left” [18].
Both of the sentences mention “cyclists” but the former refers to “a vehicle” and “cars”, while the latter refers to “motorists”. The former refers to “either side of a vehicle” and the latter uses “both on the right and the left”. These differences demonstrate how a simple search based on specific words would never find this match. Full Fact’s AI tools recognised these two sentences are a match despite being expressed in different words.
This tool allowed Full Fact to act on numerous repeats of a misleading claim made by then UK prime minister Boris Johnson on 24 November 2021. During Prime Minister’s Questions, he claimed that there are more people in work now than there were before the Covid-19 pandemic began.
According to Number 10, Johnson was referring to the number of UK workers on employer payrolls which was 29.3 million in October 2021, above the 29 million in February 2020, before the start of the pandemic. But this did not include everyone in work, as it excluded self-employed people.
Consulting data from the Office for National Statistics, which includes all people in paid work in the UK, showed that the figure was still below the level just prior to the pandemic. It had decreased from around 33 million between December 2019 and February 2020 to 32.5 million according to the latest figures available at the time (July to September 2021) [19].
After publishing the fact check, Full Fact contacted the Prime Minister’s office to request a correction regarding this claim. The organisation also wrote to the OSR in January 2022 to highlight the Prime Minister’s misuse of the data [20]. At the time of the correspondence, the AI-powered claim matching tools had alerted Full Fact to three more instances of Johnson repeating the incorrect claim on 1 December 2021 [21], 15 December 2021 [22] and 5 January 2022 [23].
The AI tool also identified other people who made the claim. Conservative member of parliament Lee Rowley said in Parliament on 11 January 2022: “There are over 400,000 more people in employment than before the pandemic” [24]. Another Conservative member of parliament Paul Scully also shared the same flawed statistic. It was also shared on Twitter by Mark Spencer, Suella Braverman and Nadine Dorries. The AI claim matching tool was able to monitor online media, Hansard and social media everyday for repeats of the claim and alert the Full Fact team [25]. This transformed what could have been a full day of monitoring work into a brief review of possible claim matches.
Ed Humpherson, Director General for Regulation at the Office for Statistics Regulation, wrote to Downing Street on 1 February 2022 to say it was “incorrect to state that there were more people in work at the end of this period than the start [26]. The work by Full Fact was one of the many factors that the team at OSR took into consideration when reviewing this response.
As of 7 March 2023, the Full Fact AI tool has identified 27 repeats of this claim which may otherwise have gone undetected. This has allowed the organisation to do two things. First it has been able to track the spread of the claim over time. Part of combating the misuse of statistics is understanding who misuses them, when, where and how. Being able to map the spread and impact of statistical misuse is a powerful way to draw attention to its consequences. Secondly, it allows for more opportunities to engage public figures on their statistical use (or misuse) and request corrections or engagements where appropriate. Full Fact has been able to take nine separate intervention actions on the misleading use of this official statistic.
4.Use of AI tools by Official Statistics Regulators
The OSR also undertakes regular monitoring of publicly available platforms, including the media and social media, to identify potential misuse of statistics. This monitoring also allows the organisation to conduct “horizon scanning”, where they detect and understand emerging trends within statistics and the narratives around them. This provides insight into how best to support statistics producers in the communication of high impact statistics and support the public’s understanding of the story that the statistics tell. Doing both of these activities manually comes with many challenges, which AI-powered tools can help overcome.
Firstly, there is a lot of information to monitor. Taking the media as an example, there are many mainstream media platforms. But there are also many smaller, independent outlets which will only report on statistics relevant to a certain domain, education statistics for example. Checking each of these requires a consistent process and good documentation of the search terms to use for each one. This level of consistency is hard to achieve manually as human memory is a significant factor. However, when using AI tools the platforms and search terms used can be hard coded and therefore no search is forgotten.
Secondly, the information being monitored changes rapidly. Each platform needs to be checked at least twice a week to ensure a timely response and that nothing is missed. In the case of social media, this needs to happen even more regularly as the speed of information exchange on these platforms is much higher. Such a regular and thorough activity takes up a lot of time, leaving less time to focus on other priorities. AI tools can run these checks at regular intervals which automatically free up regulators time to spend on other tasks.
Finally, checking for the use of specific words and phrases is tiring and prone to human error. When checking each of the platforms the OSR regulators are looking for references to official statistics which could be directly cited or, in the case of potential misuse, statistical references which are unrelated to official sources. These references could take one of many forms and that requires the regulators to have a keen eye and a high level of concentration over all platforms when searching. AI tools take away this element of human fatigue. AI tools can check many different forms of phrases and do so without tiring. This creates a higher level of job satisfaction for the regulators by avoiding repetitive processes.
AI tools have also enhanced other areas of the OSR’s work. They allow the organisation to automatically check when a misuse of statistics has been repeated either by the same original person/group or by others. Some cases of misuse require a public intervention, while others require a more personal approach. Irrespective of the action taken, the tools allow the organisation to assess the extent of the misuse quickly and easily.
The tools allow the organisation to filter statements and sentences based on whether they contain statistical information. This is done using the “quantity” claim type described earlier. The AI tools scrape publicly available platforms for content and use pre-trained Natural Language Processing (NLP) models to detect which sentences contain statistical language such as ‘percentage’, ‘increase’ or numerical information. In this way the OSR has been able to scrape information from Hansard (the official location of all UK Parliamentary debates) and review only sentences which contain statistical information.
Lastly, AI tools help answer important questions related to the organisation’s vision. For example, as part of an OSR research programme, the organisation wants to understand more about public good and how intermediaries, such as the media, describe the public good. This helps to complement other work in this space such as engagement work and literature reviews. In this example, the OSR used AI tools to search for claims made by individuals that contained the word ‘the public good’ and ‘statistics’ or ‘data’ from January 2020 to October 2022. The results of this have been fed into our workshops to help define the public good of statistics and overall have provided a view that doesn’t just rely on one time period.
5.Challenges and limitations
AI-powered tools have shown promise and impact for fact checkers and regulators monitoring the misuse of official statistics. However, there are limitations to consider.
There are benefits for statistical regulators to do their media monitoring manually. Requiring regulators to manually read and monitor information leads to more general awareness and understanding of a situation which can be beneficial for building knowledge of a domain.
Human regulators are also more likely to identify examples which are worded slightly differently to what an AI tool is expecting to see.
The tools, when used by either fact checkers or statistical regulators, still require human oversight. While they are able to identify claims and potential claim matches they are not able to understand context or nuance. A human is still necessary to ensure that the results are accurate and relevant.
However, the OSR has found that the general benefits of AI tools have outweighed this limitation. The organisation uses them, alongside regulator expertise, to extend and enhance its ability to monitor use of statistics. Similarly, Full Fact AI does not consider AI tools to be a replacement for fact checkers and the work they do. Rather the tools compliment their skills and free up their time to conduct other important tasks which cannot be automated.
6.Conclusion
Disinformation poses threats to societies around the world. Bad information can be created to mislead, without any basis in fact or evidence. In other cases, official statistics are misrepresented or misused. The implications of this are serious.
Firstly, the public is misled. This can sway public perceptions and understanding of important issues. A second casualty is the loss of trust in official statistics and the institutions that produce them.
A number of stakeholders are working to combat the misuse of official statistics and hold public figures accountable for misleading the public. These include independent fact checking organisations and statistical regulators. They share a number of common goals, including: detecting statistical errors, verifying claims shared in the public domain, holding public figures accountable for the information they share, educating the public about the misuse of statistics and encouraging transparency in statistical data. These shared goals often lead to coordinated efforts as partnerships, as is the case with Full Fact and the OSR.
The scale of the problem they are tackling is immense. False information can be created and disseminated at a speed never seen before. Traditional approaches of manual monitoring and detection are no match to the volume of information shared each day.
A solution to this problem is to empower fact checkers and statistical regulators with advanced technology, to allow them to level up their efforts. AI tools, originally designed for fact checkers, have shown their value with regulators. The current offering of tools allows both groups to monitor large volumes of public debate, find important statements and be informed when something they know to be wrong is repeated.
These technological solutions are not intended to replace fact checkers or statistical regulators. AI tools have been developed to assist fact checkers and statistical regulators in their efforts, enabling them to focus on more complex analytical work while the tools take care of routine tasks. These tools are available to users worldwide and have the potential to scale the impact of these groups’ work significantly, helping to combat the misuse of official statistics and promote trustworthy information.
References
[1] | Who we are. Full Fact. Available from: https://fullfact.org/about/. |
[2] | Who we are. (n.d.). Full Fact; (2023) March 6. Available from: https://fullfact.org/about/. |
[3] | IFCN Code of Principles. Poynter.org. Available from: https://ifcncodeofprinciples.poynter.org/know-more/the-commitments-of-the-code-of-principles. |
[4] | Google News Initiative. Helping prevent the spread of misinformation: Full Fact. Youtube; (2021) . Available from: https://www.youtube.com/watch?v=KZbzGMo3Jn4. |
[5] | Ox.ac.uk. Available from: https://reutersinstitute.politics.ox.ac.uk/sites/default/files/2018-02/graves_factsheet_180226. |
[6] | How does the UK’s GDP growth compare to the G7? Full Fact; (2023) . Available from: https://fullfact.org/economy/uk-gdp-growth-g7-february-2023/. |
[7] | Guo Z, Schlichtkrull M, Vlachos A. A survey on automated fact-checking. arXiv [csCL]. (2021) . Available from: http://arxiv.org/abs/2108.11896. |
[8] | Konstantinovskiy L, Price O, Babakar M, Zubiaga A. Toward automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection. Digit Threat. (2021) ; 2: (2): 1-16. doi: 10.1145/3412869. |
[9] | Boris Johnson makes false employment claim for ninth time in Parliament. Full Fact; (2022) . Available from: https://fullfact.org/economy/boris-johnson-makes-false-employment-claim-for-ninth-time-in-parliament/. |
[10] | Research doesn’t prove contraceptive pill reduces risk of suicide. Full Fact; (2022) . Available from: https://fullfact.org/health/contraceptive-pill-suicide-risk. |
[11] | Children should still follow COVID-19 rules after vaccination. Full Fact; (2021) . Available from: https://fullfact.org/health/easy-read-vaccine-consent-form. |
[12] | Claim ‘nine in 10’ teachers voted to strike is misleading. Full Fact; (2023) January 13. Available from: https://fullfact.org/education/nasuwt-neu-strike-ballot-tweet/. |
[13] | Public opinion in Scotland does support EU membership. Full Fact; (2023) . Available from: https://fullfact.org/scotland/alister-jack-scotland-opinion-eu-membership. |
[14] | Survey didn’t find one in ten young people plan ‘never’ to work. Full Fact; (2022) . Available from: https://fullfact.org/economy/sun-city-guilds-young-people-never-working/. |
[15] | Wilkinson K. Nigerian fact checkers fight election misinformation with Full Fact’s AI Tools. Google; (2023) . Available from: https://blog.google/intl/en-africa/company-news/technology/nigerian-fact-checkers-fight-election-misinformation-with-full-facts-ai-tools/. |
[16] | Highway Code changes infographic needs context. Full Fact; (2022) . Available from: https://fullfact.org/online/highway-code-changes/. |
[17] | Rollings G. New Highway Code rules are ‘dangerous and divisive’, blast motoring experts – but how many do you know? The Sun; (2022) Jan 24. Available from: https://www.thesun.co.uk/motors/17428143/new-highway-code-rules-motoring-cyclist-pedestrian/. |
[18] | Newman J, Quinn C, Hull R. Cyclists take to middle of the road and drivers are POWERLESS to stop them under new Highway Code. Daily mail; (2022) Jan 29. Available from: https://www.dailymail.co.uk/news/article-10454809/Cyclists-middle-road-drivers-POWERLESS-stop-new-Highway-Code.html. |
[19] | Leaker D. Number of People in Employment (aged 16 and over, seasonally adjusted):000s. Office for National Statistics; (2023) . |
[20] | Will Moy to Ed Humpherson: Statements on the number of people in work. Gov.uk. Available from: https://osr.statisticsauthority.gov.uk/wp-content/uploads/2022/02/Will-Moy-to-Ed-Humpherson-Statements-on-the-number-of-people-in-work.pdf. |
[21] | Engagements – Volume 704: debated on Wednesday 1 December 2021. Parliament.uk. Available from: https://hansard.parliament.uk//commons/2021-12-01/debates/A0E282CF-039D-4F26-8F16-946B8C6E2ABC/Engagements. |
[22] | Engagements – Volume 705: debated on Wednesday 15 December 2021. Parliament.uk. Available from: https://hansard.parliament.uk//commons/2021-12-15/debates/9FC7E88F-A7F2-47F7-9DDB-4A7252E40778/Engagements. |
[23] | Engagements – Volume 706: debated on Wednesday 5 January 2022. Parliament.uk. Available from: https://hansard.parliament.uk//commons/2022-01-05/debates/F0DD6E66-0813-4A93-814D-B6FAAFDC8A16/Engagements. |
[24] | Reducing Costs for Businesses – Volume 706: debated on Tuesday 11 January 2022. Parliament.uk. Available from: https://hansard.parliament.uk/commons/2022-01-11/debates/7EFC8B42-69ED-4F81-B9A7-FB4738A30C5E/ReducingCostsForBusinesses. |
[25] | Will Moy to Ed Humpherson: Statements on the number of people in work. Gov.uk. Available from: https://osr.statisticsauthority.gov.uk/wp-content/uploads/2022/02/Will-Moy-to-Ed-Humpherson-Statements-on-the-number-of-people-in-work.pdf. |
[26] | Ed Humpherson to Laura Gilbert: Statements on the number of people in work. Office for Statistics Regulation. Available from: https://osr.statisticsauthority.gov.uk/correspondence/ed-humpherson-to-laura-gilbert-statements-on-the-number-of-people-in-work. |
