Plagiarism Software Evaluation
for
NEW: Comparison between Safe Assignment and Turnitin.com (May 2004)
THIS PAGE IS NO LONGER UPDATED. These tests were done in the spring of 2003. Prices, performance and options may have changed since then.
On this page you can find a summary of the evaluation process that has been
performed by the members of the ELEUM Project Team
of The Faculty of Economics and Business Administration of Universiteit
Maastricht for the various types of
plagiarism detection tools that are available today.
The Virtual Academic Integrity Laboratory (VAIL)
of the University of Maryland University College (UMUC)
has developed a Faculty and Administrators Guide for Detection Tools and Methods.
Based on this document we have evaluated the currently available programs on the
basis of quality (of the results/reports), stability of the system, speediness
of the response and general user friendliness and ease of use. 10 randomly selected documents
were processed through the software tools, after which the results were
evaluated on the above mentioned factors. The results, including the
characteristics of the various tools, comments of the reviewers and some sample
reports and screenshots have been listed in the tables below. A conclusion
is presented at the end of this page.
This evaluation was conducted by Mark Arts and Bas Geus.
The plagiarism detection tools that were tested are:
| Name | EduTie.com |
| URL | http://www.edutie.com/ |
| Detection type | Cut & Paste, collusion |
| Vendor | Olexiy Shevchenko, Max Litvin, and Sasha Lugovskyy |
| Demo / Trial version | Yes; Free trial of 10 submissions |
| Price | Individual License $150 (200 manuscripts limit), Departmental License $800 (unlimited manuscripts), and Institutional License $1,400 (unlimited manuscripts) |
| Platform | Web |
| Submission formats | 1) Upload electronic files (.txt or .html) 2) Cut & Paste in a text box |
| Scope of search | System tracks Internet source(s) used in a document; shows the degree of plagiarism and provides direct links to the sources. |
| Result format | Web based report |
| Results turnaround | Submissions take approximately 24 hours to be processed. |
| Archive | Yes. The documents are submitted to an EduTie server so that the customer could view them later. However, all the documents submitted to EduTie remain the property of the author, and EduTie.com has no right to view the documents or disclose them to any third party. (EduTie) |
| General comments | Tracks all online paper mills and hosts a database of reportedly 250,000 academic papers/works. |
| Identified limitations | Submissions take approximately 24 hours to be processed; cannot access subscription websites |
| Special features | Online Paper Submission; development of secure institutional archive of submitted papers; integration into online course delivery systems; customization to your institutional Website set-up price ranges from $1000 to 4,000; administrators can monitor usage of a school or department’s account, create and change profiles and passwords and manage archives of uploaded papers. |
| ELEUM Project team comments | EduTie is a relatively easy to use tool. Reports are generated fairly fast and navigation is extremely user friendly. The only disadvantage is the fast that so far, only text and HTML files can be uploaded directly, while f.e. Word documents must be pasted in a textbox. The results page gives a clear overview of the outcome of the search and indicates the scores (number of hits) of the tested papers. On the basis of this score, the papers are rated on a five point scale (green, blue, purple, orange and red). Clicking on the report gives direct links to the website in which the series of words are found. Very convenient. Unfortunately, EduTie is not planning to integrate its tool in Blackboard. Also known as www.mydropbox.com. Based on Plagiserve software. |
| Name | PlagiServe.com |
| URL | http://www.plagiserve.com/ |
| Detection type | Cut & Paste, fraud |
| Vendor | Olexiy Shevchenko, Max Litvin, and Sasha Lugovskyy |
| Demo / Trial version | N/A |
| Price | Free |
| Platform | Web |
| Submission formats | 1) Cut & Paste in a text box |
| Scope of search | Searches the Internet and "paper mills"; has a local database of reportedly 70,000 student term papers, essays, and Cliff notes. |
| Result format | Web based report |
| Results turnaround | Generates a report in less than 12 hours. |
| Archive | ? |
| General comments | Though you must register, it provides free, unlimited use; service is based in the Ukraine. |
| Identified limitations | Instructor must submit all papers to system in one batch; only available through its website; cannot access subscription websites. |
| Special features | Provides list of resources for further information about plagiarism. |
| ELEUM Project team comments | Plagiserve is a very convenient software tool that first of all is free. The results are very extensive, giving an indicative percentage of possible plagiarism. Clicking on the reports leads to a web page, which underlines the phrases that have been found elsewhere. One can click on the lines to find the original source. Generating these result reports, however, can take up to 48 hours. On the other hand, this is a service that is free of charge. The various "sources" are indicated in a different color, so that one can see how many articles or websites have been used. The accuracy of the results is also sufficient. |
| Name | TunrItIn.com (after the tests some changes were made that improve speed and now contains all documents from ProQuest) |
| URL | http://www.turnitin.com/ |
| Detection type | Cut & Paste, collusion, fraud |
| Vendor | iParadigms, LLC/Plagarism.org |
| Demo / Trial version | Yes; Free 1 month subscription includes 5 "Originality Reports"; Tutorials available for various features of service. |
| Price | Prices were valid until sept. 2003
Individual Instructor Plan: Department Plan: Single campus institution license: (12 months) Multi-campus institution license: |
| Platform | Web |
| Submission formats | 1) Upload electronic files (.doc, .txt, .rtf, postscript, .pdf
or .html) 2) Cut & Paste in a text box |
| Scope of search | Submitted student papers are compared to internet resources; then against a database of previously submitted papers. This now includes all articles in ProQuest. |
| Result format | Web based report |
| Results turnaround | "Originality Reports" reported to be returned in more than 24 hours. During the first week of April 2003 when more tests were conducted, the results were all produced within 5 minutes. If this service stays this fast, it is a great improvement. |
| Archive | Yes. Student papers are retained in the TurnItIn.com database. All future submissions to the service are then checked against that growing database. |
| General comments | |
| Identified limitations | in March 2003, Originality reports reported to be returned in more than 24 hours of submission (this seems to have improved in April 2003); only available through its website; cannot access subscription websites. After September 2003 time has reduced to about 10 minutes. |
| Special features | Offers a "Peer Review" feature where instructors can customize assignments so that students can then use it to review and evaluate each others' work; "Digital Portfolio" allows students and teachers to review their papers previously submitted and archived. |
| ELEUM Project team comments | TurnItIn is definitely the most extensive tool that was tested, searching for matches on the web an within their growing database of submitted papers (including the ones that were posted by the user). Giving the possibility to upload documents with almost all common extensions is a large advantage. Users also have the possibility to cut and paste their texts into a textbox. The results are displayed on a five point/color scale, indicating the percentage of matching text: (1) Blue (<20%), (2) Green (20% - 25%), (3) Yellow (26% - 50%), (4) Orange (51% - 75%), and (5) Red (76% - 100%). The so-called similarity index immediately shows which papers need attention. Clicking on one of a result report gives a web-based document which underlines the plagiarized phrases. Each source has its own color, which unfortunately can not be clicked on in the text, only in the overview at the top of the page. Very convenient on the other hand is the DSC-button, to get a so-called direct source comparison. Clicking this button opens a new window with the originating source, in which the plagiarized part is underlined, so one can easily see where it came from (this feature seems to work about 50%of the time). An additional feature is the bulk-upload function, in which multiple papers can be submitted at once, without re-typing all log-in information over and over again, this still requires several clicks per paper. |
| Name | EVE2 (Essay Verification Engine 2.4) |
| URL | http://www.canexus.com/eve/index.shtml |
| Detection type | Cut & Paste |
| Vendor | CaNexus.com |
| Demo / Trial version | Yes; Free trial of 15 submissions |
| Price | Individual instructor- $19.95 (one time charge with free updates). Site license is a one time charge, start at $299.00 for high schools and grade schools, and $399 for Universities and Colleges. |
| Platform | Windows (PC); Installed on user's workstation |
| Submission formats | 1) Electronic files (.txt, .doc or .wpd) on workstation |
| Scope of search | Searches Internet resources through internet search engines |
| Result format | RTF-type document, to be opened in MS Word |
| Results turnaround | Heavily depends on the speed of your computer, and the speed of your Internet connection, and the size of the paper. A typical time is a few minutes per 2500 words. (CaNexus) |
| Archive | No. We will never share your students' work with anyone else; our software is designed to make this impossible. (CaNexus) |
| General comments | After submission the software then begins the search for the suspected text. This requires a continued connection to the Internet during the searching process. Results are presented in a pop-up window. |
| Identified limitations | Students cannot submit their own papers; cannot search for collusion within your students' coursework unless they are published on the web; cannot access subscription websites; not easily installed on server. |
| Special features | |
| ELEUM Project team comments | EVE2 is a tool that is installed on the user's workstation, but connects to the Internet while searching the document. The tool is very user friendly and can process several documents at once. The problem with this tool is that it requires a strong computer with extensive processing power. It requires enormous amounts of processor capacity, which slows down the user's operating system. Also, the fact that the tool itself is not very stable sometimes leads to runtime errors or reports without an outcome. However, the results that are generated lead to good indications, summing up the sources of the (underlined) matches that are found. The report is a RTF-document, which does not provide the opportunity to directly click on the links that are found. |
| Name | Wcopyfind (tested version + 2.2) |
| URL | http://plagiarism.phys.virginia.edu/Wsoftware.html |
| Detection type | Collusion |
| Vendor | Lou Bloomfield, Professor of Physics, University of Virginia |
| Demo / Trial version | N/A |
| Price | Free |
| Platform | Windows (PC); One file, executed on user's workstation |
| Submission formats | 1) Electronic files (.txt, .doc, .wpd, or .html) on workstation |
| Scope of search | Local files on hard drive or a local file server, including local files with .htm or .html extensions; web-resident documents that are pointed to by local Internet shortcuts (URLs). |
| Result format | HTML-type document |
| Results turnaround | Very fast; processing time is < 1 minute. |
| Archive | Student Papers N/A; program only scans through documents that you have in your collection or have identified on the web. |
| General comments | After submission the software then begins the search for the suspected text. This does not require a continued connection to the Internet during the searching process. Results are presented in a pop-up window. |
| Identified limitations | Cannot search Internet resources unless you specify an exact URL you suspect as a possible source; cannot access subscription websites. |
| Special features | |
| ELEUM Project team comments | Wcopyfind is an extremely useful tool for evaluating papers among students. The simple package is free of charge and directly checks a list of documents for matching parts. Very convenient and useful, indicating the percentage of match between 2 articles. The result report is a HTML-document which underlines the parts that are matched. |
| Name | Pl@giarism |
| URL | http://www.plagiarism.tk/ |
| Detection type | Word triples |
| Vendor | Georges Span |
| Demo / Trial version | Yes; Free trial for 2 projects of 20 documents |
| Price | TBA |
| Platform | Windows (network version) |
| Submission formats | 1) All document types that can be handled by MS Word |
| Scope of search | All documents that can be handled by MS-Word stored on either a hard drive or a local file server; web-resident documents or
web-pages that are pointed to by local Internet shortcuts (URLs). Main
goal is to compare assignments of one class. Limited functionality is
available for web checks.
Search method more advanced than WCopyFind or others because not only strings of words are searched but word triples. |
| Result format | MS-Access database with tables indicating various kinds of match percentages between all the matched document pairs. Within the program all document pairs can be shown with all or selected matches highlighted. |
| Results turnaround | Depends on amount of document compared and size of documents. Furthermore when virus-scan is enabled the process will take longer. Normally it takes 10 minutes for 200 documents to be compared with each other. |
| Archive | All original documents and generated data will be available for future comparison. Applies to local hard drive, not web based database of papers. |
| General comments | Possible to improve comparison results by excluding commonly used phrases. Possible to compare documents exclusively to a set of other documents (such as textbooks or answer templates). |
| Identified limitations | Is good at what is mainly for: cross checking assignments, limited web source functionality. More complex user interface then WCopyFind due to it larger number of possibilities. |
| Special features | Documents can be saved or printed with their matches underlined. Document pairs can be viewed with their matches underlined. Matches can be selected and highlighted differently within the document comparison view. Text from the original document can be selected and used for web search. Web results can be imported and used for comparison either in HTML-form or as a URL linking to the web page. |
| ELEUM Project team comments | Not elaborately tested and evaluated by the ELEUM project team. |
The Results
The table below gives the results of the test-papers that were used in order to evaluate the various tools. These are just the absolute values that were based on the scales that are used by the respective tool. For comparison reasons this can only be used as an indicative method of how well the tools are performing on the same types of papers.
| Test | EduTie | PlagiServe | TurnItIn | EVE2 | Wcopyfind |
| Paper 1 | Green, Score 8 | 5% Possibility | Blue | No indication for fraud | No match found |
| Paper 2 | Green, Score 11 | 6% Possibility | Green | N/A | 2% match with paper 1 |
| Paper 3 | Green, Score 6 | 7% Possibility | Green | N/A | 1%match with paper 2 |
| Paper 4 | Green, Score 12 | 9% Possibility | Blue | N/A | No match found |
| Paper 5 | Green, Score 7 | 4% Possibility | Blue | No indication for fraud | No match found |
| Paper 6 | Green, Score 16 | 15% Possibility | Green | 7,81% Possibility | 1% match with paper 10 |
| Paper 7 | Green, Score 8 | 8% Possibility | Green | 0,85% Possibility | No match found |
| Paper 8 | Green, Score 16 | 14% Possibility | Green | 3,23% Possibility | No match found |
| Paper 9 | Green, Score 18 | 13% Possibility | Green | N/A | No match found |
| Paper 10 | Green, Score 8 | 5% Possibility | Yellow | 8,29% Possibility | 1% match with paper 2 |
| Test | EduTie | PlagiServe | TurnItIn | EVE2 | Wcopyfind |
| Paper 11 | Orange, Score 66 | 61% Possibility | Yellow | 67,43% Possibility | N/A |
| Paper 12 | Green, Score 9 | 3% Possibility | Green | 0,35% Possibility | N/A |
| Paper 13 | Green, Score 15 | 13% Possibility | Green | 11,31% Possibility | N/A |
| Paper 14 | Green, Score 7 | 7% Possibility | Green | 3% Possibility | N/A |
Scales:
EduTie:
Score 1 - 100 points, resulting in colors: (1) Green, (2) Blue,
(3) Purple, (4) Orange, (5) Red
PlagiServe: Possible matching percentage between 0% - 100%
TurnItIn:
(1) Blue (<20%), (2) Green (20% - 25%), (3) Yellow (26% - 50%), (4)
Orange (51% - 75%), and (5) Red (76% - 100%)
EVE2: Possible matching percentage between 0% - 100%
Wcopyfind: Possible matching percentage between 0% - 100%
Useful links:
- Results of a similar test at the University of Oxford
Review of some text-based plagiarism detection tools at Oxford University, Computing Services- Plagiarism resources, provided by TurnItIn.com
Research resources for both students and instructors, provided by TurnItIn.com- Plagiarism.org
Online resource for educators concerned with the growing problem of Internet plagiarism, again linked to TurnItIn.com.- What is plagiarism and how to recognize and avoid it?
Plagiarism resources, provided by Indiana University- Grading by Computer (DUTCH)
http://www.leidenuniv.nl/mare/2003/27/0901.html
Testing these tools has resulted in varying degrees of success. The stability
and speediness of responses various very much among the tools. Although each and
every tool makes use of different sources (databases), in general we can
conclude that the experiments came up with similar results. The advantages and
disadvantages of each tool were evaluated. One of the disadvantages so far is
that databases with academic literature (such as EBSCO) are not yet linked to
the sources that are used by the software companies. Moreover, bulk uploads,
which allow numerous papers to be uploaded with a minimum amount of clicks, are
so far only available at TurnItIn. Based on the tests that
were conducted, the ELEUM Project-team therefore decided to advise TurnItIn and
Wcopyfind. In terms of stability,
ease of use, quality of reports and costs, these two
software tools offer the Faculty of Economics and Business Administration
sufficient possibilities to check for possible plagiarism. By themselves these tools are not enough to detect plagiarism - academics need to judge whether similarities between essays and with online sources are significant - but they can help in reaching a
diagnosis (OUCS).
