MiniJudgeJS
Version 1.1.2 - Last updated on May 1, 2018

What is MiniJudge?

SKIP INTRODUCTION

MiniJudgeJS is a tool for small-scale judgment experimentation. It helps you to design, run, and analyze linguistic acceptability judgment experiments that are as simple and powerful as possible.

Software requirements. MiniJudgeJS itself is just the page you're looking at right now: an HTML file containing JavaScript. MiniJudgeJS has been most extensively tested in Firefox 1.5 running in (Traditional Chinese) WindowsXP Professional 5.1, and it also seems to work just fine in other versions of Firefox (for Mac and Linux), as well as Internet Explorer, Safari, and Opera. Reports about any remaining incompatibilities or other bugs, as well as suggestions for new features, are most welcome!

To analyze the data collected with MiniJudgeJS, you will also need to install R, the free yet powerful statistics software package that is fast becoming a world standard (www.r-project.org). R is the one free statistics program you might actually want to own. To learn more about R, including how to download it, see HERE.

Further basics. If this is your first time trying MiniJudgeJS, you might want to scroll through the whole thing before starting a project, just to get a sense of what you're in for. MiniJudgeJS is also printer-friendly (it's about 20 pages long).

Using MiniJudgeJS involves a series of guided or fully automated steps, divided into three major parts: designing, running, and analyzing. It is safest to reserve enough time for each major part to finish it in one session, since the current version of MiniJudgeJS cannot save your work as a whole. Similarly, don't reload the page while you're working, or some information may be lost. You can copy your work from the screen at any time and paste it into a file as you work.

To use MiniJudgeJS offline, just save this page to your computer (along with this information page, if you like). Opening offline files with JavaScript may make some newer versions of IE nervous, but rest assured that MiniJudgeJS doesn't read or write anything on your hard drive (it handles input and output by asking you to paste to and copy from text windows). If you encounter security warnings using MiniJudgeJS offline in IE, either tell IE to allow blocked content, or better yet, dump IE and get a proper browser.

MiniJudgeJS is "copylefted" with the GNU General Public License. Basically, this means you can distribute it freely (e.g. create mirrors) and modify it any way you like, including incorporating its code in your own program to sell, but the contribution of MiniJudgeJS must be acknowledged and stay open-source in your new program. See the page source for more information.

MiniJudgeJS is far from perfect. If something weird happens when you use it, or if you have any suggestions about how to improve it, contact the author HERE, giving details on your operating system, browser, and what you were doing when it happened. MiniJudgeJS will be upgraded eventually. If you're impatient, feel free to upgrade it yourself!

If you use MiniJudgeJS in your research, please cite it. MiniJudgeJS has also proven useful as a classroom teaching aid.

For more information about judgment experiments, small-scale experimentation, MiniJudge, and R, click HERE.

• Design experiment
• Choose experimental factors
• Choose set of prototype items
• Choose total number of item sets
• Segment prototype set (optional)
• Choose replacement segments for additional item sets (optional)
• Approve and save master list of test items
• Run experiment
• Choose number of speakers
• Write instructions for speakers
• Save schematic survey information
• Copy survey forms to print or email
• Analyze experiment
• Download and install R (if you haven't already)
• Enter raw results
• Generate data file
• Add neighborhood densities (optional, for judgments of word-sized items)
• Save data file
• Generate R code
• Paste R command code into R
• R will summarize your findings in an easy-to-read format

To help isolate interesting syntactic factors from nuisance factors, MiniJudgeJS generates sets of linguistic items (e.g. sentences or pseudowords) in accordance with a factorial design. To make this process as simple as possible, item sets are based on a "prototype set" of items, similar to the example sets cited in linguistics papers, particularly in syntax.

If you choose two factors, MiniJudgeJS will not only test if they show main effects, but also if there is an interaction. For example, you would expect the factors AdjunctExtract and CenterEmbed both to show main effects, but you would not expect an interaction (since they presumably involve independent "modules," namely competence vs. performance). By contrast, if your factors were Adjunct and WhMove, you wouldn't expect either to show main effects (since adjunct structures and wh-movement are both grammatical), but they should show an interaction (since wh-movement from adjuncts should be bad, while wh-movement from non-adjuncts should be good).

Enter your factor(s), then click OK.

Segment prototype set. SKIP INTRO / SKIP TO MASTER LIST

MiniJudgeJS can help you create multiple item sets, but since the program knows no human language the process may be a bit buggy. If you would rather create the entire list of experimental items yourself, click HERE. Either way, generation of structurally matched item sets may be easier with the help of a dictionary or thesaurus (there are many available online); see HERE for more information.

The additional item sets will be variants on your prototype set. If you have gone through the previous steps, MiniJudgeJS has already segmented your prototype set into strings. These represent pieces that will vary as wholes across your item sets (or not vary at all across sets, if you choose).

For example, for the demo prototype set, the list of unique segments is as follows. Note the separate listings for "know" and "knew", but only one listing for "you", despite varying in capitalization in the prototype set. Note also that "who did" and "I saw" are treated as whole pieces, since these word strings are never split up in the prototype set. (If you're confused about the point of segmenting, scroll down to the following two steps to see how the segments will be used.)

TIP: You can use the special character "|" in your prototype items to demarcate strings that should not be split up (e.g. fixed phrases or words in sentences written in orthographies that don't mark them, like Chinese). If you want to edit your prototype items now, click HERE.

The segments found by MiniJudgeJS are listed in the window below. Compare this list with your prototype set, shown below on the right. If necessary, modify the list of segments to make sure that they really represent the words and word strings in your prototype set that you want to vary across the other item sets. NOTE: The current version of MiniJudgeJS only permits automatic substitution for up to ten segments (any additional segments will just be copied across all sets). To reduce the number, you can combine segments, or delete segments that will not be varied across item sets (e.g. that in a study on that-trace effects).

For each of the prototype segments, enter matching substitutes for the new item sets (Sets 02 and up) in the cells below (if arrows appear at the edge, you should scroll around to make sure you fill them all). If you want to delete a prototype segment, enter a space. If you want to reuse a prototype segment, copy it from the header. You cannot add any new segment to those in the current prototype set. If you want to lengthen your prototype items to make deletion possible, click HERE. (NOTE: Do not edit headers or set numbers.)

When you are satisfied with your new segments, click OK.

ENTER MATCHING ADDITIONAL SEGMENTS HERE

IMPORTANT: If you are restarting your work at this point, you must first paste the master list back into the window above and reapprove it.

In the current version of MiniJudge, responses must be binary YES/NO judgments (see HERE for justification of this limitation). If speakers give their responses by email, they must use "1" to represent YES (acceptable) and "0" to represent NO (unacceptable), and they must record their judgment for an item at the beginning of the line, before the item ID number. These restrictions should be made clear in the instructions.

Suggested instructions are shown in the window below (intended for use with English speakers filling out surveys by email at an institution requiring the use of informed consent forms). Modify them any way you like (e.g. translating them into the native language of your speakers, or deleting them entirely).

When you are satisfied with the instructions, click OK to generate the surveys.

If you went through all of the above steps, MiniJudgeJS has taken your master list of items and your instructions, and has used them to generate the number of survey forms that you requested. The items have been put into a different partially random order for each survey. Randomization is only partial to ensure that item types are spread evenly within each survey (click HERE for more on the algorithm used). Randomization and even spread have another benefit, explained below and HERE.

The surveys given to speakers do not explicitly mark the factors and sets, since this information may bias speaker judgments. But MiniJudgeJS needs this information for the statistical analyses. The window below shows the crucial information in schematic form (removing irrelevant information like the actual items). It looks ugly, but only MiniJudgeJS has to read it.

Copy and save the contents of the window below to a file on your computer. You can give the file any name and save it anywhere you like, since when you're ready to do statistical analysis, you'll just copy and paste the contents back into another window in a later step.

If you're curious, each line in the schematic info file is formatted like so:

Download and install R. SKIP

The data collected with MiniJudgeJS are categorical (binary) and repeated-measures data (i.e. each speaker gives multiple judgments). This type of data requires a highly sophisticated kind of statistics (called GLMM) that is hard to implement efficiently in JavaScript (this is a job for future work).

MiniJudgeJS passes the hard work over to R, so you need to download R before you can continue. R is by far the best free statistics package available, so it's worth owning if you do any quantitative research. Click HERE for more information about R, including information about how to download it.

Enter raw results. SKIP INTRO

MiniJudgeJS can handle two kinds of raw results: surveys completed electronically with a "1" or "0" near the start of each line, or a list of "1"s and "0"s that you typed in yourself from completed paper surveys.

Either way, MiniJudgeJS needs to know the survey ID number and the order number for each item (i.e. the numbers shown to the speakers).

The format that MiniJudgeJS is expecting for the raw results is as shown in the box below. Survey ID numbers are marked with "##", just as on the survey forms, followed by the data from that survey. The judgments (1 vs. 0) appear at the beginning of each line, followed by the item number inside parentheses, again just as on the survey forms. The items are numbered as on the survey forms, so item (1) on one survey is not necessarily the same as item (1) on another survey. This is why the survey ID numbers are crucial! They also permit surveys to be entered in any order that is convenient, as shown below.

Notice that the formatting above isn't completely "clean": speaker 2 didn't delete the ">" mark added when replying by email, the experimenter accidently hit "p" while typing in speaker 1's data, and speaker 3 ignored the instructions and put the judgments at the end of each line instead of at the beginning. That's OK -- MiniJudgeJS just looks for each item ID number in parentheses, and takes the first "1" or "0" before it on the same line as the judgment for that item; if it can't find a "1" or "0" there, it looks at the end of the line. It ignores everything else, including other numbers or letters accidentally typed, the instructions, and the items themselves.

Note also that speaker 2 is missing a judgment for item 3. MiniJudgeJS can handle this too, and GLMM is not as badly affected by missing data as some other statistical methods are. Similarly, it's not fatal if you didn't actually end up running as many speakers as you originally planned. Of course, too much missing data (or data missing in a non-random pattern) can cause problems with interpreting the results, which is why speakers must be encouraged to judge all items.

However, the data for speaker 4 will confuse MiniJudge (and maybe you too). Item (1) won't be counted, since there are two different judgments (both 1 and 0) on the same side of the item number. Item (2) also shows two judgments, but only the one before the item number (where the instructions say to put it) will be counted.

Finally, MiniJudgeJS doesn't mind if you ended up testing fewer speakers than originally planned (e.g. you emailed 20 surveys but only 10 speakers responded). It can also handle it if you accidentally enter a speaker's results twice (e.g. ##01 ... ##02 ... ##01 ...); it will ignore any extra copies.

Add neighborhood densities. (Optional, for judgments of word-sized items) SKIP

Acceptability judgments of words-sized nonlexical items are strongly affected by analogy with neighboring (i.e. similar) lexical items. To factor out such effects, click HERE to let MiniCorp calculate neighborhood densities for your items. (You will need the Master List file for your experiment and a corpus of all lexical items in the language.) Otherwise, click HERE to skip this step.

In the window below, paste in the neighborhood density file created using MiniCorp. Then click OK to add neighborhood densities to the data file.

ENTER NEIGHBORHOOD DENSITIES HERE

You must also decide if you want R to test for satiation, where judgment contrasts weaken over time (there may also be "anti-satiation", where judgment contrasts strengthen). Satiation has been taken by some to provide a new window into competence, performance, or their relationship, but testing it in MiniJudgeJS may have counterintuitive (though mathematically correct) effects on the rest of your analysis.

Finally, if your experiment involves word-sized items and you have already added neighborhood density information to the data file, you must decide if you want R to factor out effects due to neighborhood density.

Enter the name of your data file in the box below. Check the boxes to adjust the analysis as desired. Then click OK to generate the R code.

If all has gone well, by now MiniJudgeJS should have generated the R command code needed to analyze your experimental results. When you paste this code into the R program window^*, it will automatically test the significance of the main factor(s), their interaction (if two factors), and order. If you decided to test for satiation, R will also look for interactions between order and the main factor(s). For experiments using word-sized items, R will also factor out the effects of neighborhood density if you requested this.

NOTE: As of version 1.1, the R code generated by MiniJudgeJS refers to "Item" rather than "Sentence"; change the relevant header in older data files before analyzing them.

The analysis will be summarized in a brief, easy-to-read format. This is based on R's own detailed but technical format, which will be automatically saved in a file in the same folder as your data file.

To use the command code, start the R program on your computer. Then you must change R's directory to the location of your data file using R's FILE menu. Then simply copy and paste the code below into the R window^*. If this is your first time using MiniJudgeJS, R will first ask you to download the key package. R will then pause for several seconds while loading it into the computer's memory.

^*Pasting the R code may not work in Linux. In this case, first save the R code in a file (e.g. mjRcode), and then instead of pasting it into R, use R's source function (e.g. source("mjRcode")).

For help understanding your statistical results, click HERE.

There are other fancier versions of MiniJudge in Java, which you can try HERE. Unfortunately they still use the old version of the R code, so... they probably don't work. Does anybody who knows Java want to help fix them?

Some users may also be interested in MiniJudgeJS Deluxe. It's just like ordinary MiniJudgeJS, except that it costs US$375 and requires a dongle. Click HERE for information.

The MiniGram main page has other relevant references.

Citing R:

Citing lme4 [R package for GLMM]:

Contact James Myers with your questions and comments.