MiniJudgeJS Version 1.1.2 - Last updated on May 1, 2018
What is MiniJudge?
SKIP INTRODUCTION
MiniJudgeJS is a tool for small-scale judgment experimentation. It helps you to design, run, and analyze linguistic acceptability judgment experiments that are as simple and powerful as possible.
Simplicity. If you have ready access to native speakers, "live" or by email, you can go from hunch to statistically supported generalization in a matter of days, if not hours. This is far faster than the typical psycholinguistic experiment and only somewhat slower than the pace at which syntacticians traditionally collect data.
Power. MiniJudgeJS helps collect judgments from multiple speakers judging multiple linguistic items (e.g. sentences or words), and then analyzes them with sophisticated statistics to extract as much information as possible from the data. This way you can be more confident that judgment patterns really relate to the syntactic factors of interest, and not bias, idiosyncrasies of speakers or linguistic items, or certain processing effects.
Software requirements. MiniJudgeJS itself is just the page you're looking at right now: an HTML file containing JavaScript. MiniJudgeJS has been most extensively tested in Firefox 1.5 running in (Traditional Chinese) WindowsXP Professional 5.1, and it also seems to work just fine in other versions of Firefox (for Mac and Linux), as well as Internet Explorer, Safari, and Opera. Reports about any remaining incompatibilities or other bugs, as well as suggestions for new features, are most welcome!
To analyze the data collected with MiniJudgeJS, you will also need to install R, the free yet powerful statistics software package that is fast becoming a world standard (www.r-project.org). R is the one free statistics program you might actually want to own. To learn more about R, including how to download it, see HERE.
Further basics. If this is your first time trying MiniJudgeJS, you might want to scroll through the whole thing before starting a project, just to get a sense of what you're in for. MiniJudgeJS is also printer-friendly (it's about 20 pages long).
Using MiniJudgeJS involves a series of guided or fully automated steps, divided into three major parts: designing, running, and analyzing. It is safest to reserve enough time for each major part to finish it in one session, since the current version of MiniJudgeJS cannot save your work as a whole. Similarly, don't reload the page while you're working, or some information may be lost. You can copy your work from the screen at any time and paste it into a file as you work.
To use MiniJudgeJS offline, just save this page to your computer (along with this information page, if you like). Opening offline files with JavaScript may make some newer versions of IE nervous, but rest assured that MiniJudgeJS doesn't read or write anything on your hard drive (it handles input and output by asking you to paste to and copy from text windows). If you encounter security warnings using MiniJudgeJS offline in IE, either tell IE to allow blocked content, or better yet, dump IE and get a proper browser.
MiniJudgeJS is "copylefted" with the GNU General Public License. Basically, this means you can distribute it freely (e.g. create mirrors) and modify it any way you like, including incorporating its code in your own program to sell, but the contribution of MiniJudgeJS must be acknowledged and stay open-source in your new program. See the page source for more information.
MiniJudgeJS is far from perfect. If something weird happens when you use it, or if you have any suggestions about how to improve it, contact the author HERE, giving details on your operating system, browser, and what you were doing when it happened. MiniJudgeJS will be upgraded eventually. If you're impatient, feel free to upgrade it yourself!
If you use MiniJudgeJS in your research, please cite it. MiniJudgeJS has also proven useful as a classroom teaching aid.
For more information about judgment experiments, small-scale experimentation, MiniJudge, and R, click HERE.
Steps
Using MiniJudgeJS involves following a series of steps. Scroll down (or, if you are continuing a project, click a step to jump directly to where you left off).
Design experiment.
To help isolate interesting syntactic factors from nuisance factors, MiniJudgeJS generates sets of linguistic items (e.g. sentences or pseudowords) in accordance with a factorial design. To make this process as simple as possible, item sets are based on a "prototype set" of items, similar to the example sets cited in linguistics papers, particularly in syntax.
Choose experimental factors.
Name one or two binary factors, representing the syntactic, lexical, phonological, morphological, or other variables differentiating the items in your sets. You can choose factors that you predict to be directly relevant to grammar (e.g. "AdjunctExtract" in an experiment on adjunct islands), but you may also include a performance factor (e.g., "CenterEmbed" for center-embedded structures).
NOTE: If you have two factors, you must choose names starting with different characters, since they'll be abbreviated in some tables below. Also, factor names may not contain spaces, since this will cause trouble for the R analysis.
If you choose two factors, MiniJudgeJS will not only test if they show main effects, but also if there is an interaction. For example, you would expect the factors AdjunctExtract and CenterEmbed both to show main effects, but you would not expect an interaction (since they presumably involve independent "modules," namely competence vs. performance). By contrast, if your factors were Adjunct and WhMove, you wouldn't expect either to show main effects (since adjunct structures and wh-movement are both grammatical), but they should show an interaction (since wh-movement from adjuncts should be bad, while wh-movement from non-adjuncts should be good).
Enter your factor(s), then click OK.
Choose set of prototype items.
As with a set of examples in a linguistics paper, the prototype items should be matched as much as possible for irrelevant properties (e.g. lexical content in sentences), so that they differ only in terms of the factor (or factors) that you are interested in studying.
+A
+W
Who did you know because I saw?
-W
You knew because I saw him.
-A
Who did you know I saw?
You knew I saw him.
In the spaces indicated below, enter a set of prototype items such that each item represents one factor value (or combination of factor values). Note that factor names are abbreviated to just one character to save space (do not edit these).
When you are satisfied with your items, click OK.
Choose total number of item sets.
A single item set is probably not enough to permit statistically reliable results. Moreover, individual sets of items may have unknown idiosyncrasies. A reasonable number would be 5-20 sets.
Enter your chosen number in the box below, then click OK.
Total number of item sets:
Segment prototype set. SKIP INTRO / SKIP TO MASTER LIST
MiniJudgeJS can help you create multiple item sets, but since the program knows no human language the process may be a bit buggy. If you would rather create the entire list of experimental items yourself, click HERE. Either way, generation of structurally matched item sets may be easier with the help of a dictionary or thesaurus (there are many available online); see HERE for more information.
The additional item sets will be variants on your prototype set. If you have gone through the previous steps, MiniJudgeJS has already segmented your prototype set into strings. These represent pieces that will vary as wholes across your item sets (or not vary at all across sets, if you choose).
For example, for the demo prototype set, the list of unique segments is as follows. Note the separate listings for "know" and "knew", but only one listing for "you", despite varying in capitalization in the prototype set. Note also that "who did" and "I saw" are treated as whole pieces, since these word strings are never split up in the prototype set. (If you're confused about the point of segmenting, scroll down to the following two steps to see how the segments will be used.)
who did you know because I saw knew him
Who did you know because I saw? You knew because I saw him. Who did you know I saw? You knew I saw him.
TIP: You can use the special character "|" in your prototype items to demarcate strings that should not be split up (e.g. fixed phrases or words in sentences written in orthographies that don't mark them, like Chinese). If you want to edit your prototype items now, click HERE.
The segments found by MiniJudgeJS are listed in the window below. Compare this list with your prototype set, shown below on the right. If necessary, modify the list of segments to make sure that they really represent the words and word strings in your prototype set that you want to vary across the other item sets. NOTE: The current version of MiniJudgeJS only permits automatic substitution for up to ten segments (any additional segments will just be copied across all sets). To reduce the number, you can combine segments, or delete segments that will not be varied across item sets (e.g. that in a study on that-trace effects).
When you are satisfied that the segments are correct, click OK.
Choose replacement segments for additional item sets.
To help MiniJudgeJS generate variants of your prototype set, enter syntactic equivalents for the prototype segments below. For example, to add two more item sets to the demo prototype set, the following new segments could be entered (the prototype set is Set 01):
01
who did
you
know
I saw
knew
him
02
03
For each of the prototype segments, enter matching substitutes for the new item sets (Sets 02 and up) in the cells below (if arrows appear at the edge, you should scroll around to make sure you fill them all). If you want to delete a prototype segment, enter a space. If you want to reuse a prototype segment, copy it from the header. You cannot add any new segment to those in the current prototype set. If you want to lengthen your prototype items to make deletion possible, click HERE. (NOTE: Do not edit headers or set numbers.)
When you are satisfied with your new segments, click OK.
ENTER MATCHING ADDITIONAL SEGMENTS HERE
Approve and save master list of test items.
If you decided to create the entire list of experimental items yourself, complete the list below, adding variant item sets in accordance with the indicated factor values (note that factor names are abbreviated to just one character).
Otherwise, carefully check the items below that were created by MiniJudgeJS, editing them where necessary (Note that the first number is the item number and the second is the set number; do not edit this information or the factor name(s) shown at the top).
When you are satisfied with the master list of items, click OK.
CHECK/EDIT MASTER LIST HERE
After you have approved the master list in the window above, select, copy, and save it to a file on your computer. You can give the file any name you like and save it anywhere, but you will need this information when you generate your survey forms in the next major step. If you must stop working now, restart your work at this step, first pasting your master list back into this window and reapproving it.
Run experiment.
MiniJudgeJS doesn't actually run the experiment for you. Instead, it generates survey forms for you to distribute to native speakers. You can copy the whole list of survey forms to a word processing program and paginate it for printing, one survey per sheet of paper. Or you can copy each survey into separate emails, one for each speaker.
IMPORTANT: If you are restarting your work at this point, you must first paste the master list back into the window above and reapprove it.
Choose number of speakers.
A single speaker is probably not enough to permit statistically reliable results. Moreover, individual speakers may have unknown idiosyncrasies. A reasonable number would be 10-25 speakers.
Enter your chosen number in the box below, click OK.
Number of speakers:
Write instructions for speakers.
In the current version of MiniJudge, responses must be binary YES/NO judgments (see HERE for justification of this limitation). If speakers give their responses by email, they must use "1" to represent YES (acceptable) and "0" to represent NO (unacceptable), and they must record their judgment for an item at the beginning of the line, before the item ID number. These restrictions should be made clear in the instructions.
Suggested instructions are shown in the window below (intended for use with English speakers filling out surveys by email at an institution requiring the use of informed consent forms). Modify them any way you like (e.g. translating them into the native language of your speakers, or deleting them entirely).
When you are satisfied with the instructions, click OK to generate the surveys.
EDIT INSTRUCTIONS HERE
IMPORTANT: Your participation in this study is entirely voluntary. If you wish to participate, you must read and electronically sign the attached informed consent form. For each item below, decide whether or not it seems English-sounding, that is, whether it could be spoken by a native speaker of English. If you think it does, put "1" before the item (at the beginning of the line). If you don't think so, put "0". If you aren't sure, put "1" or "0" anyway -- just guess! Please give a judgment for EVERY item, in the order that they are shown. Some items are quite similar to each other, but please consider each one separately. When you are done, send back the whole email with your judgments. After I get enough surveys back, I will tell you and everybody else what this is all about. Thanks for your help!
Save schematic survey information.
If you went through all of the above steps, MiniJudgeJS has taken your master list of items and your instructions, and has used them to generate the number of survey forms that you requested. The items have been put into a different partially random order for each survey. Randomization is only partial to ensure that item types are spread evenly within each survey (click HERE for more on the algorithm used). Randomization and even spread have another benefit, explained below and HERE.
The surveys given to speakers do not explicitly mark the factors and sets, since this information may bias speaker judgments. But MiniJudgeJS needs this information for the statistical analyses. The window below shows the crucial information in schematic form (removing irrelevant information like the actual items). It looks ugly, but only MiniJudgeJS has to read it.
Copy and save the contents of the window below to a file on your computer. You can give the file any name and save it anywhere you like, since when you're ready to do statistical analysis, you'll just copy and paste the contents back into another window in a later step.
COPY SCHEMATIC SURVEY INFO FROM HERE
If you're curious, each line in the schematic info file is formatted like so:
Copy survey forms to print or email.
The window below shows the survey forms for distribution to speakers. Note that forms are separated from each other by lines, and each form gives the instructions, speaker number (preceded by "##"), and list of items numbered in the order in which they appear in that survey (with the numbers in parentheses).
To create printed surveys, copy the contents of the window below to a word processing program and replace the lines with page breaks. To create email surveys, copy each survey to a separate email message. The latter method will save you the trouble of retyping the judgment data for statistical analysis, but it has other disadvantages: you can't watch to make sure the speakers follow the instructions properly, and some speakers may be annoyed by unsolicited email!
COPY SURVEY FORMS FROM HERE
Analyze experiment.
MiniJudgeJS passes on the job of statistical analysis to R. MiniJudgeJS will talk to R for you, and have R give a nontechnical summary of the statistical findings.
Download and install R. SKIP
The data collected with MiniJudgeJS are categorical (binary) and repeated-measures data (i.e. each speaker gives multiple judgments). This type of data requires a highly sophisticated kind of statistics (called GLMM) that is hard to implement efficiently in JavaScript (this is a job for future work).
MiniJudgeJS passes the hard work over to R, so you need to download R before you can continue. R is by far the best free statistics package available, so it's worth owning if you do any quantitative research. Click HERE for more information about R, including information about how to download it.
Enter raw results. SKIP INTRO
MiniJudgeJS can handle two kinds of raw results: surveys completed electronically with a "1" or "0" near the start of each line, or a list of "1"s and "0"s that you typed in yourself from completed paper surveys.
Either way, MiniJudgeJS needs to know the survey ID number and the order number for each item (i.e. the numbers shown to the speakers).
The format that MiniJudgeJS is expecting for the raw results is as shown in the box below. Survey ID numbers are marked with "##", just as on the survey forms, followed by the data from that survey. The judgments (1 vs. 0) appear at the beginning of each line, followed by the item number inside parentheses, again just as on the survey forms. The items are numbered as on the survey forms, so item (1) on one survey is not necessarily the same as item (1) on another survey. This is why the survey ID numbers are crucial! They also permit surveys to be entered in any order that is convenient, as shown below.
##02 1 > (1) This experiment is boring. 0 > (2) I fun am having. > (3) Colorless green ideas sleep furiously. 1 > (4) When will this experiment be over? ... ##01 0 (1) p0 (2) ... ##03 (1) Bananas are yummy. 1 (2) Bananas yummy are. 0 ... ##04 (1) 1 Whatever. 0 1 (2) Yeah, sure. 0 ...
Notice that the formatting above isn't completely "clean": speaker 2 didn't delete the ">" mark added when replying by email, the experimenter accidently hit "p" while typing in speaker 1's data, and speaker 3 ignored the instructions and put the judgments at the end of each line instead of at the beginning. That's OK -- MiniJudgeJS just looks for each item ID number in parentheses, and takes the first "1" or "0" before it on the same line as the judgment for that item; if it can't find a "1" or "0" there, it looks at the end of the line. It ignores everything else, including other numbers or letters accidentally typed, the instructions, and the items themselves.
Note also that speaker 2 is missing a judgment for item 3. MiniJudgeJS can handle this too, and GLMM is not as badly affected by missing data as some other statistical methods are. Similarly, it's not fatal if you didn't actually end up running as many speakers as you originally planned. Of course, too much missing data (or data missing in a non-random pattern) can cause problems with interpreting the results, which is why speakers must be encouraged to judge all items.
However, the data for speaker 4 will confuse MiniJudge (and maybe you too). Item (1) won't be counted, since there are two different judgments (both 1 and 0) on the same side of the item number. Item (2) also shows two judgments, but only the one before the item number (where the instructions say to put it) will be counted.
Finally, MiniJudgeJS doesn't mind if you ended up testing fewer speakers than originally planned (e.g. you emailed 20 surveys but only 10 speakers responded). It can also handle it if you accidentally enter a speaker's results twice (e.g. ##01 ... ##02 ... ##01 ...); it will ignore any extra copies.
In the window below, paste in or type the raw results, following the above format. Then click OK.
ENTER RAW RESULTS HERE
Generate data file.
In order to convert the raw data above into something that R can analyze, MiniJudgeJS needs to match up each judgment with the factor (or factors) you are testing. The key to do this is in the schematic survey info that you saved earlier.
Paste the schematic survey info into the window below, then click OK.
PASTE SCHEMATIC SURVEY INFO HERE
Add neighborhood densities. (Optional, for judgments of word-sized items) SKIP
Acceptability judgments of words-sized nonlexical items are strongly affected by analogy with neighboring (i.e. similar) lexical items. To factor out such effects, click HERE to let MiniCorp calculate neighborhood densities for your items. (You will need the Master List file for your experiment and a corpus of all lexical items in the language.) Otherwise, click HERE to skip this step.
In the window below, paste in the neighborhood density file created using MiniCorp. Then click OK to add neighborhood densities to the data file.
ENTER NEIGHBORHOOD DENSITIES HERE
Save data file.
If all went well, the window below should now show the R-friendly data file, ready for analysis (the columns may not seem to line up, but they really do). Copy and save it as a text file.
COPY DATA FILE FROM HERE
Generate R code.
In order to run the analysis, R will need to refer the full name of the data file that you just saved. NOTE: R treats any filename extension (e.g. ".txt") as part of the filename. However, by default, Windows does not show the filename extension in directories. So if you're using Windows, be sure to include ".txt" at the end of your filename even if you don't see it when you display the file in a directory. (Mac users shouldn't have to worry about this.)
You must also decide if you want R to test for satiation, where judgment contrasts weaken over time (there may also be "anti-satiation", where judgment contrasts strengthen). Satiation has been taken by some to provide a new window into competence, performance, or their relationship, but testing it in MiniJudgeJS may have counterintuitive (though mathematically correct) effects on the rest of your analysis.
Finally, if your experiment involves word-sized items and you have already added neighborhood density information to the data file, you must decide if you want R to factor out effects due to neighborhood density.
Enter the name of your data file in the box below. Check the boxes to adjust the analysis as desired. Then click OK to generate the R code.
Name of data file:
Test for satiation:
Factor out neighbors:
Paste R command code into R.
If all has gone well, by now MiniJudgeJS should have generated the R command code needed to analyze your experimental results. When you paste this code into the R program window*, it will automatically test the significance of the main factor(s), their interaction (if two factors), and order. If you decided to test for satiation, R will also look for interactions between order and the main factor(s). For experiments using word-sized items, R will also factor out the effects of neighborhood density if you requested this.
NOTE: As of version 1.1, the R code generated by MiniJudgeJS refers to "Item" rather than "Sentence"; change the relevant header in older data files before analyzing them.
The analysis will be summarized in a brief, easy-to-read format. This is based on R's own detailed but technical format, which will be automatically saved in a file in the same folder as your data file.
To use the command code, start the R program on your computer. Then you must change R's directory to the location of your data file using R's FILE menu. Then simply copy and paste the code below into the R window*. If this is your first time using MiniJudgeJS, R will first ask you to download the key package. R will then pause for several seconds while loading it into the computer's memory.
*Pasting the R code may not work in Linux. In this case, first save the R code in a file (e.g. mjRcode), and then instead of pasting it into R, use R's source function (e.g. source("mjRcode")).
COPY R COMMAND CODE FROM HERE
After running the above code, the statistical analysis of your data should be complete.
Significant effects can be positive or negative. A positive effect for a binary factor means that the probability of getting a "1" judgment is significantly higher for the [+] value of the factor; a negative effect means the same for the [-] value. A positive effect for order means that judgments get better over the course of the experiment; a negative effect means they got worse. An interaction between some factor and order means satiation: the judgment contrast for that factor changed over the course of the experiment. The actual sign of an interaction between order and a factor, or between factors, depends partly on how you defined your factors (i.e. whether [+F] is grammatical or ungrammatical, or whether its grammaticality depends on the value of factor [G]).
For help understanding your statistical results, click HERE.
Version information. MiniJudgeJS 0.9 was the first attempt at a JavaScript implementation of minimalist experimental syntax. The first version of MiniJudge was written in Excel VBA, believe it or not.
MiniJudgeJS 0.9.1 fixed some bugs relating to compatibility with Internet Explorer.
MiniJudgeJS 0.9.5 fixed bugs relating to capitalization, punctuation, and pattern matching in the segmentation of the prototype items, required the user to choose factor names compatible with R, added R code for doing "by-item" analyses, signaled acceptance of inputs with shading rather than annoying alert boxes, and permited keyboard-based navigation in some windows. I thank my students for suggesting some of these changes.
MiniJudgeJS 0.9.9 differed from version 0.9.5 in the following ways:
MiniJudgeJS 1.0 differed from version 0.9.9 in the following ways:
MiniJudgeJS 1.1 differed from version 1.0 in the following ways:
MiniJudgeJS 1.1.1 differs from version 1.1 in the following ways:
MiniJudgeJS 1.1.2 differs from version 1.1.1 in the following ways:
There are other fancier versions of MiniJudge in Java, which you can try HERE. Unfortunately they still use the old version of the R code, so... they probably don't work. Does anybody who knows Java want to help fix them?
Some users may also be interested in MiniJudgeJS Deluxe. It's just like ordinary MiniJudgeJS, except that it costs US$375 and requires a dongle. Click HERE for information.
Citing MiniJudgeJS: Here's how to do it in APA style:
Myers, James. (2018). MiniJudgeJS (Version 1.1.2) [Computer software]. Retrieved from http://personal.ccu.edu.tw/MiniJudgeJS.htm.
The MiniGram main page has other relevant references.
Citing R:
R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
Citing lme4 [R package for GLMM]:
Douglas Bates, Martin Maechler, Ben Bolker, Steve Walker (2015). Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67(1), 1-48. doi:10.18637/jss.v067.i01.
Contact James Myers with your questions and comments.