MiniJudgeJava Help Page
Version 2.0.0 beta - Last updated on September 28, 2011
Step-by-step manual
More background information
Steps
Set up MiniJudgeJava
System requirements
Language setting
Mail setting
Server setting
R setting
Sentence lists
Create a new list
Decide factor names and description
Make protosets
Enter prototype items
Extra sets
Input other sets
Segment prototype set
(optional)
Other managements
Edit/Delete existent lists
Export
existent lists
Fillers
Instructions
Experiments
Add/Edit/Delete experiment
Run experiment
Through printed surveys
Through MiniJudgeJava
Through MiniJudgeJava's online survey
system
Analysis
Download
surveys (optional)
Save results (optional)
Analyze results
Set up MiniJudgeJava
Before starting to use MiniJudgeJava, some
steps have to be taken in order to use MiniJudge with its full
function. Below is a step-by-step instruction to install programs
MiniJudgeJava requires and set up MiniJudge in the way you prefer.
System requirements
MiniJudgeJava requires the installation of Java Runtime Environment (JRE) and the powerful yet free statistical software R.
MiniJudgeJava is written in Java. To run a Java program, the installation of JRE is necessary. To know if JRE
has been installed in the computer in use, go to this
page. If it shows that JRE has not been installed, go to this page to download JRE. It is totally free, but be cautious that it installs a search engine tool bar with JRE in default, uncheck this to avoid the tool bar from being installed.
MiniJudgeJava lets R do all the statistics. If you want MiniJudgeJava to analyze the experimental results for you, this is a software you
must have. To install R,
please go to this
page. The installation should be straightforward, but in case anyone needs, here is an installation guideline.
If JRE and
R are successfully installed, you may now open MiniJudgeJava and go to "Preferences" page.
Language
setting
MiniJudgeJava now offers two language options, English (United States) and Traditional Chinese (Taiwan). Choose the language you prefer in "Language Setting".
Note that if you change the language, MiniJudgeJava will require you to
restart the program to apply for the change. If anyone is interested in
helping translate MiniJudgeJava into other languages, please contact us.
Mail
settings
MiniJudgeJava provides the function of distributing judgment surveys to
speakers through e-mail. To use this function, you have to provide
their account information to MiniJudgeJava's built-in mailing client.
These information are usually availible in the webpage of e-mail
service providers (e.g. Gmail). Below uses Gmail as an example.
The
content of e-mail should be typed in the area below "Mail Content Template". It should
include a link to the on-line survey page,
represented as "$link". Each
link is assigned to a speaker through this link. There should also be a
brief description of
the experiment to the speakers, so that they won't consider these
e-mails as spams or feel bothered.
Server
settings
If you want generate surveys and invite speakers to finish these
surveys
online, you have to specify the link in "Server
setting" and let MiniJudgeJava connect to the server. The
server is
located at: http://72.14.189.57/MiniJudgeGWT/.
Please copy and paste to the area for server URL.
R settings
Two directories in R settings have to be set, one is the
location of
R.exe, the core of R, the other is the location where they
want to save result files. R.exe is usually in the "bin" folder under
"R-x.x.x" folder ("x"s indicate digits of version number). The
result files can be put in whichever folder users prefer. The directory
of result folder has a default setting at C:\, while the directory of
R.exe has no default setting. Users
must set its directory before running statistical analysis on MiniJudge.
Also, users can optionally choose the statistical model with which
they want to analyze their experimental results. MiniJudgeJava now
offers two kinds of statisical anlayses for users to choose, both of
which are logistic mixed-effect regression models (see here
for more information). The difference between them is that the one
testing for satiation factors out nuisance caused by presentation order
in addition to the intended factors, while the other only includes the
factor(s) users intend to test. To choose the analysis including
presentation order, check "test for
satiation", otherwise leave it unchecked.
If users want to run a different analysis through R with MiniJudgeJava,
they can load their own R commands to MiniJudgeJava by click on Set. An alternative
option is that users modify the R commands (test for satiation/no test for satiation) of analyses avaliable in
MiniJudgeJava.
If everything is set, Users may click Apply
to save and
validate the setting. Note that some changes are only
applicable after users restart the program. If the setting is
successfully applied, please go to Sentence
lists page.
Sentence
lists
MiniJudgeJava has a separate page listing all of the sentence
lists avaliable. In this page, users are allowed to create a new
sentence list, edit or delete the existent sentence lists and export
sentence lists if users want to save them elsewhere for other
purpose.
A sentence list in MiniJudgeJava means a collection of
sentences designed for a specific purpose. In this page, users are able
to find out some information about a specific sentence list (e.g. name
of factors, number of sentences, and a short description on what this
list of sentences is about).
Note that in MiniJudgeJava, different sentence lists represent
different experiments, but not vice versa., since there may be
differences in other parameters (e.g. task/participant-related
parameters) that make two experiments different.
Create a new
list
To start from scratch, click "Add a
new list" to
create a
new sentence list.
Decide factors and
description
To help isolate interesting factors from nuisance factors,
MiniJudgeJava
generates sets of items in accordance with a factorial
design. To make this process as simple as
possible, item sets are generated based on a "prototype set",
similar to the example sets cited in syntax papers.
Name one or two binary
factors, representing the syntactic, lexical, or other variables
differentiating the sentences in your sets. You have to enter your own
factor name(s) according to your research topic. You can choose factors
that
you predict to be directly relevant to grammar (e.g. "AdjunctExtract"
in an experiment on adjunct islands), but you may also include a
performance factor
(e.g., "CenterEmbed" for center-embedded structures).
NOTE: If
you have two factors, you
must choose names starting with
different characters,
since they'll be abbreviated in some of the tables. Also, factor
names may not contain
spaces, since this will cause trouble for the R analysis.
After
you have decided the factor(s), please also add a very brief
description. It will not only help you remember what this list is
about, but give an identity to it for further
use.
If you use two factors,
MiniJudgeJava will not only test if they show main
effects, but also if there is an interaction.
For example, you would expect the factors AdjunctExtract and
CenterEmbed both to show main
effects, but you would not expect an interaction (since they presumably
involve independent "modules," namely competence vs. performance).
By contrast, if your factors were Adjunct and WhMove, you wouldn't
expect either to
show main effects (since adjunct structures and wh-movement are both
grammatical),
but they should show an interaction (since wh-movement from adjuncts
should be bad,
while wh-movement from non-adjuncts should be good).
Enter your factor(s), then click "OK".
You should see the sentence list you just create is on the list in "Sentence lists". Double-click it to go "List Management".
If you
want to reset your factor names, click "Cancel".
Make
protosets
Enter prototype items
As with a set of examples
in a syntax paper, the prototype items should be matched as much as
possible for irrelevant properties like word choice, so that they
differ only in terms of the factor(s) that are under investigation.
For example, an sentence set with the
factors ComplexNP and WhMove might have the
following prototype sentences:
| [+ComplexNP, +WhMove] |
Who
did you hear the rumor that Mary kissed? |
| [+ComplexNP,
-WhMove] |
You
heard the rumor that Mary kissed John. |
| [-ComplexNP,
+WhMove] |
Who
did you hear that Mary kissed? |
| [-ComplexNP,
-WhMove] |
You
heard that Mary kissed John. |
The above sentence set illustrates how a protoset in MiniJudgeJava
should be. To input the above set in MiniJudgeJava, click "Add" in "Prototype sentences". There
should be two or four combinations of factor values and the same number
of blank areas, each with a factor value. Fill in
the
corresponding sentence in the blank area and click "OK". Note that for
each sentence list, there should be ONLY
ONE PROTOSET. If you
find your protoset problematic, double-click
the column listing the protoset to modify, or one-click it and click "Delete" to delete the
existent protoset and start from scratch.
It you are satisfied with the current protoset, you can start
to create extra sentence sets.
Extra
sets
Input
other sets
MiniJudgeJava have two ways to generate new sentences. By
clicking "Add" in
"Experimental sentences",
you can create a sentence set all by yourself, as what is done in
creating the protoset. But in case you are worried about your results
being contaminated by unexpected non-structural properties, you can ask
MiniJudgeJava to segment the protoset into the largest repeated strings
to create structurally identical sentence sets.
Segment prototype set
MiniJudgeJava can help you create
multiple item sets. This
semi-automatic operation takes three steps. First, MiniJudgeJava
will split sentences of the protoset into strings. Users then should
input the syntactic equivalents of
these strings. Finally, MiniJudgeJava will put these new
strings
back to the positions of their syntactic equivalents. Since
MiniJudgeJava does not know human languages, it might make mistakes. It
is thus recommended to double-check the outcome of each step. Also,
consulting a thesaurus for syntactic equivalents should make the work a
lot easier, see HERE
for more information.
To start, click "Replicating
sentences from prototypes". You'll see a window including
protoset
and an option to choose segmentation method. The "English" method is
generally applicable to languages using space as word boundary (e.g.
English, German, French, etc.), while "Chinese" method is applicable to
languages without apparent word boundary (e.g. Chinese, Japanese,
Korean). After setting the preferred segmentation method, click "Next" to
proceed. It will take a few seconds for MiniJudge to output word
strings.
At this step, MiniJudgeJava automatically
segments the protoset
into their longest repeating segments to make it easier to replace them
systematically when
creating additional sentence sets. If you want to force your own
segmentation, type "|" inside
your sentences. For example, "The | banana is | here" will then be
segmented into "the",
"banana is", and "here".
For example, given the prototype set below right,
MiniJudgeJava extracts
the prototype segments below left. These are the largest repeated
strings in
the prototype sets.
|
who did
you
hear
the rumor
that Mary kissed
heard
John
|
Who
did you
hear
the rumor
that Mary
kissed?
You
heard
the rumor
that Mary
kissed John.
Who did you hear that Mary kissed?
You
heard
that
Mary kissed
John.
|
Note that whenever you change the output of
segmentation, click "segmentation" to
apply for changes.
If you are satisfied with the result of segmenation, by
clicking "Next",
you'll see a new window pop out. In this window, there is a two-column
table, with the left column listing strings from the protoset. Fill the
blank with the syntactic equivalents of the existent strings. You will
see the replacement be applied below as soon as you input the string
and press enter.When you are done with all of the replacements, click "Next" to proceed.
Now,
you should see a new sentence set whose structrues are identical to the
protoset but with different lexical items. You can still modify
sentences at this step, but be cautious that any modification at this
moment may cause unexpected structural differences that confound with
the original design.When you are totally sure that you are done with
this set, click "Add
more sets" to start to create a new set if you want more,
or click "Finish"
if you think you have had enough. The ideal number of sentence sets in
a sentence list is 5 - 20.
If you click "Finish",
you will go back to "List
Management" again. This time, you will see the sentence
sets you just added, along with protoset, are listed in "Add prototype sentences".
If these are all you need for this list, close this window to go back
to the main page.
Other
managements
Edit/delete
existent lists
You can still
modify the factor names
and/or description to make them more properly describe your sentence
sets. Single-click any
sentence list then click "Edit" in
"Sentence list" to
modify these names, then click "OK"
to apply for the changes.
If you find some
sentence list flawed
or won't be used anymore, you can also choose to delete these sentence
lists by clicking "Delete".
Export
existent lists
Although MiniJudgeJava will automatically save the sentence
lists
users create, you still save the lists in separate files for your own
record. To do this, click "Export",
choose the directory of file and specify the filename, then click "save".
There is no default file extention, you can specify the file extention
you want (e.g. .txt) by suffixing it to the filename or not specify at
all. Either way, you can open the file with basically any kind of text
editor.
Fillers
Including fillers or not can be a deliema. Some have
been arguing that without using fillers, the purpose of the experiment
will become obvious for the participants and they might adopt some
judgment strategies in the experiment toward or against such purpose.
However, including fillers or not does not make a huge influence in
practice. Cowart (1997) reports that including/excluding fillers only
results in a shift in the scale, instead of the relative differences
among experimental conditions. Thus, whether to follow the standard
psycholinguistic parctice to create fillers that are two or three times
more than experimental itmes becomes a question.
Since MiniJudgeJava is an implementation of small-scale
experimentation, it does not make fillers an essential
property in an experiment. You have the right to decide to include
fillers or not. In addition, MiniJudgeJava provides a way of speeding
up the process of creating fillers. Specifically, MiniJudgeJava allows
users to compile a list of various sentences and randomly pool a
certain amount of sentences from this list as fillers when setting up
an experiments.
To add fillers in your experiment, go to "Filler" page and
click "Add",
then paste or type your filler sentences in the blank area. Note that
filler sentences should be of various kind, so the filler
list should include as many sentences of different structures as
possible.
Instructions
In the current version
of MiniJudgeJava, responses must be binary judgments
(see HERE
for justification of this limitation). But since MiniJudgeJava allows
customized names for response categories, different experiments may
name response categories differently. This must be described clearly in
the instruction. Also, the instruction should also be modified
according to how the experimentors want to run their experiments.
Participants may take a pencil-paper test, an on-line survey, or take
the experiment under your supervision. The
instructions for pencil-paper-styled experiments or on-line experiments
should thus be more detailed, while the instruction of experiments run
under experimentor's supervision can be complemented with oral
description. A sample instruction is avaliable HERE. You can modify it
according to the specific requirements of your experiment.
Note
that for the display reason, there are limits on the number of lines
and number of letters per line. The instruction should be no more than 18 lines and 120 letters per
line, or part of the instruction cannot be seen by the speakers.
Since MiniJudgeJava allows users to save multiple versions of
instruction, you can go to "Instructions"
page and click "Add"
to add a new instruction. You have to give a name to this instruction
in "Title"
and type/paste your instruction to "Content"
and then click "OK".
If everything goes well, you will see the name of this instruction
appear on the list in "Instructions"
page.
Experiments
To create and run an experiment with MiniJudgeJava, you have to go
to the "Experiments" page to
set relevant parameters first. Then you can choose how you are going to
distribute each copy of your surveys to speakers. After you have
collected a certain number of surveys, you can analyze these surveys to
see if your predictions are borne out.
Add/Edit/Delete experiment
The first thing to do is to add an experiment in
MiniJudgeJava. To do this, click "Add experiment". MiniJudge will then
require set up the following parameters:
| Experiment name: |
Name
your experiment in any way you want, except that it shouldn't be
repeated with the name of other experiments. |
| Positive answer: |
Name
the positive response in your experiment (e.g. acceptable, grammatical,
natural, etc.). |
| Negative naswer: |
Name
the negative response in your experiment (e.g. unacceptable,
ungrammatical, unnatural, etc.). |
| Select
sentence list: |
Select a sentence list
from existent lists. Sentences in this set will be the
experimental sentences of your experiment. |
| Fillers: |
Specify
a number, then MiniJudge will randomly draw this number of sentences
from filler list as fillers. According to Cowart (1997), the number of
fillers should be two times as many as that of experimental sentences. |
| Number of
surveys: |
The
ideal number will be 10 to 25, and you can choose to add more afterward. |
| Instruction: |
Select
an instruction that is proper to your experiment. |
Click "OK"
to create a new experiment if all parameters are set. Now you should
find the experiment you just create on the list in "Experiment lists" section
in the upper part of "Experiments".
You should also see some properties of this experiment (number of
fillers, experimental sentences, surveys, etc.).
Note: Once
an experiment has been created, further changes can hardly be made, so
make
sure you have everything correctly set up. The
only changable property of an experiment is its name. If you need to
change it (to avoid repetitive names or for better description), select
the experiment and click "Edit".
If you want to totally abandon an experiment, select the experiment and
click "Delete".
By clicking on your experiment on the list, you should see that the "Survey"
section below list a certain copies of surveys and some information
about them (e.g. state, build date, experiment date, etc.). If you have
not run any of them, their state should all be "New". Click on any one
of the survey and then click "Survey
preview",
you will see how the materials are ordered and whether a sentence is
filler or not. Sentences in different surveys are ordered randomly.
Randomization is only partial to ensure that
material types are spread evenly within each survey (click HERE
for more on the algorithm used). Randomization and even spread have
another benefit, explained HERE.
Run
experiment
Now you should have your experiment ready, and you can finally run the
experiment! MiniJudgeJava provides three ways of running experiments.
The least technical one is to run a paper-pencil test by printing
out surveys for speakers to complete. But if you are too lazy to key in
data for fruther analysis, you can invite speakers to your place and
ask them to make judgments through the built-in tool of
MiniJudgeJava. If you have problem meeting speakers in person (e.g.
speakers of that language live in a remote country), you can still send
them an e-mail with a link to MiniJudgeJava's online survey
system.
Note that there is no rule saying one experiment
can only be done in one way, you are free to run an experiment with all
of the three ways described above. But be cautious that this is made
clear in the instruction so that speakers will not be misled.
Through
printed
surveys
Select a copy of survey and click "Print"
to print it out. After clicking, you should see a new window with
survey preview. Each survey should contain instruction, subject number
and experimental sentences mixed with fillers (if any). By clicking "Print"
again, the printing interface of your printer will appear. Following
its instruction, you should be able to have your surveys printed and
distribute to your speakers.
Running experiments with printed surveys requires one more step,
namely, filling in speakers' judgments. You can click "Fill in judgments"
to fill in judgments according to speakers' responses. Once you finish
this step, the state of this survey will become "COMPLETED".
Through
MiniJudgeJava
You can also invite speakers to your place and run an
experiment
through MiniJudgeJava. The procedure should be like that of a
full-fledged psycholinguistic experiment. By clicking "Run",
a box will appear to ask the experimentor to enter an ID for a
particular speaker. After that, the experimenter should let the speaker
control the pace of the experiment. The speaker should decide
when
to start the experiment after reading & hearing instruction.
After
reading the instruction, he/she should start to make judgments one
after another until the end of the experiment or he/she decides to
quit, but the speakers have to finish judging one sentence before
proceeding to another.
After he/she finishes the experiment, you should
click "Save the results"
or the results will not be saved. You should make sure the results are
saved by checking if the state of this survey has become "COMPLETED".
Through
MiniJudgeJava's
on-line survey system
Another
way for speakers to do your surveys is to invite them to
MiniJudgeJava's on-line survey system and finish the surveys
on-line. Then you can download their judgments to your
computer and analyze the results. Before starting, make sure you have
set up the mailing settings and
server address correctly, or you
will experience difficulties in the following steps.
To send an invitation, choose a survey which is not completed yet and
click "E-mail". A new window will
pop out. Relevant information, like the password of your e-mail
account, the e-mail address of the intended speaker, and the subject of
this e-mail should be filled in. The description and link of the survey
will also be shown in the area below these information. If you don't
see a link starting with "http://72.14.189.57/...", you have to check
if you add "$link" HERE. You can also modify the
description of your survey here, but make sure that you give the same
information to all of the speakers you invite. If everything is set,
click "Send"
to send out this survey.
After receiving this e-mail, speakers can link to
MiniJudgeJava's on-line survey system and complete a survey only for
him/her. The procedure is exactly the same as that of running the
survey through MiniJudge except without the supervision of the
experimentor and the survey is presented through the speaker's webpage
browser. The compatibility of the web-based surveys is attested in Microsoft
Internet Explorer, Mozilla
Firefox, and Google
Chrome, and should be compatible with other browsers.
Analysis
The data collected with MiniJudgeJava are categorical
(binary) and repeated-measures
data
(i.e. each speaker gives multiple judgments). This type of data
requires a highly
sophisticated kind of statistics
and was hard to implement in previous versions of MiniJudge. However,
the complexity has been largely reduced in current version of
MiniJudgeJava. The users don't have to open R and paste R commands to R
anymore. As long as R is installed, users can have their results
analyzed through one click.
Download
surveys (optional)
If you choose to invite some/all of your speakers to
MinJudgeJava's on-line survey system to complete surveys, you
have to download the completed surveys before running any
statistical analyses. Make sure that your computer is connected to the
Internet, then click "Download
Surveys", MiniJudgeJava will download the surveys that
have been completed by the speakers for you. You'll see a message
saying "Download
completed." when all of the completed surveys have been
downloaded to your computer.
Save
Results (optional)
When you have collected a certain amount of judgments , you might want
to save these judgments in a separate file for the concern of
information security or other softwares to access and analyze the
results. To do so, click on the experiment of which you want to save
the results, then click the "Save
data file" button above. A window should pop out and ask
you to specify the location and filename of your result file. Note the
result file is saved as a text file, you can open with any word
processing software (e.g. WordPad). You can also specify its file
extention if necessary.
If you open the result file, you'll see the data are listed in
follwoing columns.
Speaker:
A numerical value assigned to each speaker.
Item:
A numerical value assigned to each stimulus.
Set:
A numerical value assigned to each set of stimuli that are
minimally different.
Order:
The presentation order of a stimulus in an experimental
session
Factor1:
The first factor of the experiment, coded with effect coding.
Factor2: The
second factor of the experiment, coded with effect coding
Judgment :
Judgments from the speakers, where "1" indicates the choice on the
positive category, while "0" indicates the choice on the negative
category.
Analyze
Results
Since you have had the experimental results in hand, the next step is
see whether there is any pattern in your results and whether the
pattern you found reaches statistical significance.
MiniJudgeJava tests the results with a mixed-effect logistic
regression model (see here
for a brief discussion). Though the computation of this statistical
model is of high complexiy, it will not bother the users of
MiniJudgeJava. After running the analysis, MiniJudgeJava will generate
a brief report of statistical results in plain language, along with a
graphic illustration that helps clarify the statistical pattern.
The current version of MiniJudgeJava provides two options for
statistical analysis. If the
"test for satiation" box in R
setting is checked, MiniJudgeJava will analyze the effect of Order in
addition to the experimental factors. If the boxed is not checked, then
MiniJudgeJava will analyze effects of the
experimental factors only.
To start the analysis, first click on the experiment you want to
analyze, then click "Analyze
results" button. Normally, the button will become
highlighted for a few second, then results of the statistical analysis
and figure will be displayed on the screen. If you find that there is
no response for button-clicking, check your R
setting
to make sure you have set up the right directories to R.exe and the
result folder. After the analysis is done, MiniJudgeJava will generate
a brief report of statistical results and a graph illustrating judgment
patterns. You can also find these files saved in the result folder you set previously. Also, you will see another file
containing details of statistical analysis. To know what exactly it
says, check here.
Illustrations of judgment trends and brief
statistical report (above: no test for satiation, below: including test
for satiation)
Version
information. MiniJudgeJava 2.0.0 is evolved from
MiniJudgeJava 1.0. It has been totally re-written and adds
several new functions:
- using a more intuitive interface of generating new items.
- adding the option of including fillers.
- adding a built-in interface of running experiments
- revising the method of running experiments from an remote end
- adding more options in statistical analyses
- letting the program directly communicate with R, reducing the
complexity of running statistical analyses
The first
version of
MiniJudge was written in Excel VBA, believe it or not. There is another
version of MiniJudge in JavaScript; go to
"http://www.ccunix.ccu.edu.tw/~lngproc/MiniJudgeJS.htm"
to give it a try.
Some users may also be
interested in MiniJudgeJava Deluxe. It's just like
ordinary MiniJudgeJava, except that it costs US$375 and requires a
dongle. Click HERE
for information.
Citing MiniJudgeJava:
Myers, James and OmniWise International Incorporation. (2010).
MiniJudgeJava (Version 2.0) [Computer software]. Retrieved from
http://www.ccunix.ccu.edu.tw/~lngproc/MiniJudgeJava.htm.
Citing R:
R Development Core Team (2009).
R: A language and environment for statistical computing [Computer software]. R Foundation for Statistical
Computing, Vienna, Austria. ISBN 3-900051-07-0. Retrieved from http://www.R-project.org.
Citing lme4 [R package for GLMM]:
Douglas Bates and Martin Maechler (2009). lme4: Linear mixed-effects
models using S4 classes [Computer software]. R package version
0.999375-31.
Help pages written by Yuguang Ko
(mailto: edk0421 at
gmail.com).