To get an idea of what the software produces, see the example screenshots.
The major steps for using this software are outlined below. Not all of them must be done; it depends on what aspects of the reviewing process you wish to automate. This software is intended to be easily customizable along several dimensions.
Note: as of 12/9/1999 I have not tested what happens when you make major customizations, such as omitting topics, not having meta-reviewers, or having more than one meta-reviewer per paper. I have tested the system using the default settings.
Table of Contents:
To use the software, you need to have available the basic Unix commands, Tcl 8.0, and a special Tcl package called cgi.tcl (written by Don Libes). All of this software is free. Information about Tcl8.0 can be found at http://www.scriptics.com/man/tcl8.0/contents.htm. Information about cgi.tcl, and the software package itself, can be found at http://expect.nist.gov/cgi.tcl
This software package does not use a database backend. Instead, all data is stored in Unix directories as clear ascii text. This decision has both strengths and a weaknesses. The strengths are that it therefore requires no propriety software, is fast, and the data is easily inspected. The weakness is that there is the potential to occassionally lose information when the server is writing to disk, and you can't issue sql queries against the data. However, I have written routines that convert data into a tabular form that can easily be read into a speadsheet or database system. Sample datafiles appear in the example-data directory.
The software should be placed in a cgi-bin directory that allows execution of cgi files by the web server. There must also be a data directory that is writeable by the web server. There should be two subdirectories within this directory, and they must also be writeable by the web server. Create the directories:
data/reviewers
data/submissions
Only one source file requires customization. This file is in src/global/definitions.tcl.
Tclsh must be installed in /usr/local/bin/tclsh. If it is installed elsewhere, the first line of all the *.cgi files must be modified to reflect this.
The following variables must be adjusted in definitions.tcl:
A set of test data is provided to let you check to see if things are working.
set laf(title,reviews) "SIGIR 99 Paper Review Information"
set laf(title,reviewers) "SIGIR 99 Reviewer Information"
set laf(title,submissions) "SIGIR 99 Submission Information"
Colors and fonts for tables, e.g.:
set table(header_color) navy
set table(header_font_color) white
set table(entry_color) "#FFFFCC"
set table(alt_entry_color) "pink"
The software package provides facilities for the following:
The function below creates a file called global/reviewerinfo.tcl that contains reviewer IDs and automatically assigns a randomly selected password to the reviewer. The password has the format word-digit-word where the words are of length 4-7.
If you want to have both first-tier reviewers and meta-reviewers, make one file for each reviewer type. You must use full pathnames unless executing in the directory containing the data file. If you want the same person to be both a meta-reviewer and a first-tier reviewer, you must give that person two separate IDs and place one ID in each of the two files.
Then follow these steps:
% cd ../src/util
% tclsh
> source prepreviewer.tcl
> create_meta_and_ftr_reviewer_entries "ftr_filename" "meta_filename"
> exit
% cd ../util
% tclsh
> source prepreviewer.tcl
> create_reviewer_entries "reviewer_filename"
> exit
If you want to add or remove an attribute, you must modify reviewerinfo(reviewerattributes) to add or remove the attribute's name. The default setting is:
set reviewerinfo(reviewerattributes) {fname lname email affil address country zip phone fax topics}
You must also add or remove the information that describes how an attribute is presented in the web pages. These attributes are stored in the rrattributes variable. Associated with all attributes is a variable called msg that indicates the label that is shown alongside the display. There are three possible data types for attributes: text, integer, and llist. There are four display types: textbox, textlist, radiobuttons, and checkbox. Certain display types have other variables that can be set. Textline can specify linelength, textbox can specify numrows and numcols. For example:
set rrattributes(address,displaytype) textbox
set rrattributes(topics,displaytype) checkbox
There is a special attribute called displayexceptions which is used to
restrict certain attributes from being shown on the main edit page.
The excepted attributes are shown on alternative pages. This allows
the breaking up of a large amount of information into separate web
pages, so the user does not have to scroll as much and is less likely
to lose partly finished work. For example, in the reviewer
information code, the topics attribute is not shown on the main edit
page, but rather has a separate page.
set rrattributes(fname,displaytype) textline
set rrattributes(fname,datatype) text
set rrattributes(fname,msg) "First Name"
set rrattributes(address,numrows) 4
set rrattributes(address,datatype) text
set rrattributes(address,msg) "Address for receiving express mail packages"
set rrattributes(topics,datatype) llist
set rrattributes(topics,msg) "Topics"
The topics can be customized as well. See customizing topics.
If you do not want reviewers to select among topic choices you need to change the notopics customization variable in definitions.tcl as follows:
set customizations(reviewers,notopics) 1
Reminder: the cgi files must appear in a cgi-bin directory readable by the web server in order for the cgi script to be executed.
To aid in system testing, the software generates reviewers named dummyftr and dummymeta along with the real reviewer IDs. These are convenient for testing purposes. You have to check the automatically generated password in reviewerinfo.tcl to access them.
When a reviewer logs into the system for the first time, a directory is created in data_directory_path/reviewers/ that is named after the reviewer ID. A file named contact.back.tcl is placed in this directory. This file contains the default (empty) settings for all of the reviewer's contact information. When the reviewer updates their contact information, the new information is placed in a file called contact.tcl and the old information shifted to contact.back.tcl. It is important not to modify these files by hand once they are created; only the system should modify them.
global reviewerinfo
It must contain the following lines in this format for every reviewer:
set reviewerinfo(reviewerid,name) "Firstname Lastname"
set reviewerinfo(reviewerid,email) "email@email.address"
You must also modify the message in the email_reviewer_password function in src/util/email.tcl to reflect your conference information. Warning: you should test this function out by first setting email address to be your own, to make sure it is sending the correct information.
Once this is done, you can run the following program.
% cd src/util
% tclsh
> src email.tcl
> email_reviewer_passwords
> exit
The software package provides facilities for the following:
Obtaining submission information is very similar to reviewer information, with one notable difference. This difference is that the software assumes you do not know in advance who will be submitting papers and how many papers will be submitted. Therefore, you cannot preassign IDs and passwords. Instead, when a submitter enters their name in the first screen, they are assigned an ID which consists of that person's last name concatenated with a 4-digit randomized number. If the submitter later wants to change their submission they must have remembered that ID; otherwise they need to create a fresh submission record. This setup allows the same submitter to submit more than one paper.
A problem with combining online submission information with offline paper submissions is reconciliation of the submission ID with the paper version. Some submitters may enter submission information online but not end up sending in a submission; others may send a physical paper without using the online system. At some point reconciliation between paper and electronic IDs has to happen.
(Future versions of this software may add provisions for electronic submissions, but these have similar issues.)
This software provides a mechanism to allow for the reconciliation between the paper ID and the online submission ID.
If you want to add or remove an attribute, you must modify submissioninfo(submissionattributes) to add or remove the attribute's name. The default attributes are:
set submissioninfo(submissionattributes) {fname lname email affil address country zip phone fax \
ptitle pabstract topics \
coname1 coemail1 coaffil1 \
coname2 coemail2 coaffil2 \
coname3 coemail3 coaffil3 \
coname4 coemail4 coaffil4 \
coname5 coemail5 coaffil5 \
coname6 coemail6 coaffil6}
You must also add or remove the information that describes how an attribute is presented in the web pages. These attributes are stored in the sattributes variable. Associated with all attributes is a variable called msg that indicates the label that is shown alongside the display. There are three possible data types for attributes: text, integer, and llist. There are four display types: textbox, textlist, radiobuttons, and checkbox. Certain display types have other variables that can be set. Textline can specify linelenght, textbox can specify numrows and numcols. For example:
set sattributes(ptitle,displaytype) textline
set sattributes(pabstract,displaytype) textbox
There is a special attribute called displayexceptions which is used to
restrict certain attributes from being shown on the main edit page.
The excepted attributes are shown on alternative pages. This allows
the breaking up of a large amount of information into separate web
pages, so the user does not have to scroll as much and is less likely
to lose partly finished work. For example, in the submission
information code, the topics attribute and the coauthor attributes are
not shown on the main edit page, but rather have separate pages.
set sattributes(fname,displaytype) textline
set sattributes(fname,datatype) text
set sattributes(fname,msg) "First Name"
set sattributes(ptitle,datatype) text
set sattributes(ptitle,linelength) 60
set sattributes(ptitle,msg) "Title of Submission"
set sattributes(pabstract,datatype) text
set sattributes(pabstract,numrows) 6
set sattributes(pabstract,msg) "Submission Abstract"
The topics can be customized as well. See customizing topics.
If you do not want submitters to select among topic choices, or do not want them to specify coauthors you need to change the notopics and/or nocoauthors customization variable in definitions.tcl as follows:
set customizations(submissions,notopics) 1
set customizations(submissions,nocoauthors) 1
Note that it must appear in the cgi-bin directory readable by the web server in order for the cgi script to be executed.
When a submitter enters information into the system for the first time, a directory is created in data_directory_path/submissions/ that is named after the submission ID. A file named contact.back.tcl is placed in this directory. This file contains the default (empty) settings for all of the submitters's contact information. When the submitter updates their contact information, the new information is placed in a file called contact.tcl and the old information shifted to contact.back.tcl. It is important not to modify these files by hand once they are created; only the system should modify them.
Once the submission deadline is over you must reconcile the paper versions with the electronic IDs. How to do this is described below, in Assigning Submissions to Reviewers.
% cd src/util
% tclsh
> source email.tcl
> email_submission_acks
> exit
You may want to edit the function to substitute in your email address in order to test it out first. The steps to accomplish
The following steps are necessary:
This software package provides routines that will create a file, called ../global/reviewinfo.tcl, that contains the necessary information for the reviewing stage. To set things up you need to complete the following steps.
First, you must decide on a directory for storing temporary results. I suggest you create a directory called "tmpdir" in src/util. Then the argument directory in the commands below can be set to the relative directory name "tmpdir".
Next, you must create a file that links submission IDs to paper IDs. That file must consist of a list of submissionID/paperID pairs, separated by whitespace, as shown below:
% cd src/util
% tclsh
> source assign.tcl
> gather_submissions_info filename tmpdir
> exit
This should place a file called stemp.tcl in tmpdir.
Next you need to create a temporary file of reviewer information (this contains reviewer topic information):
% cd src/util
% tclsh
> source assign.tcl
> gather_reviewer_info tmpdir
> exit
This should place a file called rtemp.tcl in tmpdir.
Next, you must create a file that links submission IDs to metareviewer IDs (the philosophy being that metareviewers should be assigned by hand). If you are not going to use metareviewers then you don't need to do this step. That file must consist of a list of submissionID/reviewerID pairs, separated by whitespace, as shown below:
% cd src/util
% tclsh
> source assign.tcl
> assign_meta_reviewers filename tmpdir
> exit
This should place a file called mtemp.tcl in tmpdir.
Warning: In the reviewer creation phase above you need to have indicated if a reviewer is a metareviewer or not. This function does a check to be certain that all metareviewers have been assigned the correct type, and will return an error if this is not the case.
Finally, you need to assign papers to meta-reviewers. If you do not wish to use the automated matching program, then you need to create a file by hand called matches.tcl. This file should go in the same temporary directory as used in the previous two commands. The file must begin with:
global reviewinfo
Then, for every reviewer/paper pair, it must contain the following two lines:
lappend reviewinfo(submissionid,ftrs) reviewerid
lappend reviewinfo(reviewerid,papers) submissionid
Once matches.tcl is created (either manually or using the programs described below) the final data file is generated as follows:
% cd src/util
% tclsh
> source assign.tcl
> create_review_datafile tmpdir
> exit
Once this step is completed, you are ready to start online reviewing.
This software package provides software to present and record topic choice information to both reviewers and paper submitters, and to then run the matching software on the results and produce a list of paper assignments. You may adjust the results to their liking by hand. An especially nice feature of the matching algorithm is that it allows you to specify the minimum and maximum number of papers per reviewer, and reviewers per paper, finding an optimal matching assignment, based on the weight of match between reviewer and submission.
We had to run the algorithm several times, varying these parameters a bit, since if you require, say, no more than six papers per reviewer, and want four reviews per paper, you have to have at least 1.5 reviewers per paper. If you have fewer reviewers than the minimum allowed (or you have more but you don't have enough non-zero match scores) then the algorithm will not return a result. This requires adjustment of the range of expected papers per reviewer or reviews per paper (or both).
We found that the matching software did an excellent job using overlap between topic choices. (This may have been due in part to the specificity and comprehensiveness of the topics, which I created by hand.) I heard comments from several reviewrs saying that the papers matched their interests much more accurately than usual. We had 135 submission, about 90 first-tier reviewers, and were able to assign between 3 and 6 papers per reviewer and have three first-tier reviews for every paper. (We also had a meta-review for every paper.)
The software allows you to avoid a conflict of interest if you assign a submission/reviewer pair a match score of zero.
We found that we did not like using the matching software for the meta-reviews. We wanted to ensure that all papers on a given topic (e.g., clustering) went to one meta-reviewer, and we wanted to select that meta-reviewer. So we organized the papers by topic and hand-assigned papers to meta-reviewers (we had 35 meta-reviewers vs. about 90 first-tier reviewers).
You must install this code before you can do the next steps. It is quite easy to do. Simply follow the directions in src/util/matching/PREPARE (basically you just run make).
The steps below creates a file that contains assignments of submissions to reviewers. This assumes you only want to run this with first-tier reviewers. It lets you customize the above to change the number of reviews per paper and papers per reviewer. The steps below assign between 0 and 5 papers per reviewer, and between 2 and 4 reviews per paper. (It is a good idea to allow 0 reviews for a reviewer, in case there is a reviewer whose interests do not match any papers. Otherwise the program will fail to produce output under these circumstances.)
% cd src/util
% tclsh
> source assign.tcl
> find_reviewer_matches_customized tmpdir 0 5 2 4
The steps below creates a file that contains assignments of
submissions to reviewers, using all reviewers, not just first-tier reviewers.
% cd src/util
% tclsh
> source assign.tcl
> source ../global/reviewerinfo.tcl
> find_reviewer_matches_defaults tmpdir $reviewerinfo(allreviewers)
The output appears in tmpdir/matches.tcl.
You must check the results of this step to see if it actually succeeded or not. If the constraints can't be met then no results are generated. You will need to adjust the constraints and try again.
You may want to modify the assignments that the system produces. The ascii file allows you do to so. When you have the assignments finalized, copy the file matches.tcl into src/global so it can be used by subsequent parts of the software.
An alternative to running this software is to simply assign submissions to reviewers by hand. How to do so is described above.
Once matches.tcl is created (either manually or using the programs described below) the final data file is generated as follows:
% cd src/util
% tclsh
> source assign.tcl
> create_review_datafile tmpdir
> exit
This puts all the information together that is needed to do online reviewing. The information is placed in ../global/reviewinfo.tcl.
Warning: once you have created reviewinfo.tcl you should not delete or overwrite this file, since it is needed by the reviews program. It is possible to recreate this file using the steps above, but if you've made adjustments to matches.tcl it may be difficult to do this accurately. Storing a copy of the file somewhere might be a good precaution.
Each topic has a name, and that name must be appended to the topics(topicnames) list that is found in the topics.tcl file. It is a good idea to make this name a descriptive summary of the topic itself, since it is sometimes shown as a shorthand for the topic.
The information to customize is in src/global/definitions.tcl
If you want to add or remove an attribute, you must modify reviewinfo(reviewattributes) to add or remove the attribute's name.
set reviewinfo(reviewattributes) {appropriate summary importance arguments presentation other score assessment confidence}
You must also add or remove the information that describes how an attribute is presented in the web pages. These attributes are stored in the rattributes variable. Associated with all attributes is a variable called msg that indicates the label that is shown alongside the display. There are three possible data types for attributes: text, integer, and llist. There are four display types: textbox, textlist, radiobuttons, and checkbox. Certain display types have other variables that can be set. Textline can specify linelenght, textbox can specify numrows and numcols. For example:
set rattributes(score,displaytype) radiobuttons
set rattributes(appropriate,displaytype) textbox
set rattributes(appropriate,datatype) text
set rattributes(appropriate,msg) "1. Is this paper relevant to SIGIR? (See criteria enclosed in the reviewer's packet and on the web site.)"
set rattributes(appropriate,numrows) 3
set rattributes(score,datatype) integer
set rattributes(score,msg) "7. Using the following scale, provide a numerical rating of the paper's ACCEPTABILITY."
set rattributes(score,radiochoices) {{1 "1 = Definitely Reject"} {2 "2 = Probably Reject"} {3 "3 = Could go either way"} {4 "4 = Probably accept"} {5 "5 = Definitely accept"}}
Note that it must appear in the cgi-bin directory readable by the web server in order for the cgi script to be executed.
When a reviewer enters information into the system for the first time, a directory is created in data_directory_path/reviewers that is named after the submission ID. A file named review.back.tcl is placed in this directory. This file contains the default (empty) settings for all of the review information. When the reviewer upates their review information, the new information is placed in a file called review.tcl and the old information shifted to review.back.tcl. It is important not to modify these files by hand once they are created; only the system should modify them.
You may want to have a period of time during which the first-tier reviewers can only see their own reviews. After the first deadline has gone by and the meta-reviewers are generating meta-reviews, it may be convenient to let first-tier reviewers see the other reviews for their submissons.
To allow first-tier reviewers to see the other first-tier reviews for their submissions, set the following in src/globals/definitions.tcl:
set customizations(reviews,ftrseeftrs) 1
To allow first-tier reviewers to see the first-tier reviews and the meta-reviewsfor their submissions, set the following in src/globals/definitions.tcl:
set customizations(reviews,ftrseeftrs) 1
set customizations(reviews,ftrseemetas) 1
To prevent meta-reviewers from seeing the first-tier reviews for their submissions, set the following in src/globals/definitions.tcl:
set customizations(reviews,ftrseemetas) 0
global conflict
set conflict(110) washingtong
set conflict(131) jefferstont
set conflict(202) adamsj
To create a spreadsheet tallying reviews of papers, you first have to know what the maximum number of first tier reviewers is per paper. This code only works for the default review attributes; if you have other attributes you will have to modify the code to use it.
% cd src/util
% tclsh
> source pcmeeting.tcl
> compile_stats outputfile maxnumftrs
This will produce a file in which columns are delimited by dollar signs ($) and thus can be read into a spreadsheet or database program.
To generate an html file with links to reviews for every submission:
% cd src/util
% tclsh
> source pcmeeting.tcl
> generate_html_reviews_for_pc_meeting outputfile
This places the resulting html in an appropriate file. You will need to modify it to give it headers, etc.
This requires a file has been created called accepted.tcl, in the local directory, that has the follwoing format, showing the submission IDs only of those papers that were accepted.
WARNING: if you have the accept/reject information wrong, your submitters will be notified of the wrong information. It is very important that you first manually check the results of create_review_decisions before mailing them out.
The following function writes the reviews to files in directory prefix. They can then be emailed to the appropriate people using the following function.
% cd src/util
% tclsh
> source email.tcl
> create_review_decisions prefix
> exit
This function emails the reviews created in the previous step. You can edit the function to substitute in your email address if you'd like to test it out first.
% cd src/util
% tclsh
> source email.tcl
> email_review_decisions prefix
> exit
An issue arises with respect to people cutting-and-pasting reviews from word-processing programs into the html forms. These will tend to be represented as one long string, and could through the table view off if verbatim display mode were used. However, if verbatim mode is not used then line breaks, paragraphs skips, offset bulleted items are lost.
These issues are resolved in a function called clean_up_text that
appears in startup.tcl. It substitutes \
One potential problem with not requiring passwords in the submission
process is that someone might try to load down the system by entering
a large number of bogus submissions. The software does not provide
provisions to protect against this sort of attack.
Another potential problem is that passwords and IDs are typed in the
clear, so potentially someone who sets up a sniffer on the line could
detect passwords. If someone would like to modify this software to
make it more secure, I would welcome the help.
The software passes ID information between cgi scripts. It relies on
the cgi.tcl module to handle this information securely.
Last modified December 9, 1999. Marti Hearst
Security
This software is not intended to be robust against security attacks.
It assumes the web server and the system it runs on takes proper
security cautions.