FINAL REPORT

Psychology Experiments on the Internet: Use of Shockwave Technology

Grantee Organization
University of Mississippi
University, MS 38677

Grant Number: P116B970238

Project Dates
    Starting Date: Sept. 1, 1997
    Ending Date: Dec. 31, 2000
    Number of Months: 39

Project Directors

Kenneth O. McGraw, Dept. of Psychology
Mark D. Tew, Dept. of Electrical Engineering
University, MS 38677
Telephone: 662-915-5192

FIPSE Program Officers: Brian Lekander and Karen Levitan

Grant Award:

Year 1 $69,979
Year 2 $73,563
Year 3 $74,064
Total $217,606

Project Summary

In three years, our goal of creating a web-based co-laboratory for use in training students in experimental psychology has gone from a mere concept to PsychExperiments http://psychexps.olemiss.edu , an instructional resource for 326 psychology classes at a broad variety of institutions ranging from comprehensive universities to community colleges and high schools. Over the past 19 months, better than 18,000 data sessions have been conducted at the site providing students with real data to analyze and report. Starting with three site contributors, we now have 29, many of whom participated in grant-sponsored workshops to acquire the skills needed to develop materials that could be added to the communal site.

Kenneth O. McGraw
Dept. of Psychology
University of Mississippi
University, MS 39677
662-915-5192 (voice)
pymcgraw@olemiss.edu

Executive Summary

Psychology Experiments on the Internet: use of Shockwave Technology

A. Information for FIPSE

We have been very proud to have FIPSE support through the first three years of our project because the cachet "FIPSE-funded" has immediate credibility in the academic world. This has been important in attracting site users and workshop participants.

B. Project Overview

The PsychExperiments project grew out of a faculty training workshop attended by PI McGraw at the University of Mississippi in 1996. The workshop, taught by PI Tew, was designed to encourage the use of multimedia computer applications in the classroom. Interactions during the workshop led McGraw and Tew to discover that Authorware, a Macromedia product, was an excellent experiment generator and that experiments developed using this tool could be presented via the Internet to students. This discovery fostered the vision of a web-based, co-laboratory that would permit any institution with a computer and Internet connection to offer a laboratory course in psychology. The core of the co-laboratory would be a set of experiments developed in Authorware by Tew, McGraw, and a graduate student webmaster-John Williams. The initial core would be modified and supplemented based on contributions from faculty who chose to use the site and participate in training workshops. Each experiment at the site was designed to write data to a single cumulative data archive that could be shared by all the site users. Among many advantages of the cumulative archive, it would give a means for instructors of small classes to obtain datasets large enough to address questions regarding between-subject differences and effects that might be subtle.

C. Purpose

In brief, the purpose of the PsychExperiments project was to provide a web-based, community owned and developed co-laboratory for use by psychology classes, with the main beneficiaries being students and instructors in laboratory classes. Over time and by force of communal effort and continued evolution, the set of experiments would become larger, better crafted, and more up-to-date than any of the locally available alternatives. Moreover, the experiments would link to a database where results could be stored for later download.

D. Background and Origins

Laboratory courses in which students learn to design experiments, collect and analyze data, and report findings are a required part of the curriculum at 90% of the institutions with undergraduate psychology programs. Typically, the courses use computer-based activities because computers are the preferred laboratory instrument for stimulus delivery and measurement in many experimental areas of psychology. Most psychology departments feel compelled, therefore, to locate undergraduate laboratory space and to equip the space with computers and software. With a website to serve as a laboratory, departments would be freed from the need to provide a physical laboratory, the computers used to access the site could be anywhere, and the programs run on the computers could be shareware.

E. Project Description

The project consists first and foremost of a website http://psychexps.olemiss.edu which offers the following:.

•A set of experiments (20 at present) designed either to replicate classic effects or to investigate novel manipulations regarding well known phenomena.
•Instrructors' pages for each of these experiments. The pages provide background information, references, suggested data analyses, and methodological information.
•Excel macros for each of the laboratory experiments that provide individual and group analysis plus graphs for effects of major interest.
•A set of research experiments (30 at present) that are being hosted for faculty wishing to collect data from web samples. Hosting these experiments was not part of our original purpose but it has been a natural extension of the technology.
•A cumulative Access database that includes data from every experiment session conducted at PsychExperiments since September 1999.
•A set of Active Server Pages that are used to place data in the database, extract filtered data from the database, and write a comma separated file for the filtered data.
•Free downloads of the source code for all experiments at the site

F. Evaluation/Project Results

Site Use. PsychExperiments has been used by 326 classrooms worldwide at 231 different institutions. Although the site can be accessed via four URLs, the most heavily used is the original olemiss.edu/psychexps address which is currently getting an average of 350 hits per day. Data submissions to the site from research participants will be well in excess of 20,000 by the end of the current semester. The download offerings at the site are very popular. Records for this academic year alone show 3021 data downloads, 924 downloads of our substantially improved analysis macros, and 1762 downloads of experiment source code.

Effects on Learning. Student and faculty response to the site has been uniformly enthusiastic with considerable evidence that students find the experiments engaging and stimulating. The site is applauded for convenience, quality of materials, and the added insight that comes from being able to participate in experiments rather than just read about them.

Community of developers. In order to promote community development of PsychExperiments we conducted four on-campus workshops for a total of 42 invited participants. Evaluations of the workshops in terms of total experience, amount learned, and confidence in one's ability to use Authorware to make an independent site contribution resulted in very high ratings (4.67, 4.61, and 4.61 on a Low=1, High=5 scale). The 3 ½ day worskhops have resulted in contributions to the site from 23 of the participants thus far with a few yet to come in. We had hoped for a somewhat higher return rate. Nonetheless, the workshops have been extremely valuable in promoting the community concept. The participants are now promoters of the site among their peers. Seven of them have made convention presentations reporting on their use of PsychExperiments or on some technical aspect of web-delivery of experiments.

Cost Savings. Using PsychExperiments in lieu of alternatives makes its possible for instructors to include data gathering activities in their classes without having to buy software. Also, departments do not need to provide students with computers for their laboratory work. In a survey of instructors using the site, 53% strongly agreed and 29% agreed (82% total) that longterm use of PsychExperiments would save their departments money.

Presentations and Workshops. During the three years of the grant, an important component in our plan to build a community of users has been professional presentations. To date, we have given 21 presentations and/or workshops at professional meetings or on university campuses.

Publications, Publicity, and Awards. In addition to outreach via presentations and workshops, we have submitted journal articles. The first article, published in the technical journal Behavioral Research Methods Instruments & Computers, reported on our findings in a survey of the need for a website such as PsychExperiments. The second was a book chapter in Micahel Birnbaum's Psychology Experiments on the Internet published by Academic Press. The third has been the most significant-an article reporting on the fact that data from the site mirror laboratory data. This article appeared in Psychological Science, the member's journal of the American Psychological Society.

In addition to publicity generated by articles we have published, we have been fortunate in getting some free publicity. Most notably there was a feature story in the APS Observer and two feature stories in the APA Monitor. These publications go to virtually every psychologist in this country and many abroad in that they are member publications of the American Psychological Society and the American Psychological Association, respectively. Awards received by the site include 1^st Place in the Learning Software Design competition hosted by the Univ. of Minnesota and honorable mention in Macromedia's eLearning Award competition (June, 2000), with special recognition for technical excellence.

Dissemination. Our ultimate goal for PsychExperiments at this point is that it be taken over by one of the professional organizations-either APS or APA-and offered to members as a benefit. To make this happen, we need to continue building our user base, improving the site technology, and training more developers. Funding we have received from NSF to develop training materials and to purchase a high-end server with state-of-the-art security and reliability will be very important in this effort. As the site continues to become an established part of the infrastructure for education in psychology, it will be easier to convince the professional organizations that operating the site will be good both for psychology education and for their organizations.

G. Summary, Conclusions, and Lessons Learned

The project we envisioned in 1997 for conducting psychology experiments via a communal website featuring shareware programs and a single data archive has proved viable. The most important thing we have learned is that the acceptance of innovations within the academic community is slow and requires sustained outreach. We are committed to continuing the outreach, and we have absolute confidence that PsychExperiments is a much-needed innovation with a very bright future.

BODY OF REPORT

A. INFORMATION FOR FIPSE

We have been very proud to have FIPSE support through the first three years of our project because projects with the cachet FIPSE-funded have immediate credibility in the academic world. This has been important in attracting site users and workshop participants. The FIPSE staff with whom we have worked have understood our needs and been willing to support them. For example, our project has required equipment and we have been permitted to acquire first a Macintosh computer, which was needed to develop the cross platform versions of our Authorware programs, and then an NT server to use in lieu of the campus Unix server, which gave us far more control over our site. At the end of our project we were able to apply residual funds toward the purchase of a laptop for use in the off-site demonstrations we use for dissemination and outreach. In September of 2000 we received a site visit that permitted us to show first hand some our work.

B. PROJECT OVERVIEW

The PsychExperiments project grew out of a faculty training workshop at the University of Mississippi in 1996. The workshop, taught by PI Tew, was designed to encourage the use of multimedia computer applications in the classroom. One of the software packages shown to faculty during the workshop was Authorware, a programming tool that permits users to develop highly interactive computer programs. As a final project in the workshop, participants had to develop a computer application for use in their classes. PI McGraw chose to do an experiment involving the Stroop phenomenon. The goal was to have a single experiment that could be added to the repertoire of mostly home-grown experiments that were the mainstay for McGraw's laboratory course in psychology. Working on this task jointly, McGraw and Tew discovered that Authorware is an excellent tool for generating experiments and that is relatively easy to master. Moreover, they learned that Authorware produces web-deliverable products. All it takes is a plug-in for the web browser, which at the time was called a Shockwave plug-in. (Now it is known as the Authorware Web Player.)

The combination of these discoveries set off the mutual realization that McGraw's goal of providing a stand-alone experiment for use locally by his students was absurdly small in scope. One could create a package of experiments that could be used by anyone with an Internet connection. No longer would McGraw's students have to use the lone computer then available to them in the Psychology Department to collect data at an appointed time. They could use any machine, anywhere. And access would not be limited to McGraw's students. Anyone could participate. Moreover, if McGraw and Tew were willing to share the experiments they developed with other laboratory classes, then other faculty might develop other experiments that they would be willing to share. The end result could be a co-laboratory consisting of a vast variety of experimental opportunities, some support materials, and most importantly a database with a cumulative record of the all the experimental sessions run for each of the experiments at the site. That was essentially the vision we developed for PsychExperiments in 1996 and it is the one we have realized today.

PsychExperiments consists of a set of laboratory experiments used primarily by laboratory classes and a set of research experiments that are used to collect proprietary data regarding an empirical or theoretical concern of interest to the initiator of the experiment. The site is visited by hundreds of students, faculty, and others each day for the purpose of participating in one or more of these experiments or obtaining data from them. The co-laboratory concept has been fulfilled not only by having users who share the resources but by having developers who have shared their contributions with us. As documented in detail below, the initial 3-person team at PsychExperiments launched the site and produced its initial content, but all of the novel content coming in at present is produced by others, primarily those who have participated in grant-supported workshops to learn to use Authorware.

C. PURPOSE

The purpose of the PsychExperiments project was to provide a web-based co-laboratory for use by psychology classes, with the main beneficiaries being students and instructors in laboratory classes. In addition to providing a shared resource, we planned to build a community of developers who would borrow from and, then, contribute back to the site. This purpose was established from recognition of the fact that a primary advantage of the Web is that it permits sharing. Any one experiment, any html page, and dataset is not a final finished product; rather, the experiments and html pages can be edited, added to, and improved in functionality and the datasets can be appended semester to semester and year to year. One person can do this, but in our case we wanted a community of people to contribute. In the grand vision, therefore, our purpose was to permit faculty to build jointly a website featuring an ever-growing set of experiments that could be used in the instruction of psychology students. Over time and by force of continued evolution, the set of experiments would become larger, better crafted, and more up-to-date than any of the locally available alternatives. Moreover, the experiments would link to a database where results could be stored and later downloaded. When we began the site, ours was not the only one that offered "experiments' via the Web (though many were just form-based applications of HTML), but none offered shared data. In the absence of data collection, experiments can be used for instructional purpose but they cannot be used for doing science, which is the primary goal of laboratory classes.

D. BACKGROUND AND ORIGINS

Laboratory courses in which students learn to design experiments, collect and analyze data, and report findings are considered a fundamental part of the psychology curriculum at nearly every undergraduate institution. According to a survey we conducted, 92% of departments offer such a course and 90% make degrees contingent on students taking such a course (Williams & McGraw, 1999). In addition, most departments have independent research opportunities that students can complete for credit. The introduction of inexpensive computers capable of running programmed experiments revolutionized these traditional laboratory experiences in the years after 1977, the year in which Apple IIs were introduced, to the point that there was a sudden, widespread need to establish computer labs for student use. As funds became available, departments purchased computers and laboratory software packages from developers like CONDUIT, Life Science Associates, or Psychology Software Tools, Inc. and then proudly declared the opening of their computerized undergraduate labs.

The problem with this approach then and now is that funding for equipment and software is aperiodic, whereas hardware and software developments are on a perpetual upward trajectory. Keeping up with the curve has been a problem for every department that has established student labs, and for most, the best they are able to do is to occasionally brush against the curve in those years when one-time money is available.

The Internet offered an obvious solution to the problem in that web-deliverable programs could be shareware for laboratory classes and the student computers could be any Internet connected computer. When we started our project, however, no one had exploited this opportunity. There were some websites that offered isolated experiments programmed in Java or some HTML form-based activities, but none of these sites even approached being a laboratory. They served either as demonstration sites or as a means of collecting proprietary data. They did not offer students the chance to collect data on themselves and classmates and then download the data for analysis. PsychExperiments filled this void.

E. PROJECT DESCRIPTION

The project consists first and foremost of a website http://psychexps.olemiss.edu which offers the following:.

•A set of experiments (20 at present) designed either to replicate classic effects or to investigate novel manipulations regarding well known phenomena.
•Instrructors' pages for each of these experiments. The pages provide background information, references, suggested data analyses, and methodological information.
•Excel macros for each of the laboratory experiments that provide individual and group analysis plus graphs for effects of major interest.
•A set of research experiments (30 at present) that are being hosted for faculty wishing to collect data from web samples. Hosting these experiments was not part of our original purpose but it has been a natural extension of the technology.
•A cumulative Access database that includes data from every experiment session conducted at PsychExperiments since September 1999.
•A set of Active Server Pages that are used to place data in the database, extract filtered data from the database, and write a comma separated file for the filtered data.
•Free downloads of the source code for all experiments at the site

Addiitional project components are a network of users and developers who are committed to the website and its technology. This network has been forged through diligent outreach efforts involving conference presentations, e-mails, phone calls, and workshops. One clear sign of the growing commitment to the site is that we have mirror servers at the University of Edinburgh in the UK and at the University of Iowa. The Edinburgh server is particularly important because it permits UK participation in the site at no cost. If experiments were obtained from a US-based server, the UK users would have to pay indirectly via a charge to their universities. Because of this policy universities only allow undergraduates to access US-based sites that are mirrored or available via proxy servers. PsychExperiments would be off limits to most undergraduates without the mirror. What is true in the UK may soon be true in all of Europe.

Reaching the current state of development has been an arduous task made more difficult by unforeseen problems in the technology in the areas of evolution of Authorware, cross-platform compatibility, site development, load on the server, and emerging needs.

•Evolution of Authorware: While many psychology departments have chosen Apple computer products for their laboratories, the percentage of Apple users worldwide is very small compared to Intel platforms. When the project began, an Apple computer was required to prepare web-delivered versions of the Authorware experiments. In the second year, the next version of Authorware did not require an Apple computer. In the third year of the project, the latest version of Authorware is not even available on the Apple platform. (This may be one reason for the low completion rate of first-year workshop participants). In the same time, program-to-server communication has moved from FTP to POST method URLs.
•Cross-platform compatibility: Many experiments at PsychExperiments require precise control of the presentation of text, yet Apple and Intel platforms have different form factors for character displays, so that exact duplication of displays across platforms is impossible. Text sizes are not exactly the same, so that to display a word vertically letter-by-letter requires inserting a carriage return between each character. Similar problems are encountered in timing issues, in the display of graphics, and in the presentation of sound. While the Intel standard for keyboard/computer communication is open, allowing us to make precise measurements of the accuracy with which Authorware reports keystroke intervals, the Apple keyboard/communication protocol is proprietary, preventing us from making accuracy measurements.
•Site development: We have had to master web/database communication. Our initial development was on the campus Unix server. Security concerns on the server lead to the requirement that all cgi scripts (the program that receives the data from the URL and places it into the database) be placed in a directory which can only be accessed by IT personnel. While IT personnel were very cooperative and supportive, this bottleneck slowed considerably the pace of innovation and development. This problem was solved by going to a dedicated server for PsychExperiments, where we chose an NT platform. This allows us to use asp pages and VBScript, languages that are closer to our programming background.
•Load on the server: When we began work on the NT server, we chose an Access database due to cost and ease of use. Now, after only 2 years, we can see where the PsychExperiments data will exceed the capabilities of Access. At the same time, our server is experiencing load problems and "crashing" with increasing frequency. These problems will be remedied by a new server and database provided by funding from the University and from the National Science Foundation.
•Emerging needs: In the original proposal, development of analysis tools received little emphasis. However, when we began introducing Excel macros for data analysis, this proved to be one of the most useful and used areas of the site. Initial macros were developed by graduate students in Computer Science and perform adequately, but are slow and must be customized for each new experiment. We are currently developing a new macro which is easily customized to handle new experiments. As part of the new analysis macro, the dataset used to test development of the macros shows a decrease in the time to perform an analysis from 100 seconds to 15 seconds. Development of tools that can be easily adapted to new experiments has been a significant achievement of this project.

F. EVALUATION/PROJECT RESULTS

When we first proposed PsychExperiments, we pointed to a number of advantages that Web-based experiments have over alternative technologies for both students and professors. The following is a summary of those advantages and some data to document that the promised advantages are being realized.

Convenience

•38% of students polled strongly agreed and 31% agreed (Total=69%) that using PsychExperiments to collect data was more convenient than the alternatives available to them.

•28% of student users reported using a personal (non-university owned) computer to participate in experiments. Psychology department computers were used by just 40% of the students.

•Better than 20% of the experiment sessions were conducted weekends, or at weekday times when university computers and labs would not be available.

Large datasets

•Better than 18,000 experiment sessions had been conducted at PsychExperiments by mid-March of this academic year with four experiments having in excess of 1000 data records. The top ten are Mental Rotation - 2208, Self Reference - 1753, Lateralized Stroop - 1463, Reaction Time Color - 1403, Perception of Gender - 1366, Word Recognition - 1133, Mueller Lyer - 917, Line Motion - 842, Poggendorff - 801, and Covert Attention - 546

Large datasets permit study of phenomena with small effects and the determination of null effects⁽¹⁾ Our data permit the inference that

•the effect of fin angle on the Mueller Lyer illusion is not a linear function of angle, rather it is a sinusoidal function

•the length of diagonal lines in the Poggendorff has a small but reliable effect on the magnitude of the illusion

•the Stroop effect is the same for centrally presented stimuli as for laterally presented stimuli

•there is no gender X visual field interaction in word recognition speeds (which would be hypothesized based on the view that the males' brains are more lateralized than females' brains).

•the gender difference in mental rotation time does not depend on the degree of angular disparity between the stimuli being judged (which would be hypothesized based on the view that males execute rotations at a faster rate)

•There is an unanticipated gender effect in the Mueller-Lyer and Ponzo illusions with females experiencing larger illusions than males.

Large datasets permit distributional analyses

•Typical laboratory datasets have so few observations per experiment condition that a distributional analysis of the data is uninformative. Instructors using PsychExperiments data can (and do) have students produce histograms, stem-and-leaf plots, or box plots for depicting their data as well as compute skew and kurtosis.

Speed of data collection

•The database has many sets of nearly simultaneous data submissions. This indicates that the submissions came from students working in parallel in a computer lab. Such students can get all the data they need for a class project in a single class period.

•Andi Agnew (Mississippi College undergraduate), C.J Yu (UM graduate student), Amy Criss (U. Of Indiana graduate student), and Clint Moody, Shari Gregoire, Jennifer Mann, and Frances Patterson (UM undergraduates) collected data for undergraduate and master's theses projects in a fraction of the time that it would have taken had they been collecting data serially from research participants who were given scheduled times to appear in a laboratory.

Student learning and interest

•17% of students polled strongly agreed and 46% agreed (Total=63%) that their educational experience in psychology had been enhanced by PsychExperiments; 21% strongly agreed and 48% agreed (Total=69%) that their learning in a specific class was enhanced; 19% strongly agreed and 45% agreed (Total=64%) that their interest in a specific class had been enhanced by use of PsychExperiments. Written comments were uniformally positive

•Instructors estimates of student benefits on the same three items (general enhancement, learning in a specific class, interest in a specific class) were higher: 91% strongly agreed or agreed that the educational experience had been enhanced, 94%

strongly agreed or agreed that learning in a specific class had been enhanced; and 85% strongly agreed or agreed that their students' interest had been enhanced.

Alterable programs

•The interest in learning to use Authorware as an experiment generator has been high. We had 72 applications for 40 slots in our workshops and off-site workshops drew well. For example, the SPSP (February, 2000) and CiPP (March,2000) workshops were at capacity (40 and 25, respectively).

Cost saving

•53% of faculty users polled strongly agreed and 29% agreed (Total=82%) that using PsychExperiments would save their departments money long term.

Usage Statistics

There are a number of relevant measures of site usage: Number of website visits, number of data entries to the database, number of downloads of data, macros, and code; number of classrooms registered to collect data at the site, number of listerv members.

Hits on Website. One of the simplest measures of site usage is number of hits. The data in Figure 1 show median number of hits per day on our initial web server-a campus Unix server-- during our six semesters of operation. In that the URL for this machine is the one we have broadcast most widely, it receives the heaviest use. However, the growth in site use on this server is an underestimate of actual growth in site use because we now have three additional servers. Hits on these machines are not reflected in the Figure 1 data. The additional servers are an NT server at UM, a Linux server at the Univ. of Edinburgh, and an NT server at Iowa. Adding data on hits/day for these machines would reveal even more explosive growth in site usage.

Number of data entries. A data entry is created whenever a person completes an experiment and chooses the "send data" option. Data in the figure are from the current PsychExperiments database and represent data submitted beginning Fall 99. These data show a steady increase in the number of data sessions conducted at the site with the total data entries over the two-year period projected at 23,303. This figure is based on the current data submission rate which has produced 4946 entries halfway through the Spring 01 semester. The actual data count at the time of this report (March, 2001) is 18,357. During our first year of operation, AY 98-99, we collected about 1000 additional data records. Those data are not included in the current database because many of the experiments used to create the data were modified during the summer of '99, so that the earlier data are not directly comparable.

The table to the left shows the top 20 institutional users in terms of database submissions.

Download Activity. An important feature of the site, one that in fact makes the site totally unique, is that we permit downloads of (1) data from the cumulative database, (2) macros for analyzing the data, and (3) source code for the experiments. These three offerings document that the site serves as a collaboratory promoting the sharing of resources. The download opportunities have been very popular as shown in the data to the right, which gives download information for data, macros, and source code by laboratory experiment. Data downloads from the database have been tracked since Nov 1, 1999. In the 16 month period from then until the time of this report (March, 2001), there have been 949 downloads of Excel macros, 5067 data downloads, and 2936 source code downloads. The data and source code totals in Table 1 are less than the totals given here because the table only reports downloads of laboratory experiment materials. The additional downloads were for materials from research experiments.

Number of Downloads of Macros, Data, and Source Code Broken Down by Experiment Name at PsychExperiments
Experiment Name		Macros	Data	Code
Covert Attention		25	103	45
Dichotic Listening		42	56	71
Facial Recognition		83	296	145
Gender Perception		28	467	77
Lateralized Stroop		51	315	78
Lexical Decision		19	39	68
Line Motion		35	228	69
Maze		42	82	99
Mental Rotation		102	518	180
Mueller-Lyer		50	291	100
Perception of Gender		75	461	78
Pitch Memory		34	122	52
Poggendorff		28	116	64
Political Poll		37	50	36
Ponzo		23	44	61
RT Sound		72	363	63
RT Color		26	29	196
Self Reference		76	502	107
Word Recognition		101	542	203
	Total	949	4624	1792

Listserv Members. We maintain a listserv to provide updates and information sharing regarding PsychExperiments. There are 178 listserv members at present. Most are faculty currently collecting data at the site whether for themselves or their classes.

Classrooms Using the Site. In the spring semester of 2001 there were 326 classrooms at 231 institutions worldwide signed up to collect data at our site for use by their students. This statistic more than any other speaks to the success of our 3-year project. Heaviest use is in the USA where users include 75 classrooms in the southeast, 50 in the southwest, 14 in the northwest, and 125 in the northeast. Abroad there are 2 classrooms in Africa, 7 in Asia, 9 in Australia, 2 in South America, 7 in Canada, and 25 in Europe. Because the list of registered users is so long, it is included as an appendix rather than a table. One of the most striking features of the list is the number of small, unheralded institutions that are using the site. In developing our proposal, a goal was to enable smaller universities to offer a laboratory experience to their undergraduates that they would not be able to offer without the Internet resource.

Workshops

In May and August of 1999 and 2000, we conducted Thursday-Sunday workshops for faculty invited to the Univ. of Mississippi to learn to develop programs in Authorware for use at PsychExperiments. Of the 42 people trained in workshops (see list in Appendix B), 23 have one or more experiments at the site presently (55%). We had hoped for a 75% success rate. The shortcoming is due to a number of factors including workshop design, participant selection, and inadequate post-workshop follow-up. Despite the fact that we did not meet our target, we are still pleased to have so many co-developers at the site, nearly all of whom came to us not only as Authorware novices but as novices and semi-novices (56%) for any kind of programming. In addition to those trained in workshops, we have attracted contributions from four people who were already competent in Authorware.

If one grades the workshops on participant response to the workshops rather than ultimate success in developing an independent contribution, the workshops were extremely successful. On a scale of 1 (low) to 5 (high), the mean rating for the total workshop experience was 4.67 of 5. One participant made this kind remark: "Nicest people I've dealt with in some time. Your willingness to do all this work FOR us is simply unbelievable. This is the true spirit of academia, and its VERY nice to see that it hasn't completely disappeared. Thank you." Quality of instruction was given a mean numerical rating of 4.61. Comments provided lots of praise, particularly for Mark Tew, the lead instructor. This one is typical: "I thought the workshop was well-structured. The training modules were good and I appreciated the time to recreate what was demonstrated (and/or the challenge of seeing a finished product and being asked to create it). MarkTew is an excellent trainer." Amount learned was rated 4.61. One participant said, "I have struggled with 2 previous programming languages and am dead certain that you could not possibly gain this level of programming skills [in those languages] in such a short workshop. Another said, "I know I have more to learn but I am amazed at my level of proficiency!", and when workshop participants were asked how confident they were that they could begin developing their own contributions to PsychExperiments, 72% expressed extreme confidence and just 11% said they were unsure. PsychExperiments has benefitted tremendously by having 42 faculty to work with us so closely and then carry away such positive views of the site, its technology, and their own capacity to contribute to it.

Presentations and Workshops

The following list of presentations and workshops summarizes the product of outreach activities funded by the grant.

•McGraw, K.O. Billock, A., Tew, M. & Roberts, K, Multimedia in the Classroom, a cluster session presented at the annual FIPSE Directors meeting, Washington, D.C. Oct. 1998.
•McGraw, K.O. (1998). PsychExps: An on-linepsychology laboratory for student use. An invited workshop presented at the 6^th Eastern Conference on the Teaching of Psychology, James Madison University, Oct. 1998.
•McGraw, K.O, Tew,M.D. & Williams, J.E. . An introduction to PsychExps. Society for Computers in Psychology Meeting, Dallas, TX, Nov. 98.
•Williams, J.E., McGraw, K.O., & Tew, M.D. (1998). A Survey of Laboratory Computer Use in Undergraduate Psychology Departments. Society for Computers in Psychology Meeting, Dallas, TX, Nov. 98.
•Using PsychExps for Laboratory Classes in Psychology. McGraw-Hill PRISM Conference on the Teaching of Psychology, Baltimore, April 1999. (25 participants) Invited.
•Developing Experiments Using Authorware (with Mark Tew). FIPSE-sponsored training for PsychExps developers. University of Mississippi, May, 1999 (10 faculty), August, 1999 (10 faculty).
•How to Use PsychExperiments in Your Classes, an Invited Address presented at the meeting the American Psychological Association, Boston, August, 1999.
•Demonstration of Technology in the Classroom and Lab: Interactive Technology Session presented as an electronic poster at the meeting the American Psychological Association, Boston, August, 1999. (Invited)
•PsychExperiments: A Web-based cognitive psychology laboratory. Presented at WebNet99, Honolulu, October, 1999.
•Data Integrity and Web-based Experiments in Psychology. Presented at WebNet99, Honolulu, October, 1999.
•PsychExperiments: A Resource for Instruction in Psychology. Poster presented at WebNet99, Honolulu, October, 1999.
•Using the Web for Research in Social and Personality Psychology, with Scott Plous. A 3-hour invited workshop, Society for Personality and Social Psychology, Nashville, Feb. 2000. (40 participants) Invited.
•The integrity of data from web-delivered experiments written in Authorware. A paper presented at Computers in Psychology Conference 2000, York, England, March 2000.
•Using and Contributing to PsychExperiments. A 1 ½ hr. workshop presented at Computers in Psychology Conference 2000, York, England, March 2000.
•Southeastern Psychological Association, Annual Meeting. "Using PsychExperiments in Classes," A symposium conducted by John Williams (organizer), Cyndi McDaniel, Northern Kentucky University; Ric Topolski, Augusta State University; and Andrew Velkey, Mississippi College, March 2000, New Orleans.
•American Psychological Society, Annual Meeting. Roundtable discussion on how to use PsychExperiments in classes. Conducted by Mark Tew, June 2000, Miami.
•Roundtable Conference on Interactive Multimedia Decision Simulations, Oct. 2000, Cambridge, MA, 3-days of presentations and interactions during which PsychExperiments was demonstrated.
•Society for Computers in Psychology, Annual Meeting. "Website Demonstration: PsychExperiments" with John Williams. Nov. 2001, New Orleans, 1 hour.
•Colloquuium University of Arizona hosted by Marketing Department. "Issues and Opportunities in Web-based Data Collection: An Introduction to and Demonstration of PsychExperiments," 1 hour, interdisciplinary social science
•Society for Consumer Research, Annual Meeting. "Web-based Data Collection: Issues and Opportunities for Consumer Psychologists" with Alan J. Malter, University of Arizona, Feb. 2001, Scottsdale, AZ 1 hour.

Professional Publications

The following is a list of publications that have resulted from our grant-related efforts:

•Williams, J.E., McGraw, K.O., & Tew, M.D. (1999). Undergraduate labs and computers: The Case for PsychExperiments. Behavioral Research Methods, Instrumentation, & Computers, 31,287-291.
•McGraw, K.O. , Tew, M.D., & Williams, J. E. (2000). PsychExps: A On-line Psychology Laboratory (pp. 219-233). In M. Birnbaum (Ed.) Psychology experiments on the Internet. Orlando, Academic Press.
•McGraw, K.O. , Tew, M.D., & Williams, J. E. (2000). The integrity of web-delivered experiments: Can you trust the data? Psychological Science, 11, 502-506.
•Tew,M.D., Inman, M.J., & McGraw, K.O. (Under review). The accuracy of response timing by Authorware Programs. Behavior Research Methods, Instruments, & Computers.

Publicity in the Press

The website was featured in serveral notable publications, as documented below.

•"Now Available On-Line: Interactive Experiments," Feature article in the APS Observer, Sept. 1998, vol. 11, No. 5, pp. 14-15, by Kristen Bourke (1 photo).
•"New Tool Makes Online Research Easier," Front Page article in APA Monitor, Nov. 1998, vol. 29, No. 11, Page 1, continued on page 36, by Bridgett Murray (2 photos).
•"Free of Charge: Open All Hours," Feature article in APA Monitor on Psychology, April, 2000, vol. 31, No. 4, pp. 48-49, by Bruce Beans (1 photo).
•"The Virtual Maze: A New Site Puts Classic Psych Experiments Online," Feature story in University Business, Dec. 2000/Jan. 2001, vol.3, p. 58 by David Glenn.
•"Psychology Experiments", Feature article in Tech Trends, the magazine of the Association for Educational Communications and Technology, Mar/Apr 2001, vol. 45, No. 2, pp.7-8, by Ken McGraw, Mark D.Tew, and John E. Williams.

Awards

The following is a list of awards that have been given to PsychExperiments.

•1^st Place in Learning Software Design competition hosted by the Univ. of Minnesota.
•Honorable mention in Macromedia's eLearning Award competition (June, 2000) and winner of Macromedia's Technical Excellence Award in this competiion.

Continuation and Dissemination

Our ultimate goal for PsychExperiments at this point is that it be taken over by one of the professional organizations-either APS or APA-and offered to members as a benefit at low or no cost. To make this happen, we need to continue building our user base, improving the site technology, and training more developers. Funding we have received from NSF to develop training materials and to purchase a high-end server with state-of-the-art security and reliability will be very important in this effort. As the site continues to become an established part of the infrastructure for education in psychology, it will be easier to convince the professional organizations that operating the site will be good both for psychology education and for their organizations.

G. SUMMARY AND CONCLUSIONS

The major lesson we have learned is that Romer's Rule on the diffusion of innovations prevails in the academe as it does in biological evolution. This means that change in academia is conservative change, typically forced rather than elected. For faculty who have adequate instructional resources at present, the availability of web technology that offers unprecedented opportunities to build collective resources and to establish effective collaborations is not a sufficient enticement to promote involvement in the new technology. Faculty in less well funded institutions, however, are much more likely to be dissatisfied with existing resources for doing their job, and these are the ones who are most interested in the novel possibilities available via the Web. We have been struck, therefore, by the quick and eager acceptance PsychExperiments has received from faculty at the smaller, less significant institutions in this country. When we solicited applications for our workshops, for example, they came almost invariably from faculty at schools we had never heard of. Unsolicited e-mails arrive from faculty in these schools as well. Seldom do we hear from anyone at a major state institution or any of the prestigious, well-funded private institutions. This has been a marvel to us, and it has been extremely gratifying to think that PsychExperiments might be serving to level the playing field a bit. Nonetheless, the general reluctance to adopt the new that characterizes so many academics has made clear the importance of sustained outreach. A build-it-and-they-will-come approach may work for Iowa ball fields but not for innovations in higher education.

APPENDICES

A. Registered Users in March 2001

B. Workshop Participants

C. Copies of Publications

D. Copies of Press Reports

Appendix A

List of Users

Institution	Classroom	Region
Acadia University	Sullivan's Psyc 2023	ONA
Acadia University	Sullivan's Psyc 3083	ONA
Adams State	Kubeck's Cognitive Psychology	USNW
Adams State College	Riniolo's Experimental Psychology	USSW
Adrian College	Hammerle's Research Methods	USNE
American College Dublin	Yore's Experimental Design	EUR
Anderson University	Lockhart's Capstone Research	USNE
Arizona State University	Floyd's Nonverbal Communication	USSW
Arkansas State Univ.	Yanowitz's Cog.Psy.	USSE
Auburn University Montgomery	Mills' Intro to Psych	USSE
Auburn University Montgomery	Mills' Perception	USSE
Augusta State Univ.	Topolski's Introduction to Psychology	USSE
Augusta State Univ.	Topolski's PSYC3122 Research Methods	USSE
Augusta State Univ.	Topolski's PSYC4165 Cognition and Learning	USSE
Barry University	Szuchman's Psy 333	USSE
Berkley College	Eberle's Intro Psych	USNE
Bethel College	Krehbiel's PSY211 General Psychology	USSW
Bethel College	Young's Cognitive Psychology	USSW
Birmingham Southern College	Pitts' Cognitive Psychology	USSE
Birmingham Southern College	Pitts' Research Methods	USSE
Blinn College	Thomas's General Psych 2314	USSW
Bluffton College	Nath's IntroPsych1	USNE
Bluffton College	Nath's IntroPsych2	USNE
Bluffton College	Nath's Psych Assts	USNE
Bournemouth University	Taylor's BSc Applied Psychology and Computing	EUR
Brookdale Community College	Richardson's Human Development 1	USNE
Brookdale Community College	Richardson's Intro Psych	USNE
Brown University	Cohen's Neuropsychology	USNE
Cabrini College	Tomasco's PSY 122	USNE
Cabrini College	Tomasco's Research Practicum	USNE
California State Polytechnic University	Mayo's Psy 433	USSW
California State University(San Marcos)	Grimshaw's PSYC393	USSW
Campbellsville University	Simipson's Experimental Psychology	USSE
Carl Sandburg High School	Matiya's AP Psych	USNE
Carnegie Mellon University	Fay's Research Methods	USNE
Central Lakes College	Dorschner's Psy 2421	USNE
Central Lakes College	Dorschner's Psy 2421 Summer	USNE
Cheltenham and Gloucester College of Higher Education	Mason's Investigative Methods	EUR
Cheltenham and Gloucester College of Higher Education	Mason's PY208
Chowan College	McKemy's Clinical Psychology	USSE
Chowan College	McKemy's General Psychology	USSE
Chowan College	McKemy's Developmental Psychology	USSE
Chowan College	McKemy's Psych and Law	USSE
City University of Hong Kong	Lai's Perception and Cognition	ASIA
City University of Hong Kong	Lai's Psychological Principles	ASIA
City University of Hong Kong	Lai's Psychology Principles Applied to Science and Technology	ASIA
College of Charleston	Marcell's Lab in Cognitive Psy	USSE
College of Mount St Vincent	Berger's Learning and Cognition	USNE
College of Southern Idaho	Frigo's General Psy	USNW
Columbia College	Shirley's Psych 495	USSE
Columbus State Univ.	Dugas' Research Design and Methodology	USSE
Columbus State University	Schmidt's Psyc 4285	USSE
Concordia College	Buslig's Methods of Inquiry (COM 201B)	USNW
CSIU LPN Career Center	Nunan's Psychology	USNE
Cumberland College	Huffman's Psyc 344	USSE
Curtin University of Technology	Barrett's Research Methods 111	AUS
Curtin University of Technology	Pollock's Percpetion	AUS
De La Salle University (Manila)	Janetius's Pilot Study	ASIA
Deakin University	Armatas' HPS201 Research Methods A	AUS
Delta State University	Hamon's Psychological Tools	USSE
Delta State University	Hutchens Psy 314 Social Cog	USSE
Delta State University	Hutchens Learning and Cognition	USSE
Delta State University	Hutchens Sensation and Perception	USSE
Denison University	Hassebrock's Cognitive Psychology	USNE
Denison University	Hassebrock's General Psychology	USNE
Dominican University of California	Matthews' Directed Research	USSW
East Stroudsburg University	Experimental Psychology	USNE
Eastern Michigan University	Bonem's PSY 301	USNE
Eastern Nazarene College	Ross's Psy. 492	USNE
Edinboro Univ.	Levine's Psy 227	USNE
Edinboro Univ.	Culbertson's Psy 227	USNE
Edinboro Univ.	McLaren's Psy 227	USNE
Eisenhower High School	Andrews' AP Psychology	USNE
Emory and Henry College	Crabb's Psych 440: Cognitive Science	USSE
Escuela Campo Alegre	Jenkin's IB Psychology HL	SA
Fachhochschule Kaiserslautern	Dinter's Learning Methods	EUR
Fæærøøernes Gymnasium	Alstrup's Psychology	EUR
Florida International University	Beneckson's Psy2020	USSE
Florida State University	Stanfield's EXP 3503L	USSE
Florida State University	Tuffiash's EXP 3503	USSE
Føøroya Studentaskúúli	Alstrup's Sáálarfrøøðði	EUR
Francis Marion University	Wages' Research Methodology	USSE
Francis Marion University	Warner's Sensation and Perception Lab Research	USSE
Franklin County High School	Guess's Criminal Justice	USSE
Fullerton College	Kyle's Research Methods	USSW
Fullerton College	Wolfe's Research Methods	USSW
George Fox Univesity	Koch's Cognition	USNW
George Washington University	Graf's SMPA 51	USNE
Georgetown University	Howard's Cognition	USNE
Georgia Tech	Topolski's Cognition Lab	USSE
Gettysburg College	Arterberry's Psy 205	USNE
Gettysburg College	Arterberry's Psy316	USNE
Glendale Community College	Jacobs' Research Methods	USSW
Goddard College	Weiss' Biopsychology	USNE
Grand Valley State University	Gross's Cognitive Psychology	USNE
Grand Valley State University	Gross's Cognitive Psychology 2	USNE
Grand Valley State University	Gross's PSY 365	USNE
Guilford College	Kannenberg's General Psychology	USSE
Gustavus Adolphus College	Walker's Statistics Research Methods II	USNE
Hamline Univ.	Olson	USNE
Hanover College	Krantz's Cognitive Psychology	USNE
Harriton High School	Young's Psychology	USNE
Hillside High School	Gorham's Psych Exper	USSE
Hofstra University	Dr. Nguyen's Psy 190	USNE
Hofstra University	Dr. Paul's Psy 141	USNE
Holy Family College	Meinster's Advanced Research	USNE
Holy Family College	DellaPietra's Cognitive Processes	USNE
Humboldt	Lang's Social Regulation	EUR
Illinois State University	Hardwick's Psy 331.01	USNE
Indiana Univ.	Criss's P211	USNE
Indiana Univ.	Lawton's Psy 203	USNE
Indiana University	Bordens' Research Methods	USNE
Indiana University	Bordens' Psy203	USNE
Indiana University East	Shapiro's S360	USNE
International Baccalaureate program		OTH
International Baccalaureate Psychology		OTH
International Islamic University	Ansari's Psychology Experiments	AUS
IUPUI	Wickelgren's B340	USNE
James Madison University	Serdikoff's Experimental Psychology	USSE
James Madison University	West's PS 380	USSE
Jefferson College	Bergmire's General Psychology	USSE
Jefferson Sci and Tech	Hannah's Intro Psych	OTH
Jim Hill High School	Thurman's IB Psych	USSE
John Brown University	Froman's Cognitive Psychology	USSE
Juniata College	Widman's PY301	USNE
Lafayette College	Hill's Psyc 323 Evening Lab	USNE
Lafayette College	Hill's Psyc 323 Afternoon Lab	USNE
Lafayette College	McGillicuddy's Design and Analysis	USNE
Lafayette College	Hill's Psyc 323 Demo	USNE
Lafayette College	McGlone's Cognitive Psychology	USNE
Lafayette College	McGlone's Psychology 203 Research Methods Lab	USNE
Lake Forest College	Brekke's Psyc 222	USNE
Lehigh University	McRoberts' Psyc 210	USNE
Lehigh University	Williams' Experimental Methods	USNE
Lehigh University	Williams' Psyc 210	USNE
Lincoln Southeast High School	McEntarffer's Diff Psychology	USNW
Lindsey Wilson College	Sargent's General Psychology	USSE
Lomo Linda University	Morton's Research Methods	USSW
London Guildhall University	Ungar	EUR
Loyola Marymount University	Foy's Research Methods	USSW
MacArthur High School	Rodriguear's AP Psychology	USSW
Madonna University	Woods' Human Learning and Memory	USNE
Mercer University	Peluso's Psy 385.002 Human Memory	USSE
Mercy College	Kelly's PSYN 372 Experimental Psychology	USNE
Michigan Tech University	Avond's PSY 261	USNE
Michigan Tech University	Avond's PSY 481	USNE
Middle Tennessee State University	Bernstein's Cognitive Psychology	USSE
Middle Tennessee State University	Bernstein's Psy 0307	USSE
Millsaps College	Mathis' Cognitive Psych 3100	USSE
Millsaps College		USSE
Mississippi College	Velkey's Psy 336	USSE
Mississippi College	Velkey's Psy 391	USSE
Mississippi College	Velkey's Special Project	USSE
Molloy College	Spata's Psy 210	OTH
Molloy College	Spata's Psy 211	OTH
Monmouth University	Payne's Cognition and Perception	USNE
Montclair State University	Adams' Experimental Psy 301	USNE
Montclair State University	Garcia's PSYC 301	USNE
Morris Brown College	Lancaseter's Theories of Learning Cog. Psychology	USSE
Murdoch University	Pohl's Masters Human Resource Mangement	AUS
Neenah High School	Frailing's Psychology	USNE
Neumann College	Novi's Psychology	USNE
Newnan High School	Whitlock's Psychology	USSE
Nimitz H.S.	Miller's AP Psychology	USSW
North Central College	Coon's Cognitive Psychology	USNE
Northeast Junior College	Billingsley	OTH
Northeastern University	Carney's Research Methods	USNE
Northeastern University	Quinn's Research in Psychology	USNE
Northern Arizona U.	Hunt's PSY 290	USSW
Northern Arizona U.	Hunt's PSY 290	USSW
Northern Arizona U.	McCarrier's Psychology 290	USSW
Northern Kentucky U.	Goedel's Psy210L	USSE
Northern Kentucky U.	McDaniel's Psy 210L	USSE
Northern Kentucky U.	McDaniel's Psy 338L	USSE
Northwest College	Leach's Experimental Psychology	USNW
Occidental College	Chapman's Research Methods	USSW
Ohio Dominican College	Cohn	USNE
Ohio University	Wolfs's Sensation and Perception	USNE
Oklahoma City U.	Jowaisas' Applications in Psy.	USSW
Oklahoma City U.	Jowaisas' Experimental Psy.	USSW
Pacific Lutheran Univ.	Shore's Intro Statistics/Research Methods	USNW
Pacific Lutheran Univ.	Shore's PSY 348	USNW
Pacific Lutheran University	Broeckel's Cognitive Psychology Lab	USNW
Pacific Union College	Fulton's Psych of Learning and Cognition	USSW
Park University	Oxley's PS305 Behavioral Research Methods	OTH
PennState Erie	Blasko's Psy 201W	USNE
PennState Erie	Kazmerski's Psy 201W	USNE
PennState Shenango	Metzger's PSY 002	USNE
Plymouth State College	Zehr's PS322	USNE
Pomona College	Alvarado's Psych 51 MW	USSW
Pomona College	Alvarado's Psych 51 TuTh	USSW
Pomona College	Greene's Psych 51	USSW
Poplar Springs Elementary School	Case's 2001 PSE Science Fair	USSE
Pueblo School for Arts and Sciences	Martinez's Intro to Psych	USSW
Rancho Buena Vista High School	Hermanson's IB Psychology	USSW
Rhodes College	Wetzel's Psych 200	USSE
Rice University	Hebl and Skorinko's Honors Thesis	USSW
Rio Hondo Community College	Pilati's Psychology 101	USSW
Rochester Inst. of Tech.	Isaak's Attention and Pattern Perception	USNE
Rollins College	Walker's Cognitive Psychology	USSE
Ruamrudee International School	Arti's IB Psychology	ASIA
Ruamrudee International School	Faitel's IB Psychology	ASIA
Sackville Community College	Gollan's AS Psychology	EUR
Salem State College	Miller's Sensation and Perception	USNE
Sampson Community College	Wolf's General Psychology Dreams	USSE
San Jose State U.	Jara's Psych 125	USSW
San Jose State University	Van Selst's Introduction to Research Methods	USSW
Sayville High School	Recker's AP Psychology	USNE
Shawnee State University	Stafford's Quantitative Methods	USNE
Silver Lake College	VandenAvond's Cognitive Psychology	USNE
Sofia University	Tsvetkova's Perception	EUR
Sonoma State University	Smith's Psychology 380	USSW
Sonoma State University	Smith's Psy 250	USSW
Spelman College	Ama and Ashiya's Social Psychology	USSE
Spelman College	Brakke's Cognitive Psychology	USSE
Spelman College	Smith's Experimental Design Lab	USSE
Spring Hill College	Collins' Experimental Psychology II	USSE
St. Andrews Presbyterian College	Case's Psy 101	USSE
St. Andrews Presbyterian College	Case's Psy 202	USSE
St. Anselm College	Flannery's PY09A Intro Psych	USNE
St. Anselm College	Flannery's PY09B Intro Psych	USNE
St. Anselm College	Flannery's PY12B Exp Psych II	USNE
St. Anselm College	Gonsalves' Psych of Learning and Motivation	USNE
St. Anselm College	Krauchunas's Experimental Psy	USNE
St. Anselm College	McKenna's Introductory Psychology	USNE
St. Anselm College	McKenna's Experimental Psychology	USNE
St. Bonaventure U.	Walker's Expr. Social Psychology	USNE
St. Cloud State U.	Melcher's Human Cognition	USNE
St. Norbert College	Korshavn's PS100 PS301 PS321	USNE
Stockholm University	Yonker's Intro Psy	EUR
Suffolk County Community College	Green's Introduction to Psychology	USNE
SUNY (Brockport)	Birch's Cognitive Processes	USNE
SUNY (Buffalo)	Kreiger's Cognition	USNE
SUNY (Plattsburgh)	Braje's Psy Labs	USNE
SUNY (Potsdam)	Pate's Research Methods in Psychology	USNE
Syracuse University	Bopp's Cognitive Psychology	USNE
Syracuse University	Johnson's Psy 313	USNE
Tarlac State University	Lansangan's Behavioral Studies	ASIA
Texas AM Univ.(Kingsville)	Gonzalez's Internet Addiction Disorder and Relation to Impulse Control	USSW
Texas Christian University	Bahr's pob	USSW
Texas Christian University	Bahr's juror	USSW
Texas Tech University	Reifman's hdfs 3390	USSW
The New Atlanta Jewish Communnity High School	Finkel's Psych 401	USSE
Trinity College of Graduate Studies	Linamen's DS702	EUR
U. of Alaska Anchorage	Ragan's PSY 261L	USNW
U. of Arizona	Umashankar's MKTG 361 Intro to Marketing	USSW
U. of Baltimore	Markowitz's Sensation and Perception	USNE
U. of Calgary	Bodner's Human Learning and Memory	ONA
U. of California(Riverside)	Chiarello's Psych Research Methods	USSW
U. of California(Riverside)	Chiarello's Research Methods	USSW
U. of California(Riverside)	French's Psych 11	USSW
U. of California(Riverside)	Rosenblum's Psy 1	USSW
U. of Cambridge	Brooker's Exercise Physiology	EUR
U. of Cincinnati	Reilley's Intro to Psych	USNE
U. of Connecticut	Cooper's 132 Honors Lab	USNE
U. of Connecticut	Galantucci's Psychology 132 lab	USNE
U. of Denver	Keenan's Psyc 2050 Research Methods	USSW
U. of Guelph	PsychCrew's Psych 3371	ONA
U. of Guelph	Meegan's Psyc 2650	ONA
U. of Huddersfield	Webb's Introduction to Research Methods 2	EUR
U. of Indiana	Huber	USNE
U. of Joensuu	Timonen's Experimental Psychology	EUR
U. of Kent at Caterbury	Randsley de Moura's Group Processes	EUR
U. of Liverpool	Scutt's Research Methods and Statistics	EUR
U. of Louisiana(Monroe)	Stevens' Psyc 316	USSE
U. of Louisiana(Monroe)	Stevens' Psyc 325	USSE
U. of Massachusetts	Byrnes' Experimental Methods	USNE
U. of Mississippi	Bryan's Developmental 310	USSE
U. of Mississippi	Chen's Psy 214A	USSE
U. of Mississippi	Lawrence's Pol 101 Sec 2	USSE
U. of Mississippi	McGraw's Psy 214	USSE
U. of Mississippi	Gregoire's Psy 405	USSE
U. of North Alabama	Hudiburg's Cog. Psych.	USSE
U. of Nothumbria at Newcastle	Briggs Psychology Lab Experience	EUR
U. of Queensland	Smith's PY357/423	AUS
U. of San Diego	Taylor's Cognition/Intro	USSW
U. of South Florida	Pezzo's Research Methods	USSE
U. of Southern Indiana	Evey's Psy 312	USNE
U. of Southhampton	Gregg's Online Study	EUR
U. of Sunderland	Lewis' Psy223 Research Methods II	USSE
U. of Tasmania	Summer's Learning and Skilled Perf	AUS
U. of Texas Austin	Seraphine's Experimental Design	USSW
U. of The Basque Country	Huizi's Thinking and Language	EUR
U. of Westminster	Gardner's Cognitive Psychology	EUR
U. of Wisconsin	Bermant's Exper. Psych	USNE
U. of Wisconsin Superior	Carroll's Senior Lab	USNE
U. of Zambia	Menon's Social Psychology	AFR
United States Air Force Academy	Smith's Resesarch Methods and Statistics	USSW
Universidad Catolica Argentina	Fernandez's Psicologia General	SA
Universidad de Malaga	Soler's Metodologia Experimental	EUR
Universitéé de Moncton	Muise's PS2100	ONA
Universitéé de Moncton	Muise's PS2120	ONA
University of California(Riverside)	Reynolds' Research Methods	USSW
University of Denver	Keenan's Research Methods	USSW
University of Iowa	Klohnen's Evaluating Psychological Research	USNE
University of Kiel	Doll's Experimental Design	EUR
University of Minnesota(Duluth)	Durgunoglu's Cognition	USNE
University of Mississippi	Broom's Psy 201 Section 3	USSE
University of Mississippi	Roach's Psychology 201	USSE
University of North Alabama	Bates' Thesis	USSE
University of North Carolina(Wilmington)	Pitt's Psy 355	USNE
University of Puget Sound	Hale's Cognitive Psychology	USNW
University of Rhode Island	Horowitz's General Psychology	USNE
University of Rhode Island	Horowitz's PSY113	USNE
University of San Francisco	Livesay's Research Methods	USSW
University of the North	Khumalo's Isivivane	AFR
University of the North	Khumalo's Isivivane Project	PCY
University of Wisconsin (Parkside)	Walter's Cognition and Memory	USNE
University of Wisconsin(Whitewater)	Aks' Research Methods	USNE
University of Wisconsin(Whitewater)	Eamon's Research Methods	USNE
University of York(UK)	Blundell's Practical Course	EUR
USAFA	Foster's BS 200	USSW
USYD	Sally's Psychology 1b	AUS
UW	Eamon's 216	USNE
UW	Eamon's Research Methods	USNE
Wabash College	Bost's Learning and Cognition	USNE
Washington State University	Schultz's MVTST 461	USNW
Washington State University	Whitney's Psych 592	USNW
Wellesley College	Titone's Psychology of Language	USNE
Willamette University	Stewart's Gen Studies 140	USNW
Winona State University	Fried's Experimental Psych	USNE
Wittenberg University	Brown's Wittenberg Intro Psyc	USNE
Wollongong University	Darren's 3rd Year Lab	AUS
Worcester University College	Haigney's Cog. Psychology	USNE
York College of Pennsylvania	Chang's Psych 354 Snesation and Perception	USNE

Appendix B

Workshop Participants

Name	Institution	Workshop	No. of exps
Chow, Siu	Univ. of Regina, Regina Canada	May-99	1
Hutchens, Scott	Delta State Univ., Cleveland, MS	May-99
Levine, Gary	Edinboro Univ. of PA	May-99	2
Marcell,Michael	College of Charleston, Charleston, SC	May-99	2
McDaniel, Cyndi	Northern Kentucky Univ., Highland Heights, KY	May-99	1
Robinson, Tim	Gustavus Adolphus College, St.Peter, MN	May-99
Sullivan, Jill	Acadia Univ. Wolfville, Nova Scotia	May-99
Velkey, Andrew	Mississippi College	May-99	3
Crutcher, Robert	Metropolitan State College of Denver (CO)	May-99
Wetzel, Chris	Rhodes College (Memphis, TN)	May-99	1
Noland, Keith	Univ. of Mississippi	May-99
Chen, Zhe	Univ. of Miss.	Aug-99
Flannery, Kathleen	St. Anselm College	Aug-99	1
Jowaiss, Dennis	Oklahoma City Univ.	Aug-99
Mendolia, Marilyn	Univ. of Miss.	Aug-99	1
Padmanabhan, Sudevan	Univ. of Wisconsin-Stevens Point	Aug-99
Mitchell,Mark	Clarion Univ.	Aug-99	2
Topolski, Richard	Augusta State	Aug-99	1
Varnhagen, Connie	Univ. of Alberta	Aug-99	2
Arterberry, Martha	Gettysburg College (Gettysburg, PA)	May-00	1
Birnbaum, Michael	Univ. of California--Fullerton	May-00
Blasko, Dawn	Penn State Erie, the Behrend College	May-00	1
Brown, Clifford	Witeenberg Univ. Springfield, Ohio	May-00	1
Hevern, Vincent	LeMoyne University Syracuse, NY	May-00
Kocel, Katie	Jackson State Univ. Jackson, MS	May-00
Ragan, Shawna	Univ. of Alaska (Anchroage, Alaska)	May-00	1
Titone, Debra	Harvard Medical School	May-00	2
Willman, David	Viterbo College (LaCrosse, WI)	May-00
Wilson, Bill	Gettysburg College (Gettysburg, PA)	May-00	1
Acker, Michelle	Otterbein College Westerville, OH	Aug-00
Carroll, David	Univ. of Wisconsin--Superior	Aug-00
Chalikia, Magdalene	Moorhead State Univ. (Moorhead, MN)	Aug-00
Heaps, Christopher	Linfield College (McMinnville, OR)	Aug-00
Klohnen, Eva	Univ. of Iowa	Aug-00	1
Malter, Alan	Univ. of Arizona	Aug-00	1
Metzger, Mitchell	Penn State Shenango	Aug-00	3
Pezzo,Mark	Wake Forest Univ.	Aug-00	1
Smith, Heather	Sonoma State Univ.	Aug-00	1
Stevens, Rick	Northeast Louisiana (Monroe, LA)	Aug-00
Wilson, Kelly	University of Nevada	Aug-00
Sharpe, Will	University of Mississippi	Aug-00	2

1. Logicians will assert that the null cannot be verified; however, when effect size estimates are near zero despite overwhelming power to detect an effect if present, the truth of the null becomes likely.

Convenience	•38% of students polled strongly agreed and 31% agreed (Total=69%) that using PsychExperiments to collect data was more convenient than the alternatives available to them. •28% of student users reported using a personal (non-university owned) computer to participate in experiments. Psychology department computers were used by just 40% of the students. •Better than 20% of the experiment sessions were conducted weekends, or at weekday times when university computers and labs would not be available.
Large datasets	•Better than 18,000 experiment sessions had been conducted at PsychExperiments by mid-March of this academic year with four experiments having in excess of 1000 data records. The top ten are Mental Rotation - 2208, Self Reference - 1753, Lateralized Stroop - 1463, Reaction Time Color - 1403, Perception of Gender - 1366, Word Recognition - 1133, Mueller Lyer - 917, Line Motion - 842, Poggendorff - 801, and Covert Attention - 546
Large datasets permit study of phenomena with small effects and the determination of null effects⁽¹⁾	Our data permit the inference that •the effect of fin angle on the Mueller Lyer illusion is not a linear function of angle, rather it is a sinusoidal function •the length of diagonal lines in the Poggendorff has a small but reliable effect on the magnitude of the illusion •the Stroop effect is the same for centrally presented stimuli as for laterally presented stimuli
	•there is no gender X visual field interaction in word recognition speeds (which would be hypothesized based on the view that the males' brains are more lateralized than females' brains). •the gender difference in mental rotation time does not depend on the degree of angular disparity between the stimuli being judged (which would be hypothesized based on the view that males execute rotations at a faster rate) •There is an unanticipated gender effect in the Mueller-Lyer and Ponzo illusions with females experiencing larger illusions than males.
Large datasets permit distributional analyses	•Typical laboratory datasets have so few observations per experiment condition that a distributional analysis of the data is uninformative. Instructors using PsychExperiments data can (and do) have students produce histograms, stem-and-leaf plots, or box plots for depicting their data as well as compute skew and kurtosis.
Speed of data collection	•The database has many sets of nearly simultaneous data submissions. This indicates that the submissions came from students working in parallel in a computer lab. Such students can get all the data they need for a class project in a single class period. •Andi Agnew (Mississippi College undergraduate), C.J Yu (UM graduate student), Amy Criss (U. Of Indiana graduate student), and Clint Moody, Shari Gregoire, Jennifer Mann, and Frances Patterson (UM undergraduates) collected data for undergraduate and master's theses projects in a fraction of the time that it would have taken had they been collecting data serially from research participants who were given scheduled times to appear in a laboratory.
Student learning and interest	•17% of students polled strongly agreed and 46% agreed (Total=63%) that their educational experience in psychology had been enhanced by PsychExperiments; 21% strongly agreed and 48% agreed (Total=69%) that their learning in a specific class was enhanced; 19% strongly agreed and 45% agreed (Total=64%) that their interest in a specific class had been enhanced by use of PsychExperiments. Written comments were uniformally positive •Instructors estimates of student benefits on the same three items (general enhancement, learning in a specific class, interest in a specific class) were higher: 91% strongly agreed or agreed that the educational experience had been enhanced, 94% strongly agreed or agreed that learning in a specific class had been enhanced; and 85% strongly agreed or agreed that their students' interest had been enhanced.
Alterable programs	•The interest in learning to use Authorware as an experiment generator has been high. We had 72 applications for 40 slots in our workshops and off-site workshops drew well. For example, the SPSP (February, 2000) and CiPP (March,2000) workshops were at capacity (40 and 25, respectively).
Cost saving	•53% of faculty users polled strongly agreed and 29% agreed (Total=82%) that using PsychExperiments would save their departments money long term.