Stages#

We organize a POPROX experiment into 5 stages.

  1. Intake

  2. Specification

  3. Testing

  4. Operation

  5. Closeout

Stage 1: Intake#

This is the part of the process where a potential researcher and POPROX determine if a particular study or research question is a good fit for POPROX’s capabilities at the present time.

An experiment has completed the Intake stage when the Intake Form has been approved by the POPROX team.

More about the Intake stage.

Stage 2: Specification#

The next goal is to specify the experiment fully so that it can be executed on the platform. This includes specifying and having approved the human subjects protocol, specifying and having approved the experiment design, specifying and having approved the technical design, and finally crafting the experiment manifest, which is a technical document compiled by the POPROX platform to control the experiment’s operation.

At this stage, we also expect that researchers will be writing the code to operate their recommendation pipeline, although researchers are encouraged to draw from our existing implementations as much as possible.

An experiment has completed the Specification stage when the following are complete:

  • IRB protocol (approved by local institution and POPROX)

  • Experiment design (approved by POPROX)

  • Technical design (approved by POPROX)

  • Experiment manifest (approved by POPROX)

More about the Specification stage.

Stage 3: Testing#

We expect that researchers will have completed much of their implementation before reaching the Testing stage but we expect additional implementation tasks to emerge throughout. There are three types of testing:

  • Offline algorithm testing

  • Offline newsletter production

  • Online newsletter production

A system has completed the Testing stage when, in the determination of the POPROX team, it is ready to be deployed to the POPROX user base.

More about the Testing stage.

Stage 4: Operational#

With testing complete, the experiment launches, which means that the experimenter’s code is responsible for delivering recommendations to POPROX users until the experiment is complete. The researcher team and POPROX staff will be closely monitoring the progress of the experiment for its duration.

The Operational stage is complete when all phase of the experiment design have been executed or when the experiment terminates for some other reason.

More about the Operational stage.

Stage 5: Closeout#

Once an experiment is complete, the experimenter is free to release any resources committed to their recommender’s operation. POPROX will make data and metrics computed over that data available once they are prepared for release.

After an embargo period, typically one year, the experiment design and data will be made public on the POPROX web site as part of our commitment to open science.

More about the Closeout stage.