Operational Stage

The Operational Stage is the one in which the experiment is live and potentially delivering content to POPROX subscribers. A multi-phase experiment may have phases in which only baseline recommendations are being delivered; for example, researchers may wish to ask custom survey questions before beginning recommendation delivery. Technically, the experiment is operational when its manifest is incorporated into POPROX’s newsletter delivery, regardless of what code is generating recommendations.

POPROX monitoring

POPROX staff will be closely monitoring newsletter output and other metrics related to live recommendation code and will be in close contact with researchers throughout the operational phase. Researchers should designate a member or members of their team as contacts for any questions or concerns that might arise. POPROX reserves the right to terminate an experiment if problems arise that cannot be promptly remediated.

Researcher output

Researchers will receive daily emails with statistics about the progress of their experiments, including the number of emails generated, statistics about the latency of recommendation generation, the clickthrough rate of delivered articles, the responses to surveys or other user feedback, and opt-out activity. (Note that participants can opt out of an experiment at any time and may also choose to exit the POPROX system entirely.)

Experiment termination

Experiments may be terminated in the Operational phase for a variety of reasons:

• Researcher code failing to live up to minimum technical standards. The purpose of the testing phase is to catch such problems before the experiment goes live.

• Negative participant reaction that, in the opinion of the research team, threatens POPROX as a platform: for example, very high opt-out rate.

• Researchers may also choose to terminate their experiment at any time.

Termination is decision that the POPROX team takes very seriously as it may be irreversible and means that our scarcest resource – our participant’s time and attention – is being largely wasted. We will be whatever we can to avoid terminating an experiment.