Measurement

There are five aspects of measurement for POPROX experiments:

• Offline studies using POPROX data: As experimenters develop their code, they will have access to data that they can use to train algorithms and conduct studies in advance of delivering live recommendations.

• Pre-experimental testing: During the Testing stage, we will reporting a suite of performance characteristics to experimenters to ensure that their code is performing as expected and meets POPROX requirements. See Testing stage documentation for more inforomation.

• Onboarding questionnaires: When new users sign up for the system, they are asked to complete a set of onboarding questionnaires. Some of this information can be used by researchers: for example, in selecting or sampling their users.

• Baseline survey: POPROX users are surveyed every four weeks with a set of survey questions about their experience using the system and satisfaction with the recommendations that they are getting.

• User interaction: Currently, POPROX only supports the recording of clicks that users make on the links in their emails. Clicks are tracked by the system and reported on daily in the (Operational stage)[operational.md] and in summary form when the experiment is complete (Closeout stage)[closeout.md].

Document offline data exports Document onboarding questionnaires and baseline surveys