Measurement#

POPROX collects data on participants use of and impressions of the system through interaction data and regular surveys. Researchers cannot contact or otherwise interact with users so these types of data are the only ones to which researchers have access.

Access is available through the experiment lifecycle. As experimenters develop their code, they will have access to data relative to the whole set of POPROX users that they can use to train algorithms and conduct studies in advance of delivering live recommendations. This same is available for studies are do not require intervention but are based on historical POPROX data.

During the Testing stage, we will report a suite of performance characteristics to experimenters to ensure that their code is performing as expected and meets POPROX requirements. See Testing stage documentation for more information.

Interaction data#

Currently, POPROX only supports the recording of clicks that users make on the links in their emails. Clicks are tracked by the system and reported on daily in the Operational stage and in summary form when the experiment is complete Closeout stage.

Surveys#

POPROX participants are regularly surveyed about their experiences with the platform. There are four different types of surveys.

  • Onboarding questionnaires: When new users sign up for the system, they are asked to complete a set of onboarding questionnaires. Some of this information can be used by researchers: for example, in selecting or sampling their users. Onboarding questionaire Onboarding surveys

  • Baseline surveys: POPROX users are surveyed every four weeks with a set of survey questions about their experience using the system and satisfaction with the recommendations that they are getting. Rotating surveys

  • Annual surveys: Mixed in with the surveys about their POPROX experience, we also include surveys with personal information intended to provide individual profiles of participants. Annual surveys

  • Custom surveys: We support custom surveys that experiments can send to participants in their studies. These are in addition to the other survey participants receive, so we seek to minimize them. Be sure that the existing survey questions will not cover your needs before requesting a custom survey.

Data exports#

There are two types of data exports that POPROX permits.

  • General usage data. This is data that researchers can access in order to test their algorithms. It consists of POPROX usage over a predefined period with a sampling of users.

  • Experiment data. This is data specific to the researcher’s own experiment segmented by treatments and conditions.

Per the researcher agreement, POPROX data is not for redistribution. Participant IDs are hashed in a researcher-specific way, meaning that non-compliant redistribution can be traced to the source. Please don’t get us in trouble with the IRB. Thanks!

Document offline data exports

On this page