Experiment Specification#
A POPROX experiment consists of the following:
Experiment metadata
Desired user count and selection criteria
One or more user groups, which may have their own selection criteria
One or more recommenders that provide recommendations, displays, and other components of the news recommendations to users
One or more instruments that measure user experience (e.g. survey instruments)
One or more phases through which the experiment proceeds (in order)
Within each phase, each group can be assigned to a different recommender and/or instrument. This is how users are assigned to treatments. The user remains in the same group throughout the experiment.
Experiments can also be linked: an experiment can be marked as the successor to another experiment. Linked experiments share encryption keys (so they get the same user identifiers); results of an experiment’s measurements are also available to successor experiments for use in both recommendation and user selection. Full details of the syntax for this are in the manifest format. We use linked experiments to implement things such as giving users a screening survey that determines their inclusion or group assignment in the real experiment.
The manifest is documented more fully in the following pages: