Active Sampling as Sequential Bayesian Quadrature

Introduction:

Many theories of psychology, especially in the memory and decision making literature attempt to describe parts of cognition as sampling based. One example is decision by sampling, which assumes that instances of gains and losses are sampled from memory and judgements are then performed in comparison to the sampled instances. However, most of those theories do not make specific assumption about how people might perform the sampling process, but rather always use sampling completely at random. It seems somewhat odd to believe that the sampling process, no matter if sampled from the environment or memory, should always just traverse the probability space at random. In order to test this assumption, Christoph Niemeyer (a MSc-student of Cognitive and Decision Sciences at UCL) and I designed a simple experiment in which participants had to sample from probability distributions and prior and after to that make comparative judgements.

Experiment:

The experiment we designed was pretty simple and consisted of 4 different trials that were always made up of 4 different parts. The parts were as follows:

  1. Please provide an assessment about a simple measure, for example how tall you are.
  2. Provide a comparative judgement, for example how many of 1000 randomly sampled people you think would be smaller than you.
  3. Sample from a probability distribution (for example, a histogram of 1000 randomly sampled people) sequentially.
  4. Provide a revised comparative assessment.

We recruited 49 participants via Prolific Academic and paid them £1 for their time and effort. We used the following 4 different scenario.

  1. Height (normally distributed).
  2. Emails per day (Poisson distributed).
  3. How often a fair dice has to be rolled until the first 6 appears (geometric distribution).
  4. Performance in a fictitious game (bi-modal mixture of normals).

Results:

We find that people do not sample completely at random and that a Sequential Bayesian Quadrature algorithm (one that tries to estimate the integral as exactly as possible by applying Gaussian Process models) describes participants better than uniform sampling. However, there was quite some variance in the performance of people and the algorithms worked best in scenarios with greater uncertainty (probably that is where active sampling pays off the most).

Conclusion:

At least in our scenario in which participants were allowed to sample from a probability distribution directly, their behaviour did not turn out to be completely random, but was better described by an active learning algorithm. Future research could focus on the question of whether or not people sample from their memory at random and how this assumption could possibly be tested.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s