As our paper about smoothness and Gaussian Process learning curves just got accepted to the Cognitive Science proceedings, I thought it would be good to share the following snippet with the world:
What is this about?
We have used approximations to Gaussian Process learning curves in order to establish how the smoothness, variance, and sample size influence a GP’s generalization error. It is assumed that a smaller generalization error corresponds to a higher predictability.
What can we expect?
Theoretically, the generalization error depends much more on how smooth a function is than how much noise (variance) is attached to every observation. This means a rational function learner should value smoothness more than noise.
This is intuitive, because smooth functions allow data to be more strongly aggregated across different input points, whereas anything one can learn about a complex function is very local.
What kind of experiment have we conducted?
We have presented samples from differently smooth kernels to participants and asked them to judge how well they could predict the underlying function. Additionally, we also manipulated the noise level as well as the number of sample points participants saw.
What came out of that?
We have found the following 3 results as expected:
1. More observations lead to higher ratings of predictability.
2. Less noise leads to higher ratings of predictability.
3. Smoother functions lead to higher ratings of predictability.
Moreover, we have found that smoothness is indeed more important for judgements about predictability than the attached noise. Therefore, the predictions derived from a rational function learner match participants’ judgements in this case.
We will run another validation and replication study in order to see if the results we have found so far hold. Afterwards, we want to try out what might happen if we sample from more interesting kernels, for example combinations of different base kernels, thereby assessing how compositional in nature our predictability measurement is.