Research

Methodology, limitations, and what's not measured yet

The record, corrected up front

Earlier drafts of this project reported accuracy figures derived from synthetic test data. Those figures described the data generator, not reading behaviour, and have been withdrawn pending real-participant validation.

What the classifier was trained on

Synthetic data. plainly, and why that's a real limitation

The classifier that runs on the optional camera path was trained on rows generated from hand-written rules, then duplicated with Gaussian noise and split randomly into train and test sets. That means a row's noisy near-duplicate could sit in the test set while the original sits in training. a form of leakage that inflates any accuracy figure computed from it. No real-participant reading study has been run against this classifier, or against the browser-telemetry detection that runs by default. Until one has, no accuracy number for alcoia belongs on this site or anywhere else.

That classifier is also not on the critical path for most readers: it only runs when the camera is on, and the camera is off by default. The primary detection path. pace against difficulty and your own baseline, re-reading, tab focus. doesn't depend on it.

Planned validation

Against WebQAmGaze and OneStop

Real-participant validation is planned against two published eye-tracking-and-reading datasets. WebQAmGaze and OneStop. rather than against data alcoia generated itself. Results will be published here, however modest the numbers turn out to be. A small honest figure, published openly, is worth more than a large one nobody can check.

Grounded in prior work

The mechanism isn't new. the packaging is

Reading-speed and comprehension research going back to Rayner (1998) and Just & Carpenter (1980) establishes that pace and regressions carry real signal about reading difficulty. D'Mello et al. (2016) ran a randomised trial showing that interrupting a drifting reader with just-in-time questions recovered comprehension losses (d = 0.47). alcoia's interruption policy is built directly on that finding, not on an internal metric.

Check it yourself

Dataset, notebooks and classifier are all in the repository

Nothing above is asking to be taken on faith. The synthetic dataset, the training notebooks and the classifier itself are in the repository under tldr classifier/: github.com/alcoia-group/alcoia.