The teacher was about 18 times too large for the chip in the IntoMind One and about 60 times too slow. Kris Droste asked what the alternative was: "i mean training the tiny version on its own not as a student. what is the difference?"
The answer became an experiment. On August 1 Kris set the goal: "the goal is to build a foundation model that fits on the nrf along with the firmware weve built." And the comparison: "we can compare training it 'from scratch' to training it by distilling the current model." The plan written that day describes the model as "A small EEG encoder that fits alongside existing nRF52840 firmware, as a tiny version of the same foundation model", not tied to one placement of the electrodes.
Kris added two demands: "its supposed to be versatile," and "the device can sit anywhere on the head." The plan records a third ruling, "EVERYTHING IN THE SIGNAL CHAIN IS CONFIGURABLE", and so the device was set to match the corpus: 500 samples per second and the same filter, which removes slow drift below half a cycle per second.
Could a model work anywhere on the head? The teacher was tested at 24 random placements, eight each with two, four and eight electrodes. With only two electrodes it told women from men with an AUC of 0.644, against 0.574 for band power. With four it scored 0.690 against 0.620, and with eight, 0.741 against 0.642. From one placement to another the scores moved by only 0.04 to 0.06. The plan recorded that this answered "the main risk in the whole approach".
Before any student was trained, the test it had to pass was written down: beat band power on sex and age.
Distillation works like this. The teacher looks at a window recorded at up to 19 positions and sums it up as 384 numbers. The student sees only the few electrodes a device has, two to eight of them, and is trained to produce the teacher's summary of the same window. It learns from information it will never have.
On August 1 six students were trained under the same rules, three from scratch and three by distillation, one of each for three ways of handling the electrode positions: always told where they were, never told, or told half of the time. The students trained from scratch did not beat band power. The distilled ones did. In all 12 comparisons between matching pairs, distillation came out ahead, in 11 of them by more than the margin of error. Telling the student where its electrodes were half of the time worked best. A student that was always told lost 1.24 years of accuracy on age when that information was taken away.
The model chosen had 491,904 parameters. The file the device later loaded for it, with its numbers stored at 8 bits each, is 503,072 bytes. It had been trained once, for 60,000 training steps.
The next day Kris set two directions: "i want to train to largest model that will train and fit on this gpu, and i want to train tiny models successively down to the smallest model that beats bandpower." The large one was never trained. The small ones would be.
That evening Kris asked a plain question: "wait so what does the little encoder actually do?" The answer: "It turns 4 seconds of EEG into 96 numbers." It was, in the AI's words, "a feature extractor, not a detector." Those 96 numbers carried a little more than band power did about a person's age and sex. Whatever a user wants from them, a head, a small add-on trained on those numbers, turns them into an answer.
The student would not meet the device until September.