From Dartmouth to deep learning

Two tracks ran side by side for fifty years. One put knowledge into machines by hand. The other let machines learn it from examples. The second track won, and the reason it won is the first thing to know before you put a model in production.

Knowledge by hand

The 1955 proposal for the Dartmouth summer workshop gave the field its name. Early symbolic programs solved toy problems, then hit combinatorial explosion as the problems grew. The 1973 Lighthill report to the UK Science Research Council said so in writing, and the funding cuts that followed are what people mean by the first AI winter.

Expert systems brought the approach back in the 1980s. XCON at Digital Equipment Corporation configured computer orders from rules drawn out of engineers one at a time. Every rule had to be written and maintained by a person, which is the knowledge bottleneck on the slide, and the market for specialised AI hardware collapsed in 1987.

Learning from data

Rosenblatt’s perceptron learned a linear classifier from examples in 1958. Minsky and Papert showed in 1969 that a single layer cannot compute XOR, and interest fell away. Backpropagation (Rumelhart, Hinton and Williams, 1986) trained hidden layers. LeNet-5 (LeCun et al., 1998) read handwritten digits on cheques in commercial volume.

Vision still ran on hand-engineered features until AlexNet (Krizhevsky, Sutskever and Hinton, 2012) learned its own features on two GPUs from 1.2 million ImageNet images and took the ILSVRC top-5 error to 15.3 percent, against 26.2 percent for the runner-up. Three things arrived together: a large labelled dataset, GPU compute, and a method general enough to use both. Sutton’s 2019 essay “The Bitter Lesson” states the pattern plainly: general methods that leverage computation win over methods that build in human knowledge.

The same pattern, one level down

It repeats inside an application. Hand-built rules capped expert systems. Hand-built features capped statistical machine learning. Hand-built workarounds cap your upgrades. Every regex that repairs one model’s habit, every prompt clause that steers around one version’s quirk, is a feature engineered by hand for a model that will be retired.

Where it goes wrong

A repair layer fixes the malformed JSON one model tends to emit. The next model emits valid JSON, the repair layer corrupts it, and nothing catches it, because the eval suite was never run on the upgrade.

What to do

• Keep per-model workarounds in one place, named for the model and version they serve.
• Cover each workaround with an eval case, so removing it is a test run and not a guess.
• Make a model upgrade a configuration change plus an eval run, and budget the eval run.