Can We Record What the Hands Know?

Abstract

A craftsperson rarely works from instructions alone. A glassblower feels the changing resistance of molten glass through the blowpipe. A weaver reads tension through the fingers. A carver hears and feels the moment a tool begins to work against the material rather than with it.

Much of this knowledge is hard to put into words. It is acquired through repetition, observation, correction and direct engagement with materials. If craft is to be documented as more than a record of finished objects, the problem is not simply how to record what a craftsperson does. It is how to record enough of what the craftsperson perceives, judges and corrects for that knowledge to remain meaningful, and possibly transmissible.

Two European research programs have laid important foundations. Mingei showed how to represent craft processes digitally and how to capture expert movement. CRAEFT extended that work into simulation, immersive learning and haptic interaction. A recent evaluation of CRAEFT's glassblowing pilot with apprentices at CERFAV suggests where the limit currently lies: virtual environments helped learners understand a workshop and a process, but they did not convey the fine sensory and motor knowledge needed to perform skilled gestures.

This paper proposes the next research question:

CRAFT — World Craft Depository proposes to explore it through a comparative program that connects structured craft documentation with field observation, multimodal sensing and real-world learning experiments. The objective is not to replace apprenticeship. It is to discover how much more of what apprenticeship transmits can be preserved.

1. The object is not the knowledge

Museums and archives are exceptionally good at preserving evidence of making. A vessel can survive its maker by centuries. A textile can reveal fiber, structure, dye and pattern. A tool can show the wear of repeated use. But the object is evidence of knowledge. It is not the knowledge itself.

CRAFT’s earlier paper, Archiving the Intelligence of the Hand (June 2026), set out the foundation: craft documentation should move from the finished object toward the system that produces it, covering materials, tools, gestures, sequences, decisions, corrections and the social conditions through which knowledge is transmitted. This paper takes up the hardest part of that program: the knowledge a practitioner perceives but rarely says.

That creates a harder problem. A written instruction can tell a maker to apply pressure. It cannot easily communicate how much pressure feels right. A video can show the angle of a tool, but not the resistance the practitioner feels through it. Motion capture can record the trajectory of an expert's hand, but it does not tell us why the hand changed direction.

The closer documentation gets to expertise, the more of the important information becomes invisible.

2. From recording craft to recording movement

The Mingei project (Horizon 2020, 2018–2022) was an important step. It developed a protocol for representing traditional crafts that links tangible elements (materials, tools and products) with actions, processes, practitioners and their historical, economic and social contexts (Zabulis et al., 2022). Its infrastructure included a crafts ontology and an online platform for building multimodal representations of craft knowledge. Its pilots covered glassblowing, silk weaving and mastic cultivation.

Mingei also treated human movement as a documentable part of craft knowledge. Motion capture was used to record practitioners performing craft operations. Unlike video, which records how an action looks from one viewpoint, motion capture can describe the movement of the body in three dimensions, to be analyzed, visualized or transferred to a digital human model.

This matters because craft knowledge is partly procedural: knowing the sequence of actions is not the same as being able to perform them. Mingei showed that expert craft movement can be captured and connected to a structured representation of the craft.

It also leaves questions open. Workshops are not motion-capture studios: heat, tools, restricted space and unpredictable movement all complicate recording, and instrumenting a hand may interfere with the very touch a craft depends on. Whether that happens, and how much, is likely to differ from craft to craft. More fundamentally, even a perfect record of movement would describe what the body did. It would not necessarily explain what the practitioner knew while doing it.

3. From movement to material interaction

CRAEFT (Craft Understanding, Education, Training, and Preservation for Posterity and Prosperity), a Horizon Europe project that ran from 2023 to early 2026, took the research further. Where Mingei established methods for representing craft and capturing action, CRAEFT investigated how craft knowledge might be modeled, simulated and used in education and training: digital craft representations, immersive environments, material simulation and haptic interfaces.

That is an important shift. Craft is not only movement through space. It is interaction between a body, a tool and a material. The same hand trajectory can produce different results depending on pressure, speed, temperature, material condition and tool angle, and skilled making consists partly in reading those variables continuously and responding to them.

CRAEFT's haptic research addresses this directly. A prototype hand-held controller developed within the project, described in its public deliverable D4.2, combines a force sensor with an actuator that renders surface texture, with the aim of letting learners practice the tactile side of craft actions in VR before entering the workshop.

The conceptual move is significant: from recording where the hand goes to recording something of what the hand encounters.

But does that make the knowledge transmissible?

4. The glassblowing test

CRAEFT's pilot with the European Centre for Research and Training in Glassmaking (CERFAV) offers an unusually useful test. Glassblowing is an extreme case of embodied knowledge. The material is constantly changing. Temperature affects viscosity, gravity alters form, rotation must continue while other operations are performed, and the craftsperson must watch, move, feel and anticipate a material that cannot be paused while a decision is considered.

The pilot, evaluated by Crescenzo et al. (2026), used a virtual reconstruction of the CERFAV hot shop in which apprentices could explore the workshop, identify equipment and rehearse a limited number of operations. It was delivered on a standard consumer headset with ordinary hand controllers, not with the haptic controller described above, and it was embedded in an actual apprenticeship program. The authors describe it as exploratory: fewer than twenty participants, self-selected.

Within those limits the results are revealing. The virtual environment proved useful for orientation and discovery: learners could become familiar with the workshop, learn the names and locations of tools, and gain an overview of processes without the risk and pressure of entering the hot shop. But when VR was proposed for learning skilled gestures, experienced apprentices clearly rejected it. They pointed to the missing sensory feedback, the imprecise simulation of viscous glass and the imprecision of controller input. The authors conclude that the simulator is valuable for orientation but cannot substitute for embodied gesture learning.

Because the pilot did not use a haptic device, it cannot tell us whether better force feedback would change that verdict. That is part of the open question.

The authors also tried something technologically simpler, informally: video elicitation. Apprentices and trainers reviewed first-person footage of real making and discussed gestures, decisions and errors. Trainers gave this the strongest positive response of the pilot, provided the discussion was structured, because it created a chance to stop, examine and articulate actions that normally pass too quickly to discuss while making.

A digital environment can teach someone what happens without necessarily teaching them how to do it. Perhaps the first question is not how sophisticated the simulation can become. Perhaps it is what information is actually missing.

5. What remains unrecorded?

Consider an expert correcting a mistake. The movement itself may be small: a change of angle, slightly less pressure, a faster rotation, a half-second pause. Behind it lies a chain of knowledge:

A process description may record none of this. A video may record the correction without showing why it happened. Motion capture may record its exact geometry without identifying the signal that prompted it. A force sensor may show a change in pressure without revealing what the craftsperson felt or intended.

The next challenge is therefore not simply higher-fidelity capture. It is connecting signals to judgment. For this purpose we propose five working questions, to be tested in fieldwork. They are research prompts, not fields of the CRAFT archive record.

Sensory cues

What is the practitioner actually monitoring? Resistance, pressure, vibration, temperature, sound, smell, color, surface behavior, weight or balance may all carry information, and different crafts distribute intelligence differently across the senses.

Material response

What changes in the material when the practitioner acts? A movement becomes meaningful only in relation to its consequence.

Decision

What does the practitioner infer from the sensory information? When does “soft” become “too soft”? What tells a maker that the next stage can begin?

Correction

How does an expert recognize that something is going wrong, and what happens next? The ability to recover may reveal expertise more clearly than the execution of an ideal sequence.

Learning

Which of these signals can a learner be taught to recognize? This is the difference between recording performance and recording knowledge.

6. A proposed research framework

CRAFT proposes a framework in five stages. It is a proposal, not yet a tested method.

01 — Identify

Begin with the practitioner, not the technology. Break a technique into critical operations and find the moments where expertise matters most. What is the maker looking for? What are they feeling? What tells them to continue, stop or correct? Which mistakes are hard for a novice to perceive? The objective is not to instrument an entire craft indiscriminately, but to find the points where tacit knowledge is doing real work.

02 — Capture

Only then decide what to record. Depending on the technique, this could include conventional and first-person video, motion capture, hand tracking, force and pressure sensing, audio, temperature, vibration, tool movement or material deformation. Not every craft needs every technology, and the least intrusive method that captures the relevant information should be preferred. A sensor that stops a craftsperson from feeling the material correctly produces more data but potentially less knowledge.

03 — Interpret

Data does not explain itself. Captured signals should be reviewed with the practitioner and connected to the technique: What happened here? What did you notice? Why did you change pressure? What told you the material was ready? What would a novice probably miss? The recording then becomes not just a trace of movement but an annotated trace of perception, judgment and response.

04 — Replay

Different knowledge may call for different forms of replay. Movement can be visualized, force reproduced through a haptic interface, sound isolated and compared, first-person footage slowed and annotated, material states shown alongside sensor readings. Some knowledge may be better communicated through a physical training device than through VR. The research question should determine the medium, not the other way around.

05 — Test

This is the decisive stage. A learner exposed to captured knowledge should return to the real material. Can they recognize the relevant cue sooner? Reproduce the operation more accurately? Identify an error, and correct it? Need fewer repetitions? Comparison with conventional instruction could begin to show which forms of capture genuinely contribute to learning and which merely produce convincing representations.

The useful question is not whether technology can replace apprenticeship. It is:

7. Why comparative craft research matters

A method developed around one craft risks mistaking the characteristics of that craft for characteristics of craft itself. Glassblowing privileges temperature, timing, continuous movement and changing viscosity. Weaving may depend more on tension, repetition and pattern recognition. Carving introduces grain direction, sound, resistance and irreversible removal. Clay involves plasticity, moisture and changing material states. Lacquer may depend on surface condition, viscosity, humidity and extremely fine visual and tactile judgment. No single capture system should be expected to describe all of them equally well.

This is where CRAFT can contribute something different. A research program could select cases not because they are convenient, but because they present different forms of embodied intelligence. As illustrations only:

Tension: weaving, binding, basketry. Material plasticity: ceramics, clay modeling. Tool resistance: wood, stone, shell or metal carving. Temperature and viscosity: glass, metal, lacquer. Rhythm and coordination: forging, weaving, throwing. Sound and vibration: carving, instrument making, metalwork.

The aim would not be one universal model of the skilled hand. It would be to discover what kinds of knowledge recur across techniques, what remains craft-specific, and which forms of capture suit each. The CRAFT archive, with more than 2,900 techniques across 205 territories, can serve as a discovery and comparison layer for choosing those cases. Its records are compiled from published sources, and they are a map for selecting fieldwork, not a substitute for it.

8. Recording creates responsibility

The better we become at recording tacit knowledge, the more important ownership becomes. A public photograph of a craft object and a high-fidelity model of how a practitioner produces it are not ethically equivalent. A system that records a craftsperson's movement, pressure, timing, corrections and material judgments well enough to become a training resource may be recording something economically and culturally valuable.

Who owns that recording? Who can study it? Can it be used to train people outside the community, to build a commercial simulation, or to train an AI or robotic system? Can permission later be withdrawn? What recognition or value returns to the practitioner whose expertise generated the data?

These questions cannot be added after capture. They have to be part of the capture protocol. CRAFT therefore distinguishes between documenting that knowledge exists and obtaining permission to reproduce, transmit or commercially apply it. Some knowledge may be publicly accessible; some available for research but not reproduction; some suitable for teaching only with practitioner involvement; some should not be recorded at all.

The same principle applies to partnerships. In any comparative work across regions, the practitioners and institutions where a craft lives should help define what is recorded, how it is interpreted, who may access it and what forms of reuse are appropriate. The aim is not to extract knowledge into a single technological model, but to build research in which the knowledge holders are participants.

9. From motion to meaning

One way to read the recent history of digital craft documentation is as a series of increasingly difficult questions. Can we digitize the object? Can we document the process? Mingei helped establish another: can we record the movement? CRAEFT pushed further: can we simulate the interaction and use it in learning? The results now point toward the next question:

Answering it will take more than better motion capture. It means understanding what an expert perceives in a material, how sensation becomes judgment, how judgment produces action, how an error is recognized and corrected, and which parts of that chain another person can learn from a recording.

Some of this knowledge may prove measurable. Some may become visible only when a practitioner watches their own hands and explains what was happening. Some may be reproducible through haptics. Some may require the smell, heat, weight, danger and unpredictability of the real workshop. And some may remain inseparable from years spent beside another human being.

Finding that boundary is not a failure of digitization. It is the research question.

References and research foundations

CRAFT — World Craft Depository. Archiving the Intelligence of the Hand: Toward a Documentation Protocol for Artisanal Technique. June 2026.

Crescenzo, N., Arnaud, D., Fallahian Sichani, P., Kristensen, J.W., Partarakis, N. & Zabulis, X. “Hybrid Craft Training in Vocational Education: Integrating E-Learning and VR in Glassblowing Apprenticeships.” Virtual Worlds, 5(2), 26, 2026.

CRAEFT. Craft Understanding, Education, Training, and Preservation for Posterity and Prosperity. Horizon Europe project, Grant Agreement No. 101094349, 2023–2026. craeft.eu.

CRAEFT. D4.1 Craft simulation and immersive craft training (first version, M1–M18). Public deliverable.

CRAEFT. D4.2 Haptic devices for training, simulation, and design (first draft, August 2024). Public deliverable.

Mingei. Representation and Preservation of Heritage Crafts. Horizon 2020 project, Grant Agreement No. 822336, 2018–2022.

Zabulis, X., et al. “A Representation Protocol for Traditional Crafts.” Heritage, 5(2), 2022, 716–741.