How artificial intelligence works when following a teacher’s writing on a whiteboard begins with a common classroom scene. The tutor writes an equation, steps in front of it, adds another line, then erases the first example to make space for the next one. A lesson video contains many frames, but a learner may want a tidy record of what was written and when each part appeared or disappeared.
Did you know that a whiteboard record is more difficult than extracting a single photograph? A person’s body can temporarily hide writing without erasing it. A change in camera angle can move all the strokes without changing the board. Erasing is a lasting change to written content, whereas blocking the camera’s view is not. AI-assisted whiteboard analysis must separate these events to avoid losing teaching steps or inventing words.
The practical route is video frames → identify board area → align the board through time → segment written strokes → distinguish temporary occlusion from persistent writing changes → track newly written and erased regions → create a time-stamped index → verify against the original footage. Such a system is an aid for study and retrieval, not an exact transcript of everything the teacher said or meant.
Quick teaching route
- Locate and align the board
- Find writing, not the moving teacher
- Track stable strokes through time
- Distinguish occlusion from erasure
- Preserve snapshots and source times
1. Why a video is not automatically a set of useful notes
A recorded lesson may contain an hour of frames, many of which show almost the same whiteboard. Merely saving every frame creates hundreds or thousands of duplicates. Choosing one frame at the end can be worse: the tutor may have erased the earlier explanation entirely.
Useful notes require a timeline of changes. The system needs to know when a distinct equation appeared, when it remained stable and when it was deliberately removed. This is a sequence-analysis problem rather than ordinary single-image OCR.
2. What the software can actually observe
A camera records pixels, not a live log of which pen strokes the teacher created. Between frames, dark and coloured markings may appear, shift or disappear. Their appearance can change because of writing, erasing, occlusion, lighting or camera movement.
An honest model uses language such as ‘new board content detected’ or ‘content appears to have been erased’. It cannot always infer why the teacher removed it or whether the removal marked the start of a new topic.
3. Locate the board before processing the writing
A video can show the instructor, students, projector screen, wall and whiteboard together. The detector first identifies the region of the actual writing surface. The board may be rectangular, partly blocked or seen at a perspective angle.
Classical geometry can help locate straight boundaries, while a learned segmentation model can handle varied backgrounds and board types. If the wrong rectangle is chosen, a projected slide or noticeboard may be mistaken for the handwritten lesson.
4. Align the board across neighbouring frames
Even a camera held nearly still can shift slightly. If the whiteboard slides five pixels to the left between frames, a direct subtraction may falsely report that every written letter changed. Registration aims to map matching board locations into common coordinates.
The system can use stable border corners or background features to estimate a transformation. It must account for deliberate camera zooms and parallax. Alignment should be checked before claiming that a mathematical symbol was added or erased.
5. Segment writing rather than whole-frame differences
A handwritten black line occupies a small fraction of a whiteboard image. Background objects, the teacher’s clothes and light reflections may dominate frame differences. A segmentation model attempts to identify pixels belonging to writing on the board.
FCN-LectureNet, published in IEEE Access in 2021, used a fully convolutional network to extract handwritten content from whiteboard and chalkboard lecture videos. It provided a concrete research method for creating binary representations of board writing.
6. Why whiteboard ink is different from a printed document
Handwritten equations change while the video runs. Lines may be incomplete for several seconds as the tutor draws them. Marker thickness and colour can vary; the teacher’s hand repeatedly covers the place being written.
A printed document extractor can often process one completed page. Whiteboard analysis must make decisions about incomplete strokes, continuity over time and when a unit of writing has become stable enough to store.
7. A moving teacher creates temporary occlusion
Imagine the teacher writes 3x + 4 = 19, then stands directly in front of the equation to explain it. The video temporarily shows the teacher’s jacket instead of the writing. A naive image-difference system may conclude that the equation vanished.
The correct interpretation is that visibility changed, not necessarily the content. A model can retain previously observed strokes and wait for later frames to see whether they reappear. This temporal memory is essential to distinguish occlusion from erasure.
8. How real erasing can be recognised
An erasure event removes a region of writing from the board. If the teacher wipes part of the surface and later frames consistently show that region empty, the evidence for deletion is stronger than a single obstructed frame.
The 2021 FCN-LectureNet research describes constructing a signal that approximates content deletion events from a spatial–temporal index of stable writing. Peaks can indicate useful boundaries between lecture segments, since teachers often erase material while changing topics.
9. Writing events are not always topic changes
A teacher can erase a mistake and immediately rewrite the same expression. That is an erasure, but not necessarily a new lesson section. Likewise, moving from one whiteboard panel to another may introduce new content without deleting anything.
Use deletion signals as retrieval cues, not as a complete semantic theory of the lecture. A useful system may combine video content with speech or teacher-provided lesson structure to decide whether a true conceptual transition occurred.
10. A complete fictional algebra lesson
Imagine a tutor writing ‘3x + 4 = 19’ and then subtracting four from both sides. The intermediate result appears below it. The tutor steps in front of the board, talks briefly and returns. Later, the original equation is erased to make room for a second example.
A whiteboard-aware model stores stable content snapshots before the erasure, associates them with source times and recognises that the tutor’s temporary obstruction did not remove the earlier equation. It can offer a chronological view of the working steps.
Now suppose the tutor corrects an accidental sign error while writing. The model should preserve the corrected sequence or flag a possible correction rather than synthesise a final solution that never appeared. The example is fictional.
| Observation | Possible cause | Evidence needed |
|---|---|---|
| Equation vanishes for two seconds | Teacher blocks camera | Later reappearance |
| Equation remains absent | Actual erasure | Aligned later frames |
| Whole board shifts | Camera movement | Global geometry |
| New strokes appear | Teacher writing | Temporal persistence |
| White patch covers content | Glare or exposure | Image-quality review |
11. OCR and mathematical meaning are separate
An OCR system may try to read the extracted strokes as words and mathematical symbols. But handwriting can be incomplete, and a small minus sign or exponent can change the result of an equation.
Even when text recognition is accurate, interpreting the meaning of a mathematical step is another task. A board index should preserve the visual source alongside any recognised text so that a tutor or student can check ambiguous characters.
12. How to group strokes into meaningful units
A tutor may write a heading, draw a graph and add three equations underneath it. A stroke-by-stroke log would be difficult to read. The system can group nearby stable writing into lines, expressions or regions before creating notes.
However, one arrow can connect distant parts of a diagram, and one bracket can cover several equations. Spatial proximity is not always enough to define a meaningful unit. Graphical relationships and learning context may need to be retained.
13. Why timestamps matter
An extracted board image is more useful when its source interval is known. A student looking for the chain-rule example should be able to return to the moment the tutor completed it or explained it.
Real video timestamps matter, particularly for variable-frame-rate recordings or edited files. Frame numbers converted using the wrong rate can point to the wrong part of a lecture. If the video is trimmed later, offsets need updating.
14. A keyframe is a selected snapshot, not the whole lesson
Extractive summarisation selects representative frames of stable writing rather than generating an entirely new diagram. A good snapshot can show a completed step that existed before the board was erased.
But a single final keyframe may omit the order of reasoning. A useful index can provide several successive snapshots and their times, making it clear which additions or deletions happened between them.
15. What the LectureMath benchmark contributes
The FCN-LectureNet researchers extended their AccessMath work to the LectureMath dataset for evaluating lecture-video summarisation. The project releases research methods and data for analysing board content and temporal changes.
A benchmark measures performance on particular lecture formats. Its results do not guarantee that every classroom, marker colour, camera position or handwriting style will work identically. Tests should use independent source videos from the actual environment.
16. Common false alarms
Camera shake can make writing move without changing its content. A hand, eraser or person’s body can block strokes. Reflections can temporarily wash out a faint line. Marker ink can appear lighter when the camera changes exposure.
A robust system checks persistence over several frames, compares aligned board coordinates and identifies when the writing becomes visible again. Sudden apparent absence alone is not sufficient evidence that the tutor erased something.
| Error | Consequence | Review |
|---|---|---|
| Occlusion called erasure | Earlier lesson step lost | Track return of writing |
| Camera shake treated as new ink | False content change | Align board |
| Incomplete expression stored | Misleading formula | Choose stable keyframe |
| Wrong video time | Link misses teaching moment | Use source timestamps |
| One erasure treated as topic change | False chapter | Inspect audio and content |
| OCR changes minus sign | Mathematics wrong | Verify original image |
17. How to evaluate an indexing system
Use reference annotations showing what writing existed at selected times and when genuine erasures occurred. Measure stroke extraction, stable-content grouping, deletion-event detection and keyframe selection separately.
Check actual student tasks too. Can the student find the equation they remember? Does the index preserve the sign and sequence of mathematical steps? A visually appealing summary fails if it drops a critical line or links to the wrong timestamp.
18. Classroom privacy and responsible recording
Lesson videos can contain identifiable students, voices, names on the board and private working. A board-analysis tool should operate on appropriately authorised recordings and avoid collecting more personal information than the learning purpose requires.
The source frames should remain accessible to permitted reviewers, while public summaries should avoid publishing private details. Technical ability to extract board writing does not establish permission to share everything captured by the camera.
19. A paper experiment to understand erasure
Draw four frames of a pretend whiteboard lesson. In the first, write an equation. In the second, draw a rectangle representing a tutor standing in front of it. In the third, reveal the equation again. In the fourth, erase it.
Ask which frame records a genuine deletion. A detector using only adjacent image differences might mistake the second frame for an erasure, while a temporal model can see the writing return. This demonstrates why memory over time matters.
20. The larger lesson: an absence can have different causes
Whiteboard-video analysis is a useful example of AI because a visible mark may disappear for several reasons: a person blocked it, the camera moved or it was actually erased. The computer must compare competing explanations rather than treat every missing stroke as the same event.
The best system combines alignment, writing segmentation, temporal tracking and careful keyframe selection. It helps students revisit real teaching steps without inventing missing equations or claiming that a likely deletion always signals a new topic.
21. Distinguishing a correction from a new line of working
A tutor may overwrite a mistaken coefficient on the same whiteboard line. The video can show part of the old expression disappearing and new strokes appearing. A naive summariser might keep only the final corrected version and lose the fact that the teacher explicitly demonstrated the correction.
Depending on the educational goal, it may be useful to preserve both stages. A searchable timeline can indicate ‘earlier writing’, ‘subsequent correction’ and ‘stable final version’ without claiming that the model understands why the teacher changed it.
Accuracy means tracing the actual marks and timestamps, not rewriting a polished solution that omits mistakes the class learned to recognise.
22. How short and long erasures differ
Erasing one symbol and clearing an entire board are different event patterns. A small correction can be local, whereas clearing a large region can indicate a substantial shift in the displayed writing. Both require alignment to the same board coordinates.
FCN-LectureNet’s content-deletion signal demonstrates a method of finding temporal boundaries from changes in stable handwritten regions. A high deletion score can be a useful index cue, but it is not independent proof of a new conceptual topic.
Combine visual evidence with source timestamps and, where available, authorised speech transcription. If the tutor says ‘Let me fix that sign’, the audio may help explain a local erasure without turning it into a new chapter.
23. Why board colour and marker styles matter
Whiteboards can contain black, blue, red and green ink, while chalkboards may carry white or coloured chalk. A model trained only on black strokes against white plastic may struggle when a dark board or projector glare changes the image distribution.
Test diverse examples and preserve different colour layers when they serve a teaching purpose. A tutor may use red for a correction and black for the main solution; collapsing everything into monochrome too early can remove that distinction.
However, colour is not universally tied to meaning. A model should record a red mark as red, not assume the tutor meant ‘wrong’ without a declared convention.
24. The learner-facing proof pack
For each extracted equation or diagram, show the source clip start, a representative keyframe, the later deletion or modification event and a link back to the full recording. That makes it possible to verify what actually appeared on the board.
A student should be able to compare a machine-read formula against the original pixels. If the minus sign is obscured by a hand, report the reading as uncertain rather than presenting a guessed equation with no warning.
The practical outcome is a navigable history of the lesson, not an authoritative replacement for the original teaching. The ability to check the source is what makes the summary useful.
Frequently asked questions
Can AI extract writing from a lecture whiteboard?
Research systems can segment handwritten board content and track it across video frames.
Why is it harder than OCR on one page?
Writing changes over time, while the tutor or camera can obscure or move its image.
How does the model distinguish an erasure from a person standing in front?
It can compare aligned frames over time and check whether the writing later reappears.
What is FCN-LectureNet?
A published approach to extracting and summarising handwritten content from whiteboard and chalkboard lecture videos.
What is LectureMath?
A research dataset associated with whiteboard lecture video summarisation.
Are erasures always new topics?
No. A tutor may erase an error within the same explanation.
Can AI produce exact Maths notes automatically?
It can propose an index, but uncertain handwriting and mathematical symbols need verification.
Why keep the timestamps?
They let learners return to the actual teaching moment rather than trust a detached summary.
Can camera shake cause false handwriting changes?
Yes. Frame registration helps align stationary board regions.
What should students review?
The original frames, equations, corrections and timing before using extracted notes.
Research and connected lessons
- FCN-LectureNet: Whiteboard and Chalkboard Lecture Summarisation (IEEE Access 2021)
- University at Buffalo: Research publication and methodology
- LectureMath: Research methods and datasets
- How AI Aligns Two Photographs Taken at Different Times
- How AI Detects Camera Shake Without Confusing It With Moving Objects
- How AI Aligns Lecture Slides With Spoken Explanations
- eduKateSG | How X Works Hub
Editorial note: The algebra lesson and filmed tutor are fictional teaching examples. Research on whiteboard video indexing does not guarantee perfect recovery of erased or hidden writing. Preserve original frames, verify important expressions and follow recording permissions.