athletic data protocol

the origin

it started with a chris heria video, just after eleven at night, the phone too bright for the dark room. i have trained seriously for over a decade, seriously enough that i keep progressive overload in a spreadsheet and can recite the difference between a romanian and a stiff-leg deadlift at half past five in the morning, even when nobody at that hour is asking. i saved heria's clip, weighted calisthenics, perfect for the next day, and fell asleep with the small satisfaction of having prepared something. in the morning i opened my training app to plan the week and hit the same wall i have hit a hundred times. the heria clip lived on youtube. my log lived in a spreadsheet. a mobility routine sat in an instagram folder i had named "flexibility stuff" six months earlier, in one of those fits of tidiness that never come back. a stretching sequence was buried in a yoga app. and a hiit finisher i had saved on tiktok was simply gone, swallowed by a feed i would never scroll back through. four apps, three browser tabs, one spreadsheet, all full of workouts i genuinely wanted to do, and not one of them aware the others existed.

and even if i pieced a plan together out of all of it, i still would not know where i actually stood. heria's video showed weighted muscle-ups with a twenty-kilo vest; i cannot do a single one, vest or no vest. so what is my path from here to there, which progressions have i already tried, where did i plateau, what worked last time? those answers are scattered across three apps, and good luck assembling them at the exact moment you need them. i have hundreds of bookmarked workouts across a dozen platforms, thousands of hours of content, all filed away with the best intentions and never opened again, and i cannot build one coherent training plan out of the whole pile.

the knowledge is there, it is just trapped

the frustration sent me down a rabbit hole, and what was at the bottom was worse than i expected. the raw material is everywhere: years of fitness content piling up on youtube, tiktok and instagram, and exercise databases that catalog more than a thousand movements in structured form. what does not exist is any standard way to connect a single piece of it to the next. every video sits alone, every training app sits alone, and the two ecosystems that were supposed to fix this, apple health and google fit, are not even interoperable with each other, let alone with the mountain of video outside their walls. a study in the journal of medical internet research put it plainly years ago: the lack of interoperability and the presence of data silos prevent users and health professionals from getting an integrated view of their fitness data. the infrastructure simply is not there, and a decade later it still is not.

stop waiting for the world to standardize

the traditional fix is to ask every platform and every creator to agree on a common format. that coordination problem has proven intractable, which is exactly why we still have no fitness-data interoperability after decades of trying. athletic data protocol inverts the bet. instead of waiting for the world to structure its data, i am building a system that pulls structure out of content as it already is: a video, a handful of screenshots, a caption, the audio of someone counting reps, whatever the source, it all lands in the same schema. not a standard forced on creators, but intelligence that adapts to what they already make.

reading an exercise off a video

an exercise is not just a name, it is a dense little information structure: a movement pattern like push, pull, hinge or squat, the primary muscles, the equipment, the loading, whether it is worked one side at a time. a youtube title says "chest day" while the footage actually shows particular movements done a particular way, so the system has to infer the whole structure from partial, implicit signals, often spread across modalities. that means fusing three streams that do not always agree: what the camera shows, what the voice says, and what the text claims. the genuinely hard part is that fusion, deciding who to believe when the streams complement, repeat or flatly contradict each other, as when a video titled "best bicep exercises" is in fact showing triceps. there the visual evidence has to overrule the label, and the system has to learn when to trust which signal.

the knowledge side is not solved either. researchers have built ontologies to formalize exercise, from PACO's couple hundred classes to the exercise-medicine ontology published at the end of 2024, and practical catalogs list well over a thousand movements, but a systematic review of twenty-eight of these ontologies gave them, on average, just 4.3 of 12 quality criteria. nobody has a complete, agreed-upon map. so part of the work is building on those frameworks while stretching them to cover the messy diversity of real fitness content.

the architecture

i started where almost everyone starts: pose estimation. the open tooling is impressive, mediapipe tracks thirty-three body landmarks in real time, and the research is encouraging, with workout-classification work feeding exactly those joint coordinates into a small neural net and clearing ninety percent. so i built a prototype to see whether i could read reliable joint angles off an ordinary phone video and let the geometry tell me what the movement was. it did not survive contact with reality. on a romanian deadlift filmed from the side, the one joint that mattered most, the knee, the exact angle that separates a romanian from a stiff-leg deadlift, was the one the model reliably got wrong: sometimes it snapped the skeleton onto the coach in the background, sometimes it lost the lower body to a loose trouser leg. it worked on the trunk and failed on the detail, which is roughly the opposite of useful. so i threw it out, and the failure pointed straight at the better design.

the better design is to hand the whole video to a model built to watch it. a natively multimodal model, gemini 2.5 flash, takes the clip with its audio and its on-screen text together and returns the exercises it sees as structured json, splitting a single piece of footage into its separate movements and sets. crucially it does this name-blind: it never sees the caption or the creator's title, only the footage, so it cannot be talked into agreeing with a wrong label. and it deliberately speaks in buckets, not numbers. there is no field anywhere for "knee at 137 degrees", because that is exactly the kind of number a model will invent with total confidence; what it can emit instead is a discriminator like the hip-to-knee relationship, the load position, the equipment, and each of those can stay empty when the footage does not show it. a value the system cannot see is one it cannot invent.

around that visual pass run the other two streams in parallel. whisper transcribes the audio, because reps counted out loud rarely show up on screen. the caption and any text contribute their own claims. then claude sonnet 4.6 does the reconciliation, taking the name-blind visual reading, the transcript and the caption and resolving them into one workout object through a tool call whose schema it has to satisfy, so the output is valid json because it must be, not because we hoped. cheaper, faster work drops down to claude haiku 4.5, which classifies a movement's muscles and pattern and settles the close calls. it is the familiar shape of putting each model where it is strongest, a vision model to watch, a careful one to reconcile, a fast one for volume, instead of pretending one model does everything well. one rule sits above all of it: never invent a number. every rep count carries its source, whether it was shown on screen, called out, written in the caption, or merely demonstrated, and when there is no honest source, the field stays empty.

the matching problem

extracting a clean "romanian deadlift, 3 sets of 8" is only half the job. the other half is recognizing that it is the same movement i did three weeks ago, that someone else calls "RDL", and that a third creator films as a "stiff-leg" when the footage clearly shows a soft knee and a clean hip hinge. without that you do not have a training history, you have a pile of disconnected entries, which is the exact problem i started with. so every extracted exercise runs down a ladder against a curated catalog built from established strength-training references. an exact name match resolves at once. failing that, a learned alias table that has been taught, from real corrections, that "RDL" means romanian deadlift. failing that, a name search, then a semantic embedding of the name, and then the part i am most attached to: a second embedding of the movement itself, so two clips find each other because they are performed the same way, not because they happen to share a word.

a name is a weak signal, so before any match goes through it has to clear a set of structural gates: the equipment, the muscles worked, the joint action and the kinetic chain, plus a family check that keeps an incline press from dissolving into a flat one, or a romanian deadlift into a conventional pull. in the gray zone, where two candidates sit within a hair of each other, the system does not guess; it asks claude for a verdict, same, variant or different, and when it still is not sure it abstains, flags the entry for a human and lays out the alternatives instead of digging into a confidently wrong answer. the governing instinct is "better to create a new exercise than to match the wrong one", because a false merge corrupts your history quietly, in a way that is very hard to notice later. every one of these decisions is stored with its confidence, which is what will eventually let the thresholds be tuned against real outcomes instead of my gut. and because the same movement can be loaded one side at a time, the model tracks the side down to the set and keeps a separate log and a separate personal best for left and right, which most apps quietly average away.

where it stands

built in public, half finished. the import pipeline runs today across instagram, tiktok, youtube and plain screenshots, and the heavy work happens asynchronously, taking around twenty-five seconds for a fresh clip; a viral reel someone imported before me comes back almost instantly, because the expensive extraction is cached and only the per-user matching runs again. the matching ladder, the structural gates, the per-side tracking and the confidence telemetry are all real and in the code.

what i will not do is put a headline accuracy number out there, because i do not have one i would vouch for with the word "research". there is no honest single figure without a labeled evaluation set held back from the system, and building that set, a few dozen ground-truthed videos i can measure the pipeline against, is the next real piece of work, not a thing already done. any percentage i printed before that would be theater. what i can say is that it is good enough that i use it on my own training, and that the errors show up where you would expect, in poor video quality and at the edges of the schema, not in the model misreading a clean lift.

ain

athletic data protocol is the engine; ain is the app that runs on it. the idea is simple. import a workout from anywhere, a youtube video, an instagram reel, a tiktok, a few screenshots, and watch it become a structured entry in one personal library that is searchable, comparable and tied to your own history, so your follow-along sessions and your gym log finally sit in the same place and count toward the same record. today ain does the import, the catalog and search, and tracks a session live with per-side personal bests; the bigger pieces, offline-first sync, a proper workout editor and a coaching layer over the history, are still ahead of me. early access is coming.

the limits

i would rather name the hard parts myself than let them surface later. without a formal evaluation set there is no validated accuracy figure yet, and that is a real gap, not a rhetorical one; the system is explicitly not clinically validated and has no business near injury rehab without a professional in the loop. it assumes one person in frame, it struggles in bad light and at awkward angles, and it thinks in english, so group classes, partner work and the rich german and spanish fitness vocabularies are all open ground. there are ethical edges too. reading structured data off social platforms is a genuine legal grey zone, so the system works only with content someone has already made public, and it takes the facts of a workout, the sets and the reps, not the creative expression around them. and a training history is sensitive in its own right, because when and how you train says more about your life than it looks like.

the point

fitness-data fragmentation is not a technical limit, it is an architecture failure. we already have the videos, the exercise databases and the models that can watch a clip and read it; they just do not talk to each other. athletic data protocol is the layer that gets them talking, not by waiting for the world to standardize but by reading structure out of content as it is, and abstaining where it cannot. what lies ahead is honest research, not a finished claim: a real evaluation set, a fusion that handles contradiction better, more languages, more than one body in frame. for anyone who has ever tried to assemble a real training plan out of bookmarks scattered across five apps, this should have existed ten years ago. i am building it now.

references

1. Gay, V., & Leijdekkers, P. (2015). Bringing Health and Fitness Data Together for Connected Health Care: Mobile Apps as Enablers of Interoperability. Journal of Medical Internet Research, 17(11):e260.
2. ExerciseDB. (2025). ExerciseDB API Documentation.
3. Jin, Q. et al. (2019). Developing a Physical Activity Ontology to Support the Interoperability of Physical Activity Data. Journal of Medical Internet Research, 21(4):e12776.
4. Liu, X. et al. (2024). Core reference ontology for individualized exercise prescription. Scientific Data, 11, 1349.
5. Braun, M. et al. (2023). Content and quality of physical activity ontologies: a systematic review. International Journal of Behavioral Nutrition and Physical Activity, 20(1):28.
6. Google. (2024). MediaPipe Pose Landmarker.
7. Bang, G.-S., & Park, S.-B. (2024). Workout Classification Using a Convolutional Neural Network in Ensemble Learning. Sensors, 24(10):3133.
8. Google. (2025). Video understanding — Gemini API.
9. OpenAI. (2022). Introducing Whisper.
10. Tang, Y. et al. (2025). Video Understanding with Large Language Models: A Survey. IEEE Transactions on Circuits and Systems for Video Technology.
11. Anthropic. (2026). Structured outputs — Claude Docs.

last updated: Jun 2026