How it
is built.
Small, explicit pieces, each with one job and a test that holds it to that job. The model is never the only thing between a customer and a correct sheet.
The map
ontology/bespoke.json- One source of truth: 21 tags, forms in English, Hinglish and Devanagari, Hindi terms, ease and depth tables, slot costs
engine/- Python: lexicon, data generation, the training ladder, ONNX export
web/lib/engine/- The TypeScript mirror of the engine that runs in the browser
web/components/- The studio (Talk, Board, Sheet, Shop), the market, the site
web/api/- Vercel functions: assistant.js (Ask), understand.js (reads a message), next.js (writes the next question), transcribe.js (speech to text), vision.js (reads board pictures)
web/lib/assistant/- BM25 retrieval over the guides, and the guardrails
web/supabase/- Schema for accounts, requests, messages, boards and designs, all behind row-level security
mcp/- The MCP server
site/- These docs
From words to a specification
- Tokenize and tag. One regex tokenizer and one ontology-driven gazetteer, identical in Python and TypeScript and held together by 352 parity fixtures. A fine-tuned MiniLM tags what the gazetteer misses; decoding is Viterbi with BIO constraints, so
I-FITcan never followO. - Bind. Clauses are cut at punctuation and conjunctions, including
aurandlekin. Values attach to the nearest part cue (waist,neeche se), intensity (thoda,bahut) and negation (no,nahi,mat) to the nearest value. Numbers that describe the person, such as an age, are never read as lengths. Typos are grounded by character-trigram similarity. - Track state. Each message yields updates (
set,add,exclude,compare); the design is a fold over them, so it is versioned and replayable. Words beat the board; answers and edits beat both. - Ask one question. Every open slot is scored by the cost of a wrong guess × uncertainty × how much it matters for this garment family and occasion; who it is for and the age range lead while unknown, and neckline depth waits for the shape. Only the top question is asked, never one already answered or skipped. A language model may phrase it in the customer's own register, but only for a slot and options the engine allows.
- Read the board. The canvas becomes a graph of numbered pictures, circles and notes. CLIP scores each picture and each circled crop zero-shot; evidence is pooled per slot with salience and signed polarity.
deeper than #1 but less than #2becomes an interval in inches. - Write it out. The tailor sheet in English or Hindi with finished measurements (body plus ease), and plain search links for ideas, shopping, fabric and tailors.
The tagging ladder
Every system is trained on the same 12,000 generated requests and scored on the same held-out, hand-written ones. Strict span F1 (seqeval) with 95% bootstrap intervals over requests.
| Rung | System | F1 | 95% CI |
|---|---|---|---|
| L0 | Gazetteer, longest match | 0.936 | 0.898 to 0.967 |
| L2a | HMM, Viterbi | 0.751 | 0.693 to 0.809 |
| L2b | CRF | 0.832 | 0.789 to 0.876 |
| L2b | CRF + gazetteer feature | 0.900 | 0.853 to 0.942 |
| L3 | MiniLM-L12 (33M) + BIO-constrained Viterbi | 0.812 | 0.754 to 0.866 |
| L3 | MiniLM + gazetteer (deployed) | 0.953 | 0.918 to 0.982 |
What we learned
The first transformer run scored 0.715, below the gazetteer. The errors showed positional bias: the templates had taught it that the second word of {pattern} {fabric} is a fabric. Fixing the data, not the model, took it to 0.884: attribute bags in random order, distractor words that must stay outside, and three gold-label mistakes corrected.
On its own the transformer does not beat a good gazetteer on words the gazetteer knows. Its value is recall on what it has never seen: georget, organzaa, anarkalli, new spellings, unlisted words in context. Gazetteer first with the transformer filling gaps is the best system, with recall of 0.967.
The test set is 52 requests written separately from the templates. It is a seed set and it is small; the intervals say how small. The planned benchmark is 1,200 requests, 40% Hinglish, two annotators.
The text analytics underneath
The pipeline is a tour of the field, each technique used where it earns its place:
- Tokenization and normalisation across scripts, with transliterated Hinglish and Devanagari mapped to one ontology.
- Dictionary-based named entity recognition with longest-match gazetteers.
- Sequence labelling: HMMs with Viterbi, CRFs with hand-built and gazetteer features, and a fine-tuned transformer with constrained decoding.
- Weak supervision and data generation: templates, attribute bags and distractors, plus catalogue weak labels.
- Fuzzy matching with character trigrams for spelling variants and typos.
- Dependency-style binding of values to parts, intensity and negation within clauses.
- Dialogue state tracking as a fold over typed updates, with a cost-aware question policy.
- Zero-shot vision-language classification with CLIP for the moodboard.
- Information retrieval: BM25 over the guides, with a calibrated grounding threshold.
- Retrieval-augmented generation with guardrails: input screening, redaction, fenced context and output checks.
- Speech to text: the browser's recogniser, or Whisper on the device.
Reading pictures
The moodboard is read by CLIP (ViT-B/32), the open vision-language model from OpenAI, released under the MIT licence. Bespoke runs Xenova's int8 ONNX conversion of it inside the browser through Transformers.js (Apache 2.0), as a zero-shot classifier: the model scores a picture against sentences such as a garment with three quarter sleeves and says which fits best. Nothing is trained on the customer's pictures, and the pictures never leave the device.
CLIP itself is used as published. What Bespoke adds is everything around it, which is what turns a general image model into a reader of a tailoring board:
- The board as a graph. The Excalidraw scene becomes pictures numbered in reading order, circles that sit inside a picture, and notes bound to a picture by an arrow or by being next to it.
not this colouris a negative note. - Prompts written for tailoring. Each slot has its own set of plain sentences: colours, prints, garment types, sleeve lengths, silhouettes, embellishment, and four neckline depths tied to inches (4.5, 6, 7.5 and 9).
- Whole pictures for what they show well, circles for detail. From a whole picture only garment type, colour and print are read. Sleeves and necklines come only from a part the customer circles, cropped and scored on its own.
- Salience and polarity. A picture someone wrote about or circled counts fully; an untouched one counts half. A negative note flips the sign, so a strong reading becomes an exclusion.
- Pooling with a bar to clear. Readings across pictures are combined with weighted, signed log-linear pooling. A value is used only if it reaches 0.55 and leads the next one by 0.2, and picture evidence is capped at 0.6 so a customer's words always win.
- Comparisons into inches.
deeper than #1 but less than #2becomes an interval from the depths read off those pictures, with the proposal toward the shallow end, or a question if the pictures contradict each other.
Next on the plan: FashionSigLIP attribute heads in place of zero-shot CLIP, trained on a fashion catalogue, for finer details from whole pictures.
Big data analytics
The text analytics runs on every request in the browser. The big data side is the layer underneath it: building the data the models learn from, holding it to one schema, and learning from many people at once what one conversation cannot.
Built now
- A generated training corpus of 12,000 labelled requests from templates, attribute bags in random order and distractor words, with a separate synthetic set of 1,000 for tuning and 82 hand-written requests held out for scoring.
- Weak labels from a retail catalogue. The data step reads the 44,000-row Myntra product catalogue (Kaggle's Fashion Product Images) and turns its product names and attributes into labelled requests when the file is present.
- One schema for everything. The ontology is the contract every component reads: 21 tags, every surface form, Hindi terms, ease and depth tables and the cost of each question.
- Evaluation that says how sure it is. Strict span F1 with 95% bootstrap intervals over requests, so a small test set is never mistaken for a large one.
- An event log by design. Every design is a fold over typed updates, already the shape of an event stream, and requests, messages, boards and designs are stored in Postgres behind row-level security.
Next, in the plan
Designs, answers and orders are written as events to a lake of Parquet files partitioned by day, next to the public H&M and Amazon fashion datasets. Spark DataFrame jobs clean and join them, and their execution plans are kept as evidence of how the work splits across executors. MLlib pipelines then learn what one customer cannot teach: TF-IDF and Word2Vec over request text to find new words for the ontology, FP-Growth over finished designs to find attributes that go together, and ALS over customers and attributes to suggest fabrics and colours.
The exit test for that phase is that its suggestions beat a popularity baseline on held-out orders. Until it passes, these docs describe it as planned, not built.
Run it
git clone https://github.com/riteshroshann/onestyle
cd onestyle/web
bun install
bun run dev # http://localhost:3000
bun run build # static export into out/
bun tests/assistant.test.ts # Ask: retrieval, guardrails, handler
Without keys the studio runs fully on the device and the marketplace keeps its data in the browser. To connect the services, set these in web/.env.local or on Vercel:
NEXT_PUBLIC_SUPABASE_URL=... # accounts and data
NEXT_PUBLIC_SUPABASE_ANON_KEY=... # public by design
LLM_API_KEY=... # server only, for Ask
Apply web/supabase/schema.sql, then boards.sql and designs.sql. To retrain the tagger, see engine/ and the ladder commands in the README.
How we build
Words first, pictures suggest. The device answer always stands. Every value from a model is checked against the ontology. Every number in these docs can be reproduced from the repository. Nothing about a person is logged.