# SFT Desk

> Check one fine-tuning dataset before the training run, then write its dataset card. Paste or drop the JSONL
> (or one JSON array) for an SFT, DPO, ORPO, KTO or GRPO run - and the eval split if you have it. The browser
> reads every row and runs 19 deterministic checks for free; a paid audit judges what a redacted sample of rows
> teaches and names the rows to drop; a paid card lane writes the six fields the run is gated on.

URL: https://sft-desk.skillsafe.ai/
API tutorial: https://sft-desk.skillsafe.ai/api.html
Model: gpt-terra (SkillSafe app runtime). The free check runs in the browser and uploads nothing.

## What the free check does

- Reads JSONL (one JSON object per line) or one JSON array, up to 50,000 rows and 12 million characters. Row
  numbers are line numbers (array positions for an array).
- Works out each row's shape: ChatML `messages`, ShareGPT `conversations` (from/value), instruct
  (`instruction`/`output`), `prompt`/`completion`, preference pairs (`prompt`/`chosen`/`rejected`), KTO
  (`prompt`/`completion`/`label`), prompt-only (`prompt` plus answer or verifier), flat `text`.
- Holds the rows to the method you pick, or the one the main shape implies: SFT takes messages, instruct or
  prompt/completion rows; DPO and ORPO take preference pairs; KTO takes boolean-labelled rows; GRPO takes
  prompt-only rows.
- Catalogue: DC-01 line not valid JSON; DC-02 shape does not match the method, or mixed shapes; DC-03 no
  assistant turn or an unknown role; DC-04 turn order (system not first, same role twice, ends on a user turn);
  DC-05 empty turn or response; DC-06 flat text or chat-template markers in the content (defeats assistant-only
  loss); DC-07 ShareGPT rows not converted; DC-08 exact duplicates (after whitespace and case); DC-09
  near-duplicates (same words ignoring case, punctuation and numbers); DC-10 one prompt with different answers;
  DC-11 train prompts also in the eval split; DC-12 rows estimated longer than max_length; DC-13 under the
  ~1,000-row SFT floor; DC-14 real data under 25% (from a source/origin/synthetic field); DC-15 refusals and
  boilerplate; DC-16 identical chosen and rejected, KTO labels that are not booleans or only one label, GRPO
  prompts with no answer; DC-17 email addresses, phone numbers, credentials; DC-18 canned answers and very short
  responses; DC-19 system prompts that vary; DC-20 context.
- Lengths are estimates at about four characters per token (1.5 for non-Latin scripts), plus template overhead;
  the model's tokenizer is the ground truth. With max_length it also estimates packed sequences and padding
  waste.
- Clean export: the rows as JSONL with non-JSON lines and exact duplicates left out (first copy kept) and
  ShareGPT rows converted to role/content; it never rewrites a row's text. A drop log lists every row left out.

## The paid lanes (input field `task`)

A run is sent `rows` - a sample of up to 40 rows (the rows the flags point at first, then rows spread across
the file), each line `#<row> <json>`, with email addresses, phone numbers and credential values replaced by
`[email]`, `[phone]`, `[credential]` and string values over 1,200 characters cut in the middle - and `facts`, a
JSON string with the page's counts, flags, the sample's row numbers and the dataset-card facts. Optional:
`note` (where the rows came from, the template at inference, the loss masking, the packing inspection) and, for
the card lane, `audit_handoff`.

- `audit`: one JSON object with `verdict` (blocked, needs_work, ready), `headline`, `dataset`, four `areas`
  (format, template, content, mix: sound, fragile, broken), `findings` (id, flags, rows, severity, area, title,
  evidence copied from a sampled row, why, fix, drop), `flag_answers` (every flag confirmed or dismissed once),
  `plan`, `questions`, `summary`.
- `card`: one JSON object with `verdict` (complete, incomplete), `headline`, six `fields` in order (provenance,
  counts, synthetic_ratio, dedup, template, packing - each complete, partial or missing, with value, source and
  what is still needed), six `checklist` items (format, template_first, loss_mask, packing_inspected,
  real_floor, card_complete - pass, fail or unknown), `card_markdown`, `questions`, `summary`.

The page checks every reply: audit findings must name rows it sent or flagged and quote evidence that is in
them, drops only rows they name, and every flag answered once; the card's counts and real share must be the
page's numbers, no checklist item may pass against a flag the page raised, and the verdict must follow. Neither
lane may write an email address.

## Sources

- Derived from the agent skill dataset-curation by Seth Hobson - https://github.com/wshobson/agents
  (plugins/llm-finetuning/skills/dataset-curation, MIT) - https://skillsafe.ai/skill/@wshobson/dataset-curation
- The skill's rules restated: format by method; apply the chat template before concatenation or packing; train on
  assistant turns only and keep conversations as messages; decode 5-10 packed sequences before a full run; at
  least 25% real data, with general-domain replay rows counting as real; exact then semantic dedup (about 0.92
  cosine); a dataset card with provenance, counts, synthetic/real ratio, dedup method, template and packing
  config.