Half of a training set is usually the same photo four times.
Drop a folder of images. Loraprep finds the burst frames that are near-identical, picks
the sharpest one from each group, and hands back a training set laid out for
kohya_ss and Hugging Face — plus a report naming every
image it set aside and why. Nothing is uploaded and nothing on your disk is touched.
1 Choose your images
Drag a folder of photos here
…or a .zip of them. JPEG, PNG, WebP, GIF, BMP and AVIF. Nothing is
uploaded.
2 Before it runs
Step one always runs: a
of every image, which finds re-saved, re-compressed and resized copies. No download,
no model.
3 Working
Keep this tab in front. A backgrounded tab is throttled, and a phone may discard the page
entirely.
4 What it found
Nothing has been deleted and nothing will be. Loraprep only decides what
goes into the export; every original is untouched on your disk, and every image you gave
it appears in dedup-report.csv with a reason. Click any image below to keep
it instead.
Same picture, saved twice
Matched on the
,
which is the only signal here allowed to set an image aside on its own. It catches
re-saves, re-compressions and resizes. It cannot see a mirror flip or a rotation.
Looks alike — for you to decide
Nothing below is set aside automatically, and it never will be.
These are the pairs the vision model thinks look most alike, strongest first. It is a
way to go through a folder quickly, not a duplicate detector — measured on the standard
LoRA reference sets, two genuinely different photographs of one subject reach 0.99
similarity while a real duplicate that has been mirrored or turned greyscale scores
0.86. The two overlap, so no cutoff can separate them.
.
Click an image to set it aside; click it again to bring it back.
5 Your training set
Look at the groups before you train. A near-duplicate detector working
on brightness patterns and a small vision model will sometimes group two photographs that
you would not have. It cannot see a mirror flip or a 90° rotation at all.
.
1 · Nothing is uploaded, and the page header says so
Your images are opened with the browser's own file API and read inside this tab. There is no
endpoint in this application that receives them. The Content-Security-Policy in the page
header names every origin this page is allowed to talk to — open DevTools and read it.
2 · Every image is reduced to a 256×256 grid, once
Each photograph is decoded straight down to a fixed square working grid — a 12-megapixel
photo never exists as a 48 MB bitmap — and every measurement comes off that one grid. The
reduction from there is done in our own code rather than by the browser's resizer, because
the browser's choice of filter genuinely changes the answer: the same image at
resizeQuality: 'low' versus 'high' produces fingerprints 21 bits
apart on hard-edged content. Doing it ourselves means Chrome, Safari and Firefox agree.
3 · One signal decides, the other only suggests
A asks
whether the pixels are arranged the same way. It catches re-encodes, resizes and the
sub-pixel jitter of a burst, it costs nothing, and it is the only thing here allowed
to set an image aside on its own — at 5 bits it produced zero false matches across all
12,403 pairs of a 158-image library of genuinely distinct photographs.
An asks
what is in the picture, and it is deliberately given no power to remove anything.
The obvious design — threshold the similarity, cluster it, keep one per cluster — was
measured and does not work: on those same reference sets, different photographs of one
subject score up to 0.99 while a genuine duplicate that has been mirrored scores 0.88. The
distributions overlap the wrong way round, so no cutoff exists. Clustering made it worse,
collapsing six distinct photographs of one bowl into a single survivor. So the model ranks
pairs and a person decides.
4 · The model checks itself before it is trusted
The vision model runs on your machine, and a quantised model can be broken in a way that is
completely silent — right shape, plausible numbers, every image similar to every other one.
That is not hypothetical: the smallest published build of this exact model scores 0.09
against its own full-precision output. So at startup Loraprep embeds a fixed synthetic image
and compares the result against a reference committed to its source. Below 0.985 the backend
is refused, and the log says so.
5 · The keeper is chosen on two signals, not one
Sharpness alone gets this wrong, and it is worth knowing why.
measures high-frequency energy — and JPEG blocking is high-frequency energy, so
across three copies of one photograph saved at quality 92, 82 and 76 the measure rises as
the file gets worse. A sharpest-wins rule keeps the most damaged copy every time.
So each signal is used where it is trustworthy. Sharpness decides what is genuinely
in focus, which it settles convincingly — a smeared or downscaled frame scores a
tenth of its sharp sibling. Among the frames that are in focus, the one with the most bytes
per pixel wins, because the least-compressed file is the better training image. Two frames
off the same camera at the same quality land in the same bracket and are decided on
sharpness, which is what you want for a real burst. Click any other image to override it.
6 · You leave with a training set, not a verdict
A ZIP laid out for kohya_ss (the numbered concept folder, one
.txt caption per image, and a dataset.toml for the config path),
metadata.jsonl for Hugging Face imagefolder and diffusers,
dedup-report.csv naming every image you supplied and what happened to it, and
optionally the raw vectors so you can re-cluster later without doing this again.
Protected
Every image is decoded, measured, embedded and packaged inside this browser tab. There is
no endpoint in this application that receives a photograph.
This matters more here than in most tools. A LoRA set is faces, a client's unreleased
photography, a product that has not shipped, or someone's child. The alternative on offer
is renting a cloud GPU and uploading all of it to run a Python script.
The vision model runs on your machine. There is no inference API, no account and no
per-image cost.
No cookies, no fingerprinting, no third-party fonts, no telemetry about your images.
One thing persists between visits and it is named here: loraprep.theme. Never
your images, never your file paths.
Not protected
The model is downloaded from Hugging Face the first time you use it — one
anonymous GET of a public file, about 22 MB, pinned to a fixed revision and logged in the
event drawer as it happens. Hugging Face sees your IP address and that you asked for that
file, exactly as it would if you clicked the link yourself. It does not see anything about
your images, because nothing about your images is in that request. After the first run it
is cached and the tool works offline.
The exported ZIP contains your original image bytes unmodified, including
whatever EXIF they carry — GPS coordinates, camera serial numbers, timestamps. Loraprep
does not strip metadata and does not pretend to; if you are publishing a dataset, run it
through a metadata scrubber afterwards.
dedup-report.csv and metadata.jsonl contain your original file
paths, which frequently name people, clients and projects.
GitHub Pages logs the initial page load, as any host would.
Trust surface
The static bundle served by GitHub Pages, and the TLS chain to it.
The model file on Hugging Face, pinned to an immutable commit rather than a branch — so the
weights cannot be swapped after the fact — and checked at startup against a reference
embedding committed to this repository.
The ONNX Runtime WebAssembly binaries are served from
this origin, not from a CDN. The library's default is to fetch them from
cdn.jsdelivr.net; Loraprep overrides that, which is why the policy in the page
header does not name a CDN and why the tool still works with the network off.
A Cloudflare Web Analytics beacon records anonymous page views — no cookies, no
fingerprinting, no cross-site tracking; your files and data are never sent to it.
Feedback you choose to send is sent to feedback.benrichardson.dev. Nothing is
sent unless you open the feedback form and press Send; your files and data never are.
frame-ancestors is ignored in a <meta> policy and GitHub
Pages cannot set headers, so Loraprep makes no anti-framing claim.
What the fingerprint cannot see
These are measured, on a real 2000×1500 photograph, as
from
the original. An unrelated photograph sits at about 31.
A mirror flip is 30 bits. A 90° rotation is 31. Both are statistically
indistinguishable from a completely unrelated image, so a flipped copy of a photo will
never be detected as a duplicate. This is not a tuning problem; it is what the method is.
Crops go quickly. 2% off each edge is 3 bits and is caught. 5% is 14 bits
and is not, at any sane setting. 10% is 26.
Re-encodes and resizes are free: a JPEG at quality 40 is 1 bit, a downscale to a quarter
size is 1 bit, a 6-pixel blur is 1 bit.
Above about 14 bits, chance collisions start to dominate: at 2,000 images the expected
number of unrelated pairs that collide is 0.02 at 10 bits, 7 at 14, and 77 at 16. The
slider warns you when you cross into that.
What the sharpness number is not
It is not a blur verdict. The
scales with resolution and with how much fine detail a scene happens to contain, so a sharp
photograph of a clear sky scores below a badly focused photograph of a brick wall. Any
fixed "below N is blurry" threshold is wrong on somebody's dataset.
It also rises as JPEG quality falls, because blocking artefacts are
high-frequency energy and the measure cannot tell them from detail. On the bundled sample
the same photograph at quality 92, 82 and 76 scores 40, 53 and 73. A pre-filter before the
Laplacian was tried and does not fix it — blocking is structural, not noise.
So Loraprep uses it only to decide what is in focus, relative to the best frame in
the same group, and then picks the least-compressed of those. That is why the sharpness
shown under a thumbnail is not the whole story of why a frame won.
The single case where sharpness is used absolutely — a frame with no group at all and
essentially no edge energy anywhere — is set very low deliberately, and catches a smeared
pan or a lens cap rather than a merely soft shot.
Why the vision model is not allowed to delete anything
This is the most important design decision in the tool, and it was made by measurement.
Using MobileCLIP-S0 on google/dreambooth — the canonical LoRA subject set, 158
hand-curated photographs containing no duplicates at all:
Different photographs of the same subject, which must be kept: median similarity
0.80, 99th percentile 0.97, maximum 0.9916.
Genuine duplicates made from those photographs, worst case per transform: 2× upscale 0.98,
re-encode 0.96, 10% crop 0.93, half-size 0.92, mirror flip 0.88, 5°
rotation 0.86, greyscale 0.86, watermark 0.73.
The two overlap almost entirely, and in the wrong direction. At a 0.90 cutoff it misses
flips, rotations, greyscale, crops and watermarks and falsely condemns 49 of 342
same-subject pairs. At 0.98 it catches one of thirteen duplicate families — the one the
fingerprint already finds at a distance of 1 bit.
Clustering those scores is worse still. At 0.90, single-linkage collapsed six visually
distinct photographs of one bowl into a single survivor, through a chain containing a pair
scoring 0.809 — nine hundredths below the threshold that formed the cluster. 35 of
158 hand-curated images would have gone.
So the model ranks pairs for a human to look at, and never sets anything aside itself. If you
have seen a tool that clusters CLIP embeddings and deletes the extras, these are the numbers
it is not showing you.
How the duplicate groups can still be wrong
Grouping on the fingerprint is
:
if A matches B and B matches C, all three end up together even if A and C do not match. The
risk is far smaller here than on similarity scores because the threshold is tight, but every
group is shown before anything is exported.
Two genuinely different photographs of the same flat sky can collide on the fingerprint.
Low-texture images are where this method is weakest.
Machine limits
GitHub Pages cannot make this page cross-origin isolated, which means no
SharedArrayBuffer, which means ONNX Runtime runs the model on exactly one thread. Measured
on an M3 Pro at 256×256, that is about 7 images a second. Where
is available
and passes the self-check it is used instead and is considerably faster.
The stated ceiling is 2,000 images. Not a hard limit — the tool will let
you go past it — but decoding thousands of full-resolution photographs is the likeliest way
to have a phone discard the tab, and it is the point past which the honest advice is to
work in batches.
HEIC is not readable by any browser. Not Safari, despite the platform
having a decoder. HEIC files are counted and named rather than silently ignored; convert
them first.
The archive cannot exceed about 3.9 GB or 65,000 files. The ZIP writer
does not implement ZIP64, and past those points it would write a silently corrupt archive
rather than fail — so Loraprep checks first and refuses with an explanation.
Firefox and Safari have no save-file picker, so on those browsers the archive is built in
memory before it downloads. Chrome documents its blob ceiling as 2 GB on desktop but
roughly a hundredth of physical memory on Android.
A backgrounded tab is a paused run, and there is no resume — the run restarts.
The sentence that matters most
Loraprep never deletes anything. It decides what goes into an export. Your
originals stay exactly where they are, and every image you gave it is accounted for in
dedup-report.csv with the reason and the file it was judged against.
Loraprep turns a folder of photographs into a training set: near-identical
frames grouped, the sharpest kept, and the result laid out for the trainers people actually
use. It runs entirely in your browser.
It exists because the alternative is renting a cloud GPU to run a Python script you did not
write, on photographs you would rather not upload — and because the dull, mechanical part of
preparing a LoRA set is most of the work and none of the fun.
The vision model is
MobileCLIP-S0
(Apple ML Research), converted to ONNX by Xenova and run through
transformers.js. ZIP reading and writing use
fflate.
Everything else — the perceptual hash, the sharpness measure, the grouping, the header
parser, the dataset layout and the report writer — is first-party and unit-tested.
