spotlit: a 6.7 MB two-stage element ranker for browser agents

Most browser agents send the whole page to their language model at every step. A page has hundreds to thousands of elements, and the next action usually needs one of them. On one recorded step on Viator, the page has 1,373 elements, about 75,131 tokens of text, and the goal needs exactly one: the Coffee & Tea link.

spotlit ranks a page's elements against the agent's goal and hands the model the 50 that matter. It is 6.7 MB of open weights, runs in the browser on WebAssembly with no server and no GPU, and takes about a quarter of a second per page. This is how it got there, including what did not work.

One page step

1. 1,373Pageevery element, about 75,131 tokens as cleaned HTML
2. 400Stage 1hashed byte n-grams, 2.0 MB, about 40 ms
3. 50Stage 2BERT-tiny cross-encoder, 4.7 MB, 185 ms for 400 pairs
4. 1Agent's modelreads 50 lines, about 507 tokens, and picks the action

Replay

Where it started

It began in late July, when I came across Moka, a tiny Go-playing network that runs in the browser. Seeing how small a useful model can get made me look for other things a model that size could do, so I tried my own. My first was a text embedder with no tokenizer: byte 3, 4 and 5-grams hashed into a 6,144 x 320 table, 1.99 MB at INT8, fitted to imitate bge-small-en-v1.5 [4]. The goal was to add it to keyword search (BM25) and beat BM25 alone by at least +0.03 nDCG@10.

Corpus	BM25	BM25 + student	BM25 + teacher
Rust book, 458 queries	0.589	0.613	0.687
Private project docs, dev, 597 queries	0.265	0.262	0.321

On technical prose it added +0.024; on dense design documents it made search worse. I tried gradient-trained students, more and different training text, two or three hash functions per n-gram, a table twice the size, a 100K-word static vocabulary and a small transformer. None closed the gap, and a single change of hash function moved the score by 0.015, so most of those differences were noise.The conclusion I wrote down then: without pretraining, a model this small learns too little language from half a million labelled texts to beat BM25 on varied documentation. That lesson came back later.

A new job for the same table

So I gave the 2 MB table a different job: rank a page's elements against an agent's goal, trained on Mind2Web [1], where people completed tasks on real websites, one target element per step: the element they clicked, typed into or chose from.

Every element becomes one short line: its tag, its labels (aria-label, title, alt, placeholder and a few others), up to 24 words of its own text, then | and up to 12 words of its parent's text. A link reads a Coffee & Tea | Coffee & Tea. The goal is the task plus up to three previous actions.

Stage 1 cuts the goal and every line into overlapping 3, 4 and 5-byte pieces, hashes each piece into one of 6,144 rows [3], averages the rows into a 320-dimensional vector, and ranks by cosine. There is no vocabulary, so typos, case and punctuation change little; it is the subword idea of fastText [5] taken down to bytes. Type below to see the pieces two texts share.

Byte pieces

TextCompare

27 of 51 distinct pieces of the first text also appear in the second (53%).

• ·to
• tok
• oky
• kyo
• yo·
• o·t
• ·te
• tea
• ea·
• a·c
• ·ce
• cer
• ere
• rem
• emo
• mon
• ony
• ny·
• ·tok
• toky
• okyo
• kyo·
• yo·t
• o·te
• ·tea
• tea·
• ea·c
• a·ce
• ·cer
• cere
• erem
• remo
• emon
• mony
• ony·
• ·toky
• tokyo
• okyo·
• kyo·t
• yo·te
• o·tea
• ·tea·
• tea·c
• ea·ce
• a·cer
• ·cere
• cerem
• eremo
• remon
• emony
• mony·

I split Mind2Web's training websites by a salted hash: 6,669 steps to train on, and 1,106 dev steps from websites the model never sees. Training ran one page per update, with a softmax over the target element and every other candidate on the page, at temperature 0.05: Adam, learning rate 5e-4 with cosine decay, 3 epochs, 18,804 updates, 20 minutes on an M4 Pro at 0.34 GB.

Dev recall@50, the share of steps whose target element lands in the top 50, went from 38.2% for keyword search to 55.3%. Then it flattened: on its logging sample it levelled off near 53% from update 11,000 while training loss kept falling. It was memorising websites, not learning what elements mean.

A second stage that reads

The fix was a second stage with pretraining: BERT-tiny (google/bert_uncased_L-2_H-128_A-2, 4.4M parameters) [2] as a cross-encoder that reads [CLS] goal [SEP] element [SEP] for each of stage 1's top 400 and scores the pair. Reading the two together is what separates "Cancel" from "Subscribe" for the goal "Cancel my subscription", which share letters.

It trains on stage 1's own mistakes: each batch holds 8 steps of 32 pairs, the target element and 31 negatives drawn from stage 1's top 400, with a softmax per step. AdamW at 1e-4 with warmup and decay, 3 epochs, 2,350 batches, five and a half minutes on the laptop's GPU at 1.1 GB.

Dev recall@50 went from 55.3% to 72.5%, and on the Viator step the Coffee & Tea link climbs from #363 to #10.

Stage 2 on the Viator step

Replay

Choices, and what decided them

Question	What I measured	Chosen
BERT-tiny or BERT-mini?	Mini is 6 points better on dev (78.7% vs 72.5% recall@50) but re-scoring 400 pairs takes 1.16 s in Chrome against 0.19 s	BERT-tiny: the target was one second per page step
INT8 or full precision?	INT8 export moved dev recall@50 by +0.6 points for BERT-tiny and shrank it to 4.49 MB	INT8
WebGPU or WebAssembly?	INT8 graphs on ONNX Runtime Web's WebGPU backend gave wrong scores, agreeing with Python's top 50 on only 9-13 of 40 steps	WebAssembly
Fuse stage 1 with BM25?	Reciprocal rank fusion scored 54.3% recall@50 against 55.3% for stage 1 alone	No fusion
How deep should stage 2 read?	Stage 1's top 300 / 400 holds the target element on 84.7% / 87.6% of dev steps, a ceiling for any re-ranker	400

One page step in Chrome

• BERT-tiny, INT8, WASMreleased: 4.49 MB, correct236 ms
• BERT-tiny, FP32, WebGPUcorrect, 17.6 MB download111 ms
• BERT-tiny, INT8, WebGPUwrong scores322 ms
• BERT-mini, FP32, WebGPUcorrect, 44.8 MB download505 ms
• BERT-mini, INT8, WASMcorrect, 11.4 MB1,203 ms

Results on dev

Every decision above was made on the dev split. The table and chart compare the systems I tried; "released" is the INT8 files that ship.

System	Recall@1	Recall@10	Recall@50	Recall@100	MRR	Tokens, top 50
BM25	3.6%	13.0%	38.2%	50.8%	0.073	21x fewer
bge-small, untrained	3.3%	12.5%	31.4%		0.066	21x fewer
Stage 1	7.8%	31.1%	55.3%	66.9%	0.155	34x fewer
Stage 1 + BM25 fusion	5.1%	24.1%	54.3%	68.7%	0.114	26x fewer
Stage 1 + BERT-mini	16.5%	54.2%	78.7%	84.2%	0.285	44x fewer
Released, stage 1 + BERT-tiny, INT8	10.9%	49.3%	73.1%	80.1%	0.229	46x fewer

Recall at k, dev split

• BM25
• Stage 1
• spotlit (released)
• Stage 1 + BERT-mini

Data behind the chart

k	BM25	Stage 1	spotlit (released)	Stage 1 + BERT-mini
1	3.6%	7.8%	10.9%	16.5%
10	13%	31.1%	49.3%	54.2%
50	38.2%	55.3%	73.1%	78.7%
100	50.8%	66.9%	80.1%	84.2%
200	64.1%	79.3%	85.6%	86.6%

Opening the test splits once

Then I froze the files, recorded their checksums, and opened Mind2Web's three test splits once.

Test split	Steps	BM25 @50	Stage 1 @50	spotlit @50	@10	@100	MRR
Cross-task	2,094	44.2%	69.2%	80.8%	57.2%	85.8%	0.274
Cross-website	1,373	43.8%	56.7%	75.8%	48.7%	82.3%	0.219
Cross-domain	5,911	44.5%	57.7%	75.9%	51.5%	83.6%	0.251

The columns are recall at 50, 10 and 100. On every split the top 50 is 32-43x fewer tokens than the full page.

Dev agrees with cross-website and cross-domain within three points; cross-task is higher because its websites appear in training. MindAct's ranker from the Mind2Web paper [1] reports 88.9 / 85.3 / 85.7% on the same splits with 184M parameters, so spotlit is 8-10 points behind at about a 29th of the size, with element descriptions that differ, so the gap is approximate.

In the browser

Stage 1 and the WordPiece tokenizer are plain JavaScript, stage 2 is an INT8 ONNX file on ONNX Runtime Web. I checked every runtime against the Python reference before trusting a number from it:

Check	Result
Byte n-gram encoder, JavaScript and C++	31 of 31 golden cases identical
WordPiece tokenizer, JavaScript, against the reference fast tokenizer	667 of 667 pairs give identical token ids, including accents, CJK, emoji and truncation
Whole pipeline in Chrome	top-50 membership agrees with Python on 40 of 40 dev steps

A page step takes 236 ms on eight threads: 35-45 ms for stage 1, about 10 ms for tokenization, 185 ms for stage 2. Without cross-origin isolation, as on most hosted pages, it runs on one thread and takes about a second.The tokenizer was the hardest part to match: the Rust tokenizer's pair truncation is not the textbook longest-first rule, so the JavaScript version was fitted to 1,849 length pairs.

Inside real agents

Offline recall is not task success, so I gave Claude Code a small browser server with two modes: one returns every visible element, the other only spotlit's top 50 for the goal the agent passes.

Seven multi-page tasks, Claude Sonnet, one run each	Every element	spotlit top 50
Tasks finished	7 of 7	7 of 7
Input tokens	2,700,384	427,787
Cost	$3.55	$0.48
Time	149 s	173 s
Tool calls	48	55

I also replayed 30 random dev steps, asking the model to pick the element from each list:

Agent	List shown	Right element	Target element in the list	Input tokens
Claude Code, Sonnet	every element	15 of 30	30 of 30	964,166
Claude Code, Sonnet	spotlit top 50	13 of 30	26 of 30	76,740
Codex	every element	12 of 23	23 of 23	560,558
Codex	spotlit top 50	9 of 23	20 of 23	210,525

Thirty steps is too few to separate a difference of two from noise, but the pattern is consistent: a little accuracy for a large saving in tokens. It was not faster: with spotlit the agent made more calls and took longer.

What I learned

• Pretraining carried over; table size did not. Stage 1 memorised the websites it saw, and every bigger or cleverer table I tried before it stalled. The gains on new websites came from stage 2's pretrained weights.
• Train the second stage on the first stage's mistakes. Negatives drawn from stage 1's own top 400 taught stage 2 exactly the confusions it would face.
• Measure in the runtime you ship. The WebGPU bug only showed up in Chrome, and only for INT8; Python said everything was fine.
• Watch for leaks in the data. Mind2Web lists the target element first among the candidates, so every tie breaks by element id, never by position.
• Open the test set once. All the choices above were made on dev, and the test numbers were the last thing I looked at.

Where it falls short

Limit	Detail
A shortlist, not a picker	The target element is first on about 11% of dev steps; the agent's model still decides
Misses about one step in four on new websites	Cross-website recall@50 is 75.8%; an agent needs a fallback such as a wider list or the full page
Stage 1 caps stage 2	Its top 400 holds the target element on 87.6% of dev steps
Look-alikes	For "Sign in to my account", stage 2 ranks "Sign up free" first and "Log in" fifth
English sites from 2023, text only	No screenshot, position or visibility; iframes and shadow roots are not collected

Try it on a real page in the demo, which also has the weights to download, and see each stage interactively in the full write-up.

References

1. Deng, Gu, Zheng, Chen, Stevens, Wang, Sun and Su. Mind2Web: Towards a Generalist Agent for the Web. NeurIPS 2023, Datasets and Benchmarks Track. The data, and the MindAct ranker spotlit is compared with.
2. Turc, Chang, Lee and Toutanova. Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. 2019. The BERT-tiny checkpoint stage 2 starts from.
3. Weinberger, Dasgupta, Langford, Smola and Attenberg. Feature Hashing for Large Scale Multitask Learning. ICML 2009. Signed hashing into a fixed number of rows, as stage 1 does.
4. Xiao, Liu, Zhang, Muennighoff, Lian and Nie. C-Pack: Packed Resources For General Chinese Embeddings. 2023. The BGE models, including the bge-small-en-v1.5 teacher.
5. Bojanowski, Grave, Joulin and Mikolov. Enriching Word Vectors with Subword Information. TACL 2017. Words as bags of character n-grams.