Turning one sentence into a test

From Seeing without Looking (arXiv 2605.22903). Decisions for running it on an 8 GB Apple M1. Code: github.com/ashe-many/vlm-token-drop.

Decision 1 What does the sentence claim?
"Even at 75% token removal, accuracy remains nearly unchanged." Figure 2 caption, on the POPE benchmark
Figure 2 of the paper: POPE accuracy for five vision-language models at 0%, 25%, 50% and 75% image token drop. All lines stay between about 90% and 97%.
Figure 2 from the paper: accuracy on POPE as image tokens are dropped. Source: arXiv 2605.22903, CC BY 4.0.
Claim
Accuracy stays flat when image tokens are removed.
What it needs
Accuracy measured at several drop rates on the same questions, so the shape of the curve is visible.
Drop rates to test
  • 0%
  • 25%
  • 50%
  • 75%
  • 100%
Decision
Measure accuracy at each drop rate on the same POPE questions.
Decision 2 Which model, and where to remove tokens?
"Image embeddings are taken after the visual encoder and projection/merger module, i.e., the final image-token representations that are normally concatenated with text tokens before entering the LLM." Appendix A.4, Motivation Example Setup
In ASD-STE100

The paper takes the image tokens after the vision encoder and after the merger. At this point, the image tokens are in their last form. Usually, the code puts these image tokens into the text tokens. The text tokens are the chat format, the two image markers and the question. The image tokens go between the two image markers. Then, the LLM receives all the tokens.

How the image becomes image tokens Image: 640 × 480 pixels Cut into 16 × 16 pixel patches 40 × 30 = 1200 patches One patch 16 × 16 pixels × 3 colors = 768 numbers ⋮ ⋮ 768 1024 Vision encoder 1200 × 1024 4 patches 4 × 1024 Merger 4096 → 2560 1 1 image token 2560 numbers 32 × 32 pixels Vision tower 1 · Patch embedding 2 · Encoder 3 · Merger 3 · Merger: details in next diagram ↓ 640 × 480 image → 1200 patches → 300 image tokens (20 × 15). Each image token covers 32 × 32 pixels.
Vision tower, 3 · Merger, as table shapes 1024 1200 1200 × 1024 from 2 · Encoder A B C D 4096 = 1024 × 4 300 300 × 4096 4 rows → 1 row + GELU 4096 → 4096 300 × 4096 numbers mixed 4096 → 2560 2560 300 300 × 2560 300 image tokens Rows: 1200 → 300. Row length: 1024 → 4096 → 2560.
Qwen3-VL-4B · 4-bit · example image 640 × 480 Image 480 × 640 × 3 Vision tower Vision encoder 24 layers · 16 heads width 1024 768 → 1024 per patch 1200 patches × 1024 Merger 2×2 → 1 token 4096 → 2560 1200 → 300 tokens Cut point: qwen3_vl.py, lines 97–104 Image tokens, each with a position number 1 3 4 6 7 Remove 0–75% 300 × 2560 Kept: 300, 225, 150 or 75 Kept image tokens go into the image slots Tokens that the LLM receives, in this order Chat start Vision start 1 3 4 6 7 Vision end Is there a car in the image ? Chat end Kept image tokens keep their numbers Question Sequence: L × 2560. L = text tokens + kept image tokens. Text tokens come from the embedding table: 151,936 × 2560. Text token Image marker Image token Copies from vision layers 6, 12, 18 go in after LLM layers 1, 2, 3 · each 300 × 2560 Layer 1 Layer 2 Layer 3 Layers 4–35 Layer 36 Final norm Output head Yes/No 1 token Each layer: ≈ 101 M numbers, ≈ 57 MB at 4-bit 2560 2560 → 151,936 same table as the embedding Language part (LLM): 36 layers, ≈ 3.6 B numbers. Each token: 2560 numbers. Inside one layer (all 36 layers have this form) · ≈ 101 M numbers Input L × 2560 RMSNorm 2560 Attention Q: 2560 → 4096 (32 × 128) K, V: 2560 → 1024 each (8 × 128) Out: 4096 → 2560 Q-norm, K-norm: 128 M-RoPE: 128 = time 24, height 20, width 20 pairs 26.2 M numbers + RMSNorm 2560 MLP Gate, up: 2560 → 9728 Down: 9728 → 2560 SiLU gate 74.7 M numbers + Output L × 2560 Skip path Skip path
Decision 3 How do we know the patch is correct?
In ASD-STE100

Remove the same image tokens from four places: the input embeddings, the three deepstack copies, the M-RoPE positions and the image mask. The kept tokens keep their original positions. The paper does not give this detail. This is our decision.

Then do a test. Remove 0% of the tokens. The output must be the same as the output of the model without the patch.

Test
5 POPE questions, 0% removed. Compare the scores (logits) for the first answer token with the unpatched mlx-vlm model.
Result
Largest difference: 0. The answers are the same.
Code
check_logits.py, check_zero_drop.py
Result Does accuracy stay flat?
In ASD-STE100

Yes, up to 75%. When we remove 75% of the image tokens, accuracy goes down by only 4 points: from 91.3% to 87.3%. This agrees with the paper.

When we remove all image tokens, the model says "No" to almost all questions. Accuracy goes down to 50.7%. This is the same as a guess, because half of the answers are "Yes".

All 300 questionsAnswer is Yes (147)
POPE accuracy as image tokens are removed 0% removed: all 91.3%, yes-subset 85.7%; 25% removed: all 90.0%, yes-subset 83.0%; 50% removed: all 90.7%, yes-subset 84.4%; 75% removed: all 87.3%, yes-subset 76.2%; 100% removed: all 50.7%, yes-subset 0.7% 0% 25% 50% 75% 100% guess 0% 269.3 kept 25% 202 kept 50% 134.7 kept 75% 67.3 kept 100% 0 kept Image tokens removed All 300 questions, 0% removed: 91.3% All 300 questions, 25% removed: 90.0% All 300 questions, 50% removed: 90.7% All 300 questions, 75% removed: 87.3% All 300 questions, 100% removed: 50.7% 51% Answer is Yes (147), 0% removed: 85.7% Answer is Yes (147), 25% removed: 83.0% Answer is Yes (147), 50% removed: 84.4% Answer is Yes (147), 75% removed: 76.2% Answer is Yes (147), 100% removed: 0.7% 1%
RemovedImage tokens kept (mean)Accuracy, allAccuracy, answer is YesModel says YesNot yes/no
0%269.391.3%85.7%43.7%0
25%20290.0%83.0%42.3%0
50%134.790.7%84.4%43.0%0
75%67.387.3%76.2%38.3%0
100%050.7%0.7%1.0%0
Data
300 POPE questions (random split), seed 0. The paper does not say which 300 questions it used. 147 of our 300 have the answer Yes (paper: 149).
Differences from the paper
4-bit MLX model on an Apple M1, not bf16 on an RTX 4090. One random seed. Our own choice of positions for the kept tokens.
Prompt
Question + "Answer the question using a single word or phrase." Greedy decoding, max 6 new tokens.
Run time
About 2 hours for 300 questions × 5 drop rates on an 8 GB M1.