|
August 5, 2026
|
|
Why LLMs fail at tables: it's the dimensions
|
Zero-Mem runs agent memory with zero LLM calls — and 57.6% less memory-operation time
|
|
|
|
|
IN TODAY'S ISSUE
|
|
📊 Tabular Learning·Four explanations for why LLMs flunk table prediction are falsified; the survivor is input dimensionality.
🧠 Agent Memory·Zero-Mem runs structured agent memory with no LLM calls outside final answering, cutting memory-op time 57.6%.
🧰 Tool Use·A benchmark that scrambles tool semantics finds agents build an internal map of the API — then ignore it and search exhaustively.
📐 Evaluation·Nearly half of 60 language-model benchmarks show saturation; expert curation, not a private test set, predicts longevity.
🎛 Test-Time Compute·GradCuit optimises hidden states instead of sampling more, adding 6.6 points over chain-of-thought on a frozen LLM.
|
|
|
|
|
|
|
|
Quick Brief:
|
|
Frontier LLMs are strangely bad at the most ordinary machine-learning job there is — predicting a label from a table of numbers. A new study tests five plausible explanations, kills four of them, and finds the survivor is the number of input columns.
|
|
The Details:
|
- The setup is deliberately bare. One generation pass over a prompt holding the full training set and the test points — no tools, no scaffolding, no fine-tuning — isolating the model's own inductive behaviour.
- Four hypotheses falsified. Controlled experiments rule out noisy or non-linearly-separable data, the flattened CSV format obscuring columns, numeric tokenisation, and the number of test points per query.
- Dimensionality is decisive. Sweeping random linear projections of 31 benchmark datasets, the LLM is the only one of nine methods whose accuracy falls as dimensionality grows; every classical baseline stays flat or improves.
- In 2D it behaves like a nearest-neighbour method. Compared against 252 configured classical models, the LLM's decision grid matches a local, distance-based learner up to 91.6% of the time. In higher dimensions no classical model reproduces its predictions — not even with tuned, dimension-dependent noise injected to mimic degradation.
|
|
Why It Matters:
|
|
The authors claim no mechanism, only that the capability dissolves with dimension in a way no noise-corrupted classical learner imitates. For tabular foundation models that reframes the target: the problem isn't formatting or arithmetic, it's what in-context learning does to geometry as columns pile up.
|
|
|
|
|
|
|
Quick Brief:
|
|
Most agent-memory systems spend LLM calls on memory itself — summarising interactions, writing records, mediating retrieval. Zero-Mem asks whether that generation step is needed at all, and answers no.
|
|
The Details:
|
- No tokens outside the answer. Only the final QA reader invokes an LLM (encoder compute is counted separately), and raw interaction traces stay as the system of record — so no detail gets merged away by a summary.
- Two views, weighed per query. An entity–context graph exposes links across sessions; a temporal hierarchy preserves conversational locality. Retrieval draws from both, and a deterministic calibration step drops conflicting evidence before answering.
- Competitive accuracy, much cheaper. On long-memory and long-context QA benchmarks it matches baselines while cutting memory-operation time by 57.6% versus the fastest compared system, at the same reader and context budget.
|
|
Why It Matters:
|
|
A useful counterweight to the "add another LLM call" reflex in agent design: structured memory can be a retrieval-and-indexing problem rather than a generation problem.
|
|
|
|
Of 60 language-model benchmarks studied, nearly half show saturation, with rates rising by age. Resilience tracked with expert curation — not with keeping the test set private.
|
|
When AI Benchmarks Plateau
|
|
Agents brute-force even when they've already mapped the answer. ScrambleToolBench hides tool names and parameters, then drifts their behaviour mid-episode, forcing agents to learn the API by interacting with it. State-of-the-art agents build a usable internal model of the tools, then ignore it and revert to exhaustive search — a structural-reasoning failure more time or memory doesn't fix. huggingface.co →
|
|
OpenAI publishes safeguards after third-party cyber-eval incidents. A follow-up to the exploit chain surfaced during external cyber-capability testing, with process changes for how third parties evaluate its models — worth a look if you red-team under someone else's terms. openai.com →
|
|
|
Inserts optimisable latent vectors between Transformer layers and backpropagates reward-weighted gradients from the answer into them, reaching 64.5% average accuracy (+6.6 over chain-of-thought) on a frozen model.
|
|
|
|
|
A multi-agent pipeline turns ~2,000 public skills into 27k+ executable trajectories, kept only after rule- and agent-based verification with feedback-guided repair, plus a disjoint-pool benchmark.
|
|
|
|
|
Conditioning chain-of-thought supervision on ground-truth futures teaches driving models to rationalise a known answer; revealing the trajectory only after the decision, as RL feedback, restores causal reasoning.
|
|
|
|
|
One decoder-only pass emits dense vectors plus sparse lexical weights — built by partitioning the vocabulary — for text and image retrieval with no separate sparse module.
|
|
|
|
|
Places Gaussians on depth-derived surface supports rather than a pixel grid, so one photo yields state-of-the-art novel views under large viewpoint change while staying feed-forward.
|
|
|
|
|
That's a wrap!
|
Have feedback — suggestions, requests, or something you'd change? Share it anonymously here, or just reply to this email.
|
|
Stay curious ✨
|
|
|
|
|
You're receiving this because you subscribed to AI Research Daily.
|
|
|
|
|
