Factr-I vs Hermes
Factr-I scored higher than Hermes on all six benchmarks and held lower peak memory on all six, finishing the same 1,067 tasks on 68.1M tokens against 103.8M. Hermes costs less in dollars, $3.24 against $3.81, because Factr-I's batched calls produce more output tokens and output tokens cost more than input tokens.
Every harness ran the same 1,067 tasks from six public benchmarks on GPT-6 Luna at medium reasoning effort. Results as of 2026-10-12.
Scores
| Benchmark | Factr-I | Hermes |
|---|---|---|
| Long-context pairsOOLONG-Pairs · 20 tasks | 73.8%F1 | 29.8%F1 |
| Long-context reasoningOOLONG · 50 tasks | 90.0%45/50 | 34.0%17/50 |
| General assistantGAIA · 165 tasks | 40.0%66/165 | 28.5%47/165 |
| Terminal tasksTBLite · 100 tasks | 68%68/100 | 60%60/100 |
| Code editingPolyglot · 225 tasks | 86.2%194/225 | 81.8%184/225 |
| Long reasoningLongCoT-Mini · 507 tasks | 19.3%98/507 | 17.8%90/507 |
Peak memory
| Benchmark | Factr-I | Hermes |
|---|---|---|
| Long-context pairsOOLONG-Pairs · 20 tasks | 84MiB | 231MiB |
| Long-context reasoningOOLONG · 50 tasks | 96MiB | 221MiB |
| General assistantGAIA · 165 tasks | 53MiB | 234MiB |
| Terminal tasksTBLite · 100 tasks | 27MiB | 241MiB |
| Code editingPolyglot · 225 tasks | 390MiB | 416MiB |
| Long reasoningLongCoT-Mini · 507 tasks | 68MiB | 226MiB |
Tokens and cost
| Benchmark | Factr-I | Hermes |
|---|---|---|
| Total tokensAll 1,067 tasks | 68.1M | 103.8M |
| Total costAll 1,067 tasks | $3.81 | $3.24 |
Where each one leads
Where Factr-I leads
- Higher score on all six benchmarks, from 86.2% against 81.8% on code editing to 90.0% against 34.0% on long-context reasoning.
- Lower peak memory on all six benchmarks, including 27 MiB against 241 MiB on terminal tasks.
- Fewer tokens for the same work: 68.1M against 103.8M across all 1,067 tasks.
Where Hermes leads
- Hermes is cheaper in dollars across the full run, $3.24 against $3.81, despite using more tokens.
- Code editing is the closest score in the report, 86.2% against 81.8%.
Questions
Is Factr-I more accurate than Hermes?
On these benchmarks, yes. It scored higher on all six of them.
Which costs less to run?
Hermes, at $3.24 against $3.81 for the full run, even though Factr-I used fewer tokens to do the same work.
Why does Factr-I cost more on fewer tokens?
Its batched calls produce more output tokens, mostly on OOLONG, and output tokens are priced higher than input tokens.
Which uses less memory?
Factr-I, on every benchmark in the report, by margins from 26 MiB to 214 MiB.