mlx-community/Qwen2.5-14B-Instruct-4bit local
Dialogue overall
—
no runs yet
Gaeilge overall
2.11
11 prompt(s)
Best gameplay $/hr
—
no perf cost data
Best p50 latency
—
no perf data
Gaeilge fluency
judged by ⚖ gaeilge_fluency_judge_v1 (claude-sonnet-4-6) · 11 prompt(s) · english-leakage flag rate 9.1%
| Axis | Score |
|---|---|
| Overall | 2.11 |
| fluency | 2.09 |
| grammar | 2.27 |
| idiom | 2.09 |
| task_fulfillment | 1.91 |
| english_leakage | 4.82 |
Gaeilge prompts & responses
All 11 judged prompt(s) with per-axis scores and the judge's reason.
11 of 11 prompt(s)
gaeilge-0001 flue 1.0gram 1.0idiom 1.0task 1.0engl 5.0 overall 1.5
Prompt: Translate into natural Irish Gaeilge: "The harvest was good this year."
Reply: Ba mhaith an t-seóil sa bhliain seo isteach.
Judge: The candidate response uses 'seóil' (a sail/luck idiom) rather than 'fómhar' (harvest), misses the past tense of 'bí', and the phrase 'sa bhliain seo isteach' is not standard Irish for 'this year' (i mbliana); the sentence does not convey the meaning of a good harvest this year.
gaeilge-0002 flue 1.0gram 1.0idiom 1.0task 1.0engl 5.0 overall 1.0
Prompt: Translate into natural Irish Gaeilge: "I have to go to the bank."
Reply: Máthair mhair dom chon banc.
Judge: The candidate response is nonsensical Irish that does not convey the meaning 'I have to go to the bank' — it appears to be a garbled string of Irish-looking words with no coherent grammar or meaning related to the source sentence.
gaeilge-0003 flue 1.0gram 2.0idiom 1.0task 1.0engl 5.0 overall 1.5
Prompt: Translate into natural Irish Gaeilge: "I want to tell you the truth."
Reply: Is é an tionscal a mhaith liom duit a rá.
Judge: The candidate response does not translate 'I want to tell you the truth' — 'tionscal' means 'industry/enterprise', not 'truth', and the construction is garbled; the meaning is entirely lost despite no English words appearing.
gaeilge-0004 flue 2.0gram 2.0idiom 2.0task 1.0engl 5.0 overall 1.5
Prompt: Translate into natural Irish Gaeilge: "I ate the bread."
Reply: Bhíomar ag ithe an briosca.
Judge: The candidate translated 'we were eating a biscuit/cookie' instead of 'I ate the bread'; wrong subject (plural), wrong tense (progressive past), and wrong noun (briosca = biscuit, not arán = bread), completely failing to preserve the source meaning.
gaeilge-0005 flue 1.0gram 1.0idiom 1.0task 1.0engl 3.0 overall 1.0
Prompt: Translate into natural Irish Gaeilge: "You will buy the book today."
Reply: D'éirigh leat an leabhar a cheeksú inniu.
Judge: The response is largely nonsensical and does not translate 'You will buy the book today'; it uses a past tense construction (d'éirigh leat) meaning 'you succeeded' and contains the invented/nonsense word 'cheeksú' which is not Irish.
English leakage: cheeksú
gaeilge-0006 flue 1.0gram 1.0idiom 1.0task 1.0engl 5.0 overall 1.0
Prompt: Translate into natural Irish Gaeilge: "I'm looking forward to the snow." Prefer an idiomatic Irish phrasing rather than a word-for-word calque.
Reply: Is min an t-sneach atá agam a bpróimhbhricfeasaiocht.
Judge: The candidate response is largely invented pseudo-Irish gibberish ('próimhbhricfeasaiocht' is not a real Irish word) and fails to convey the meaning of looking forward to snow in any recognizable Irish phrasing.
gaeilge-0007 flue 5.0gram 5.0idiom 5.0task 5.0engl 5.0 overall 5.0
Prompt: Translate into natural Irish Gaeilge: "Does she have money?"
Reply: An bhfuil airgead aici?
Judge: The candidate response is an exact match to the reference answer, correctly using the question form with 'an bhfuil', the noun 'airgead', and the prepositional pronoun 'aici' for possession.
gaeilge-0008 flue 1.0gram 2.0idiom 1.0task 1.0engl 5.0 overall 1.2
Prompt: Translate into natural Irish Gaeilge: "I am going to the shop."
Reply: Tá muid ag imrí go dtí an teach-aitheach.
Judge: The candidate uses 'muid' (first-person plural) instead of singular, 'ag imrí' (playing) instead of 'ag dul' (going), and 'teach-aitheach' is not a real Irish word for shop (siopa); the translation is semantically wrong and contains invented vocabulary.
gaeilge-0009 flue 5.0gram 5.0idiom 5.0task 5.0engl 5.0 overall 5.0
Prompt: Léigh an abairt seo agus freagair i nGaeilge: "Bhí an fear ag siúl leis an mbean." Cé a bhí ag siúl agus cé leis?
Reply: Bhí an fear ag siúl leis an mbean.
Judge: The candidate response exactly matches the reference answer, correctly identifying the man as the walker and the woman as the companion in perfect Irish.
gaeilge-0010 flue 3.0gram 3.0idiom 3.0task 2.0engl 5.0 overall 2.5
Prompt: Athscríobh an cheist dhiúltach seo mar ráiteas dearfach simplí: "Nár labhair Seán?"
Reply: Níor labhair Seán.
Judge: The candidate produced a negative declarative ('Níor labhair Seán') rather than the required positive declarative ('Labhair Seán'); the task asked for a positive statement, not just a change from interrogative to declarative negative.
gaeilge-0012 flue 2.0gram 2.0idiom 2.0task 2.0engl 5.0 overall 2.0
Prompt: Mínigh an abairt seo i nGaeilge shimplí: "Tóg go bog é."
Reply: Is é "tóg go bog é" abairt a úsedar chun aitheantas a ghabháil ar bheith in éineacht le daoine eile, gan éigean a bhrath.
Judge: The response is in Irish with no English leakage, but the explanation is semantically wrong — 'tóg go bog é' means 'take it easy/calm down', not something about being in company with others without feeling compelled; the grammar is also awkward ('úsedar' is non-standard, 'aitheantas a ghabháil ar bh
Perf by provider
No perf measured for this model yet.