* 本條目專注在 Prompt 和 Application 層面的評估
* 相關條目
* [[LLM Evaluation]]
* [[RAG Evaluation]]
* [[Aligning Evaluation with Human Preferences]]
* Evaluation Metrics for LLM Applications In Production 介紹各種方式
* https://docs.parea.ai/blog/eval-metrics-for-llm-apps-in-prod
* RAG 任務
* Agent 任務 e.g 達到目標和訊息量
* 摘要任務
* Evaluation of LLMs (2024/2)
* https://blog.premai.io/evaluation-of-llms-part-1/ 各種指標
* https://blog.premai.io/evaluation-of-llms-part-2/ 用 LLM 評分
* LLM as a Judge: Numeric Score Evals are Broken!!!
* https://twitter.com/aparnadhinak/status/1748368364395721128
* 推荐人工编写多个评分细则,让模型针对某个细项打0/1分或者0/0.5/1分,打分前先输出reason再给出分数
* https://twitter.com/9hills/status/1787439509665403252 2024/5/6
* eval/llm-as-judge 資料整理 https://x.com/eugeneyan/status/1803224103782064170 2024/6/19
* Claude 的 Create strong empirical evaluations 文件
* https://docs.anthropic.com/en/docs/build-with-claude/develop-tests
* LLM Evaluation doesn't need to be complicated (2024/7/11)
* https://www.philschmid.de/llm-evaluation
* LLM as a Judge: 用語言模型來評估好壞 (2024/8/25)
* https://ywctech.net/ml-ai/paper-llm-as-a-judge/
* G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (2023/3)
* https://arxiv.org/abs/2303.16634
* Tips: 評估也需要 CoT 思考過程,不應該只直接輸出結果分數
* 在 [[Waves in AI 2024]] 利用大型語言模型自動評分學生作業 中,李宏毅 老師做的實驗也是這麼認為的!!
* 後來出 paper 了: A Closer Look into Using Large Language Models for Automatic Evaluation
* https://aclanthology.org/2023.findings-emnlp.599/
* Enhancing LLM-as-a-Judge with Grading Notes (2024/7/22)
* https://www.databricks.com/blog/enhancing-llm-as-a-judge-with-grading-notes
* 評估 prompt 裡面附上 Grading Notes 評分指南
* Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge)
* https://eugeneyan.com/writing/llm-evaluators/
## App 相關
* Evaluating LLM Applications (2024/2/6)
* https://humanloop.com/blog/evaluating-llm-apps
* 比較 overview 講 LLM app 評估
* Your AI Product Needs Evals (2024/3/29)
* https://hamel.dev/blog/posts/evals/
* 評估實戰
* Data Flywheels for LLM Applications (2024/7/1)
* https://www.sh-reya.com/blog/ai-engineering-flywheel/
* Large Language Model System Evals in the Wild (2024/9) 電子書
* https://forestfriends.gumroad.com/l/001_llm_system_evals_in_the_wild?layout=profile
## Prompt
* from https://blog.langchain.dev/the-prompt-landscape/
* https://smith.langchain.com/hub/simonp/model-evaluator?ref=blog.langchain.dev
* https://smith.langchain.com/hub/wfh/automated-feedback-example?ref=blog.langchain.dev
* https://smith.langchain.com/hub/smithing-gold/assumption-checker?ref=blog.langchain.dev