https://learn.deeplearning.ai/quality-safety-llm-applications/lesson/1/introduction
> (2024/2) 主要就是介紹 whylabs 這家的工具，可以做一些 Metrics 檢測
## Overview
https://github.com/whylabs/whylogs
https://github.com/whylabs/langkit
https://github.com/whylabs/langkit/blob/main/langkit/docs/modules.md
https://whylabs.ai/
![[Pasted image 20240224223054.png]]
![[Pasted image 20240224223243.png]]
![[Pasted image 20240224223406.png]]
![[Pasted image 20240224223519.png]]
![[Pasted image 20240224223504.png]]
![[Pasted image 20240224223511.png]]
![[Pasted image 20240224223534.png]]
![[Pasted image 20240224223659.png]]
![[Pasted image 20240224223852.png]]
![[Pasted image 20240224223905.png]]
![[Pasted image 20240224223948.png]]
![[Pasted image 20240224224050.png]]
![[Pasted image 20240224224113.png]]
![[Pasted image 20240224223935.png]]
![[Pasted image 20240224224522.png]]
![[Pasted image 20240224224556.png]]
![[Pasted image 20240224224642.png]]
![[Pasted image 20240224224652.png]]
![[Pasted image 20240224224720.png]]
![[Pasted image 20240224224733.png]]
![[Pasted image 20240224224928.png]]
![[Pasted image 20240224225001.png]]
![[Pasted image 20240224225020.png]]
![[Pasted image 20240224225034.png]]
## Hallucinations
![[Pasted image 20240224225247.png]]
![[Pasted image 20240224225259.png]]
![[Pasted image 20240224225308.png]]
![[Pasted image 20240224225804.png]]
![[Pasted image 20240224225815.png]]
![[Pasted image 20240224230114.png]]
![[Pasted image 20240224230209.png]]
![[Pasted image 20240224230858.png]]
![[Pasted image 20240224230910.png]]
![[Pasted image 20240224230948.png]]
![[Pasted image 20240224231043.png]]
![[Pasted image 20240224231342.png]]
![[Pasted image 20240224231401.png]]
![[Pasted image 20240224230751.png]]
![[Pasted image 20240224230804.png]]
## Data Leakage
![[Pasted image 20240225005000.png]]
![[Pasted image 20240225005817.png]]
![[Pasted image 20240225010324.png]]
![[Pasted image 20240225010200.png]]
![[Pasted image 20240225011011.png]]
![[Pasted image 20240225011038.png]]
![[Pasted image 20240225011026.png]]
![[Pasted image 20240225011052.png]]
## Refusals and prompt injections
![[Pasted image 20240225011403.png]]
![[Pasted image 20240225011614.png]]
![[Pasted image 20240225011758.png]]
![[Pasted image 20240225012712.png]]
![[Pasted image 20240225013002.png]]
![[Pasted image 20240225013421.png]]
![[Pasted image 20240225014102.png]]
> source code 裡面 injections 的 embedding 直接給這個....
https://whylabs-public.s3.us-west-2.amazonaws.com/langkit/data/injections/embeddings_all-MiniLM-L6-v2_harm_v2.parquet
## Passive and active monitoring
![[Pasted image 20240225015653.png]]
![[Pasted image 20240225015729.png]]
![[Pasted image 20240225015757.png]]
![[Pasted image 20240225015907.png]]