TRACE-992A-44BF8 · Stale Source

User Query

"2025년 4월 이후 변경된 환불 정책은 무엇인가요?"

34
RagScore
Failure
F8. Stale Source
Latency
1,240ms
Tokens
3,847
Cost
$0.024
Model
GPT-4o
Timestamp
10:58:12 KST
Retrieval Pipeline Trace
QueryNatural language0ms
Embeddingtext-emb-3-large45ms
BM25 Search12 candidates32ms
Dense Search18 candidates67ms
Merge24 unique8ms
Rerankbge-reranker-v2180ms
Top-K SelectK=52ms
ACL FilterPassed5ms
GenerationGPT-4o890ms
Retrieved Chunks (Top 5)
#1
refund_policy_2024_v2.pdf2024-03STALE

Section 3.1 — "고객은 결제 후 7일 이내에 100% 환불을 요청할 수 있습니다."

Vector
0.92
Rerank
0.88
#2
refund_policy_2024_v2.pdf2024-03STALE

Section 3.2 — "엔터프라이즈 계약의 경우 별도 환불 협의가 필요합니다."

Vector
0.89
Rerank
0.85
#3
billing_faq_2023.pdf2023-09STALE

Q: 30일 이후에도 환불 가능한가요? A: 아닙니다.

Vector
0.87
Rerank
0.82
#4
terms_of_service.pdf2025-01

Article 12.3 — Payment terms shall be governed by the billing agreement.

Vector
0.84
Rerank
0.79
#5
refund_policy_2025_v3.pdf2025-04CORRECT SOURCE

Section 2.1 — "2025년 4월부로 환불 기간을 14일로 확대합니다."

Vector
0.81
Rerank
0.76
Generated Answer

환불 규정에 따르면 결제 후 7일 이내에만 100% 환불이 가능합니다.

Source: refund_policy_2024_v2.pdf · Outdated by 13 months
Expected Answer

2025년 4월 개정된 환불 정책에 따르면 결제 후 14일 이내 100% 환불이 가능합니다.

Source: refund_policy_2025_v3.pdf · Latest version
AI Diagnosis
Root Cause: Stale Document Priority

The correct document (refund_policy_2025_v3.pdf) was retrieved as chunk #5. However, three stale versions from 2024 scored higher because they share near-identical embeddings and have more indexed chunks, resulting in higher aggregate similarity.

Contributing Factor: No Time-Decay Scoring

The effective_date metadata field exists but is not used in retrieval scoring or reranking. All document versions are treated equally regardless of recency.

Recommended Fix

1. Enable effective_date metadata filter: >= 2025-01-01

2. Apply time-decay scoring in reranker with λ=0.1 exponential decay

3. Mark superseded document versions as deprecated in metadata

Projected Impact if Fixed
RagScore
3478
Recall@5
6088
Citation
2085
Freshness
3095