CHANWOO
SONG 한국어 ↗
← All work
LANGUAGE DATA / EVALUATION2023

Travel
Chatbot

Reference examples for booking, cancellation and refund intents.

CLASSIFIER EVALUATION / DATA GUIDELINES

Case study by

Reconstructed test · booking feedback misrouted as a new request
Reconstructed test · booking feedback misrouted as a new request
01 / CONTEXT & CONTRIBUTION

The problem.
My part in it.

Background

  • A new classifier required a review of existing sentences used as reference examples.

Objective

  • Shared criteria for retaining or adding examples and for peer review.

My role

  • Designed ten test types covering example count, length, synonyms and word order; analyzed routing results.
02 / SYSTEM DESIGN

How it connects.

Experiment workflow
Reference-example experiment and review workflowJuly 2023 experiment: reference examples, not model trainingReference-example experiment and review workflowJuly 2023 experiment: reference examples, not model trainingCONTROLLED EXAMPLE CONFIGURATIONSTEST → COMPARE → RECOMMENDExample countExample countSentence lengthSentence lengthSynonymsSynonymsWord orderWord orderTen test types — Compare reference configurationsTen test typesCompare reference configurationsAnalyze intent routing — Booking / cancellation / refundsAnalyze intent routingBooking / cancellation / refundsAuthoring & review guidance — Retain / add / peer reviewAuthoring & review guidanceRetain / add / peer reviewOutput: short-query coverage, longer examples and warnings about false triggers.
Zoom for detail
Processing / data flowIteration / feedbackService / environment boundary

System diagram

100%

Drag to move · Pinch or use + / − to zoom · Arrow keys to move

03 / RESULTS & SCOPE

The outcome.

10Test types

The June development-test record is separate from the July reference-example experiment. No post-fix outcome is inferred.

NEXT PROJECT / 11Wonderline↗