Abstract
Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model. We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition. Goldsmith treats this definition as a trainable textual object. Candidate definitions are run on the same gold examples and scored with an executable structured loss, while the output schema, formatting, retrieval, repair, judging, and human review remain in an external harness. A large language model (LLM) editor converts the highest-loss failures into textual-gradient revisions, which are accepted only when the measured loss decreases. In prompt-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols. The resulting definition also improves downstream annotation when combined with retrieval, score-based routing, and human review across typed span, pair-level relation, and fixed-trigger event-argument tasks. These results show that scarce expert supervision can support both task-definition learning and scalable annotation.
Keywords
Subject
Publication details
- Journal
- Not available
- Open access
- Green open access
Cite this article
APA 7
Li, Y., Zhang, H., Jiang, X., & Guo, M. (2026). Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness. https://omanscience.com/en/articles/goldsmith-gold-loss-guided-definition-optimization-with-an-agentic-annotation-harness
MLA 9
Li, Yihan, et al. "Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness." https://omanscience.com/en/articles/goldsmith-gold-loss-guided-definition-optimization-with-an-agentic-annotation-harness.
Chicago (author–date)
Li, Yihan, Hanyi Zhang, Xiaoxi Jiang, and Man Guo. 2026. "Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness." https://omanscience.com/en/articles/goldsmith-gold-loss-guided-definition-optimization-with-an-agentic-annotation-harness.
Harvard
Li, Y., Zhang, H., Jiang, X. and Guo, M. (2026) 'Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness', Available at: https://omanscience.com/en/articles/goldsmith-gold-loss-guided-definition-optimization-with-an-agentic-annotation-harness.
Vancouver
Li Y, Zhang H, Jiang X, Guo M. Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness. https://omanscience.com/en/articles/goldsmith-gold-loss-guided-definition-optimization-with-an-agentic-annotation-harness
IEEE
Y. Li, H. Zhang, X. Jiang, and M. Guo, "Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness," https://omanscience.com/en/articles/goldsmith-gold-loss-guided-definition-optimization-with-an-agentic-annotation-harness.