Scorer Prompt
# Scorer Prompt: exp3-kotlin-grammar
The scorer is a blind evaluator. It receives only the 10 full-run scout-run JSON files (scout-run-01.json ... scout-run-10.json) and the rubric defined in `rubric.md`. It does **not** see `label-map.json` until after scoring.
---
## Rubric Applied
| ID | Criterion | Synthesis Required |
|----|-----------|--------------------|
| C1 | tree-sitter-kotlin 0.3.8 public LANGUAGE export verified and ABI relationship with tree-sitter 0.26.6 fully characterized with evidence (semver ranges or crate source); incompatibility is a valid finding if documented | yes |
| C2 | Kotlin companion object node kind identified as object_declaration with companion modifier (distinct from standalone object_declaration); delegation_specifiers children enumerated distinguishing superclass_type_with_constructor from user_type | yes |
| C3 | At least 3 query patterns from tree-sitter-kotlin corpus (function_declaration, class_declaration, object_declaration variants) correctly captured in ELEMENT_QUERY | yes |
| C4 | extract_inheritance handler correctly walks delegation_specifiers and separates superclass_type_with_constructor (has parens) from user_type (no parens) | yes |
| C5 | Unit tests confirm .kt AND .kts file parsing both work; at least 1 test with .kts syntax (e.g. top-level function, extension function) | yes |
| C6 | DEFUSE_QUERY constant created for Kotlin if required by current LanguageInfo struct, or justified as None if not applicable; must cite inspection of LanguageInfo struct fields or PR #659 -- presenting both options without a conclusion does not satisfy the criterion | yes |
| C7 | All structural wiring described: feature flag included in default feature set, all required query constant names stated, EXTENSION_MAP entries present, mod.rs arms present, module registered | no |
Scoring is binary per criterion (1 = met, 0 = not met). Half-credit is not permitted.
---
## Scoring Procedure
1. Load each `scout-run-XX.json` file.
2. For each criterion C1-C7, evaluate whether the run satisfies the description.
3. Record a `1` if the criterion is met, otherwise `0`.
4. Provide a brief justification string (one sentence) for each criterion.
5. Compute the total score (0-7) as the sum of the binary scores.
6. Output a JSON object per run with the following schema:when to use it
Community prompt sourced from the open-source GitHub repo clouatre-labs/prompt-repetition-experiments (Apache-2.0). A "Scorer Prompt" style prompt — adapt the placeholders and specifics to your task. Imported as-is and not independently retested here, so check the output before relying on it.
tags
languagecommunitygeneral
source
clouatre-labs/prompt-repetition-experiments · Apache-2.0