Test kit · original Oprill material
Test the questions people actually need to answer.
An original guide to evaluating workplace search with a small source set, explicit permissions and answerable questions. Use it to turn a search demonstration into a reviewable exercise.
Use this guide to work through:
- Assemble a small fictional source set.
- Define several kinds of question.
- Test a permission difference.
Assemble a small fictional source set.
Create three short documents: a current travel policy, an archived travel policy and a project note that mentions travel without defining policy. Label each with an owner and a version date. Keep the content synthetic for the first exercise.
Give the current policy a simple rule: travel requires manager approval before booking. Give the archived policy a conflicting rule: approval can follow booking. The project note should point to the current policy without restating it.
Define several kinds of question.
Try an exact lookup, such as finding the current travel policy. Then ask a meaning-based question: “What must I do before booking a trip?” Finally ask a question absent from the source set, such as which hotel to choose.
For the meaning-based question, the expected result should identify the current approval rule and its source. For the unsupported hotel question, the correct response should acknowledge that the source set does not establish a choice.
- Check which version is shown first.
- Check whether the answer cites the current policy.
- Check that an unrelated project note does not become the authority.
Test a permission difference.
Add a fictional restricted document with a distinctive phrase. Evaluate from two test identities whose access differs. The restricted phrase should not appear in results or answer text for the identity without access.
Document the intended access model before the test and include behavior after a permission change. Actual implementation details depend on the selected system; this workbook does not claim that Oprill provides these controls.
Review the result beyond rank.
Record whether a person can recognize the source, assess its currency and open the supporting material. A relevant-looking title is insufficient if the result hides that the content has been superseded.
Keep separate notes for retrieval errors, misleading summaries and presentation problems. These failure categories lead to different fixes and should not be compressed into a single impression of quality.
Expand only when the exercise is understood.
After correcting the small test, add representative tasks with source-owner approval. Keep expected answers and permission cases available to reviewers so the evaluation can be repeated.
Record the sample and its limits alongside any measured result. A synthetic test is useful for finding obvious problems; it is not evidence of broad accuracy, time savings or readiness for unrestricted use.