OSS Contribution Copilot — measure before you build
A supervised multi-agent system that finds claimable issues in open-source repositories and drafts the claim comment — a human approves every send. Before building the agent, I audited the assumptions underneath it. Four were wrong, and each wrong one made the design better.
GitHub REST API · 2,568-issue label audit · needs-info calibration across 5 ecosystems · LLM readiness judge · MLflow · MCP server on PyPI (oss-issues-mcp) · LangGraph + Postgres checkpointer (in progress)
The premise
A new open-source contributor drowns before writing a line of code: the repositories worth contributing to carry thousands of open issues, with no signal for which ones are a genuine fit and which are actually available — unclaimed, still wanted, not already fixed by a linked PR. OSS Contribution Copilot is my answer: a supervised multi-agent system that watches a set of repositories, judges whether each new issue is actionable, filters to what's claimable, and drafts the "I'd like to take this" comment — which I approve or reject before anything reaches a maintainer. I'm building it to find and land my own first upstream contributions, so I'm also its first user.
The story here is not "I built an agent." The agent is the least interesting part, and it isn't finished — the pipeline is in active development, and this page says so wherever it matters. The story is that before building it, I tested the assumptions it would stand on. Four of them failed. Every failure improved the design.
Why this opens with measurements
Most project write-ups open with an architecture diagram. The diagram is the cheapest part — you can sketch a supervisor/worker graph in an afternoon. What a system's metrics mean is decided lower down, by the datasets underneath it: if the labels a judge is calibrated on don't encode real judgment, the judge's agreement score is a number about nothing. So the first phase of this project produced no agent at all. It produced audits.
It's the same thesis as DocuMind — no claim without a measurement behind it — applied to a messier problem. DocuMind's ground truth I could build and validate by hand. Here, ground truth had to be mined out of other people's process, and most of what looks like ground truth on GitHub turns out not to be.
The findings
Five measurements, all run before any pipeline code existed. Four of them broke an assumption I was about to build on; one — the fourth — found the signal worth building on instead.
I assumed a type label was an expert verdict — that bug means a maintainer read the issue and agreed. So before calibrating anything against labels, I audited who actually applies them, across 2,568 issues in 5 repositories. The issue author applies the first type label 90.7% of the time; maintainers 7.6%; bots 1.7%. And 92.1% of those labels land within two minutes of the issue being opened — median time-to-label, 0.00 hours. Nobody reads and judges an issue in under two minutes. The mechanism is mundane: GitHub issue templates attach labels at submission, so a type label mostly records which form the author picked, not what an expert concluded.
Consequence: the issue-type classifier — originally the headline feature — was demoted to a fallback path. Training a model to reproduce these labels would mean training it to predict template choice.
| applied by | share |
|---|---|
| issue author | 90.7% |
| maintainer | 7.6% |
| bot | 1.7% |
The complement is its own finding: a system that triages by reading existing labels is blind to nearly a third of the stream. And not a random third — the unlabelled issues skew toward the neglected ones, which is exactly where an unclaimed opportunity for a new contributor lives. This is why the demoted classifier survived as a fallback instead of being cut outright: its job is the unlabelled 29%, not second-guessing the labelled majority.
Duplicate detection was on the roadmap, with pgvector underneath it for embeddings. Before building either, I measured how often duplicates actually occur: 19 issues out of 2,568. That is far below the rate that would justify the feature — and far too few positives to evaluate it honestly. So I cut the feature, and pgvector went with it: duplicate detection was one of its two planned consumers, and the other didn't survive the audit either. The measurement is committed to the repo as a null result, treated the same as any other result.
One label class survived the audit: needs-info — the label a maintainer applies when an issue can't be acted on without more from the reporter. Unlike type labels, it can't come from a template; in the sample I checked, every single one was applied by a non-author. It's the one place in the data where a real maintainer judgment is recorded.
So reproduction-readiness — could a maintainer act on this issue as written? — became the system's primary judgment: an LLM judge calibrated against those maintainer decisions across five ecosystems — pandas, kubernetes, angular, ray, gradio.
The first calibration set had a flaw I only caught by reading it. It paired maintainer-flagged needs-info issues as positives with completed issues as negatives — reasonable on paper. Hand-checking rows, I noticed the negatives read like they were written by people who already knew the codebase. Measured: 51.3% of negatives were insider-authored versus 12.0% of positives — a 39.3-point gap between the classes, and in kubernetes, the worst case, 71.7% versus 13.3%.
That gap is a shortcut. An LLM judge could score well on this set by detecting writing style — does this read like an insider? — without ever assessing information sufficiency, the thing I claim to measure. And a style detector collapses exactly where the system deploys: on incoming issues, roughly 88% of which are outsider-authored. The fix held the confounder constant: both classes restricted to author_association=NONE. The confounded v1 stays in the repo, because the difference between v1 and v2 is the evidence that the confound existed and was handled.
The MCP server
The data layer shipped as a standalone artifact: oss-issues-mcp, an MCP server published to PyPI and listed in the official MCP Registry. Any MCP client can point it at a set of repositories and triage issues with it today, independent of the rest of the pipeline.
GitHub already ships an official MCP server, so wrapping the same API again would be busywork. The justification for a custom server is that these four tools are derived, not passthrough — each answers a triage question, not an API call. "Is this issue claimed?" is not a GitHub endpoint; it's a judgment assembled from assignees and linked pull requests:
get_actionable_issue— fetches one issue and strips the template scaffolding, returning only the triage-relevant fieldslist_new_issues— new open issues across a repository allowlist, with pull requests excluded (the GitHub API returns both as "issues")get_claim_status— whether an issue is already being worked on: assignees plus linked pull requestsget_repo_context— the repository's CONTRIBUTING.md and issue-template rules, so a drafted comment can follow the house rules
The security posture assumes the worst: a repository allowlist bounds what the server will touch; the token scope is read-only and least-privilege; there are no write tools at all — none will exist until the human-approval gate does; and all issue text is treated as untrusted input, because issue bodies flow into prompts and are therefore a prompt-injection surface. Ten offline unit tests pin the pure-function logic — no network, no live GitHub state in the loop.
The architecture — in progress
Everything above this line is built and measured. This section is design. The pipeline described here is in active development, not shipped software.
planned pipeline — everything upstream of the approval gate is read-only
The orchestration is a LangGraph supervisor/worker graph with a Postgres checkpointer, so an interrupted run resumes instead of restarting. Two rules keep it boring in the right places: routing lives in deterministic Python, not in prompts — the LLM judges individual issues, it doesn't decide where the pipeline goes next — and iteration is capped in code, not by asking a model to please stop. Nothing outbound is autonomous: the only write in the whole system, the claim comment, sits behind a human approval — and until that gate exists, the system has no write capability at all.
The decision log
Every component either earned its place with a measurement, follows from a stated principle, or was cut. The cuts matter as much as the keeps — most of this table is what the system deliberately doesn't do.
| component | call | why |
|---|---|---|
| reproduction-readiness judge | PROMOTED | the one signal backed by real maintainer judgment — finding 04 |
| human-approval gate | ARCH | nothing outbound is autonomous |
| MCP domain server | LEARN + ARCH | derived triage tools, not a GitHub passthrough |
| type classifier | DEMOTED | fallback for the unlabelled 29% — findings 01–02 |
| duplicate detection | CUT | 0.7% measured duplicate rate — finding 03 |
| pgvector | CUT | both of its consumers were eliminated |
| personalized ranking (Layer B) | CUT | no reliable answer key — deferred to behavioral logging |
| PII redaction | CUT | issues rarely contain PII |
PROMOTED / DEMOTED / CUT trace to a measured finding · ARCH is a design principle stated up front
The limitations
Three caveats I'd volunteer in any interview, because each one changes how the numbers should be read:
- The calibration set is 50/50 by construction; real needs-info prevalence is around 3%. Agreement measured on the balanced set is not deployment precision — it answers "can the judge tell the classes apart?", not "how often is it right in the wild?"
- The judge is calibrated on 5 external repositories and applied to 5 different watched ones. That transfer is an assumption, stated as one; the check is a held-out evaluation set — 40 examples, 100% non-author — tracked in MLflow.
- The fallback classifier is evaluated on labelled issues but deployed on unlabelled ones — a distribution shift by definition, so its reported F1 is optimistic. It's a fallback, and it's treated like one.
Built and measured: the 2,568-issue label audit across 5 repositories, the reproduction-readiness calibration sets (the confounded v1, kept as evidence, and the corrected all-outsider v2), a held-out 40-example evaluation set tracked in MLflow, and the published MCP server — four derived tools, PyPI + MCP Registry, ten offline unit tests.
In progress: the LangGraph supervisor/worker pipeline behind the human-approval gate, and a CI-gated evaluation suite for the readiness judge.
The payoff metric, once live: issues surfaced → PRs opened → PRs merged.