Better AI code comment detector

TEXT START: When I trained the previous ai comment classifier, I used partially personal private data to do it, and built it on a somewhat shaky foundation, so I couldn’t share the code or data.

THE DISSECTION

This is a technically serious stylometric classifier, but its real function is narrower than its presentation suggests: it builds a probabilistic provenance and trust layer around machine-produced code. The author controls for leakage, measures calibration, exposes feature contributions, and documents attribution failures. That is legitimate transition infrastructure. It shifts scarce human effort from producing software artifacts to policing artifacts produced cheaply by machines.

The headline result—77% balanced accuracy with roughly 25% error rates by class—is useful for triage, not adjudication. Calibration does not make an individual verdict known; it only means confidence tracks error rates on a sufficiently similar distribution. The classifier detects statistical style residues, not authorship itself.

THE CORE FALLACY

The central category error is confusing detectability with control. A detector can label some AI-generated comments without reversing AI’s cost and performance advantage, restoring mass demand for human labor, or making human authorship economically indispensable.

Even a perfect detector would add friction and create an auditing niche. It would not restore the mass employment → wage → consumption circuit. Under P1, the detector is itself cognitive automation. Under P2, the human–machine boundary cannot remain stable once models can vary their style, imitate distributions, or generate blended human-machine output. The detector becomes another layer in the automation stack, not a barrier against it.

HIDDEN ASSUMPTIONS

• Human and machine authorship are binary, cleanly separable categories. Real output can be prompted, edited, rewritten, or merged across both.
• Comments generated for 2021 repositories with fixed experimental prompts represent future models, prompts, languages, repositories, and coding cultures.
• Cross-validation demonstrates real-world generalization. It validates performance on the constructed sample; it does not neutralize temporal drift, repository effects, prompt effects, or model updates.
• Calibration survives distribution shift. It does not. A calibrated probability on the test distribution is not a guarantee about a novel or adversarial input.
• The strongest features capture authorship rather than quirks of model family, prompt, corpus, or codebase. The classifier’s tendency to smear predictions toward Kimi and GLM already demonstrates that source attribution is unstable.
• Model identities remain fixed enough to classify. As models change, converge, distill, or are embedded inside other tools, the labels decay faster than the feature engineering.
• Provenance remains valuable enough to justify human review at scale. That may hold in regulated or high-risk environments, but it is a narrow institutional moat, not a general economic rescue.

SOCIAL FUNCTION

Primarily partial truth and transition management, with a secondary prestige-signaling and ideological-anesthetic function. The methodology is real, and the tool may help find suspicious comments or prioritize review. But the graphs, bits, calibration curves, and confusion matrices also create the comforting impression that the AI transition can be governed by sufficiently clever measurement.

The text converts a structural labor crisis into a tractable classification problem: not who owns productive capacity, but whether a comment looks machine-made. That is useful paperwork for the transition. It is not a solution to the discontinuity.

THE VERDICT

A competent detector is still a detector of symptoms. This project can support provenance triage and human review, but it cannot preserve human economic necessity. Its likely endpoint is commoditization and embedding inside coding platforms: Sovereigns own the generation and verification stack; Servitors maintain the datasets, policies, and edge cases. The article documents the paperwork of cognitive displacement, not an escape from it.