RedInk: A Multi-Agent Grading Assistant / Etsub Feleke

Where This Came From

Having been a teaching assistant, I know firsthand how messy grading can get. You have a rubric that makes sense when you read it, but starts feeling ambiguous once you're actually scoring the 15th submission. And by the time you hit the 30th, your standards have probably drifted from where they started.

I kept wondering: could AI handle the tedious parts (parsing rubrics, comparing against examples, drafting initial feedback) while keeping a human in charge of the actual scoring decisions? That question turned into RedInk.

You can try the live demo here

First Attempt: One Big Prompt

My first try was simple: give one LLM the rubric and the submission, ask it to grade everything at once. The results were... not great. It would skip criteria, lose track of point values, or write feedback that didn't match the scores it gave. It became clear pretty quickly that this needed to be broken into smaller, focused steps.

Breaking It Into Agents

I split the grading workflow into four agents, each with one job, orchestrated by LangGraph to manage the shared state between them:

Rubric Analyzer --> Standard Setter --> Grading Agent --> [Human Review] --> Feedback Synthesizer

Rubric Analyzer takes the uploaded rubric (PDF, DOCX, Markdown, or plain text) and extracts structured criteria with point values. This was harder than I expected because rubrics come in so many formats: tables, bullet lists, freeform paragraphs, percentage ranges, letter grades. Using an LLM for this parsing step handled that diversity way better than any regex approach would have.

Standard Setter is optional. If the instructor provides a "golden" example (a reference assignment), this agent analyzes it and figures out what excellence looks like for each criterion. I wasn't sure how much this would matter, but it turned out to make a noticeable difference. Without it, the grading agent tends to be more lenient since it's interpreting rubric language in the abstract.

Grading Agent does the evaluation. It takes the student submission, the structured rubric, and the standards, then proposes scores with reasoning and evidence for each criterion.

class GradingResult(BaseModel):
    criterion: str
    score: float
    max_score: float
    reasoning: str
    evidence: List[str]

Feedback Synthesizer runs after human review and generates constructive feedback: strengths, areas for improvement, and next steps.

The Human Review Step

This is the part I feel most strongly about. No grade gets finalized without a TA looking at it. The workflow pauses, shows the AI's proposed scores alongside its reasoning and evidence, and waits for the TA to approve or adjust.

I've seen too many AI tools that try to fully automate things that really need a human in the loop. The philosophy here is simple: AI proposes, humans decide.

Demo in Action

Here's a quick look at the grading workflow in practice (sped up 2x):

Some Technical Choices

Temperature = 0 for all LLM calls. Grading should be deterministic. Run the same submission twice, get the same proposed scores.

Pydantic validation on every agent output. If the LLM returns malformed data, the system catches it before it moves downstream. This saved me from a lot of subtle bugs during development.

Score boundary enforcement. If a criterion is worth 10 points, the agent can't propose 11. LLMs will absolutely hallucinate out-of-range scores if you don't enforce this.

No persistent storage of student data. Everything processes in memory, scoped to the session, cleaned up after.

The system supports OpenAI GPT and in the future will support Anthropic Claude, selectable at runtime. Users bring their own API key.

Unexpected Challenges

Rubric format diversity was the biggest headache. Point values in parentheses, percentage ranges, descriptive levels (Excellent/Good/Fair/Poor) without numbers. The Rubric Analyzer needed a lot of prompt iteration to handle all of that.

Getting agents to work together through shared state, handling failures gracefully (what if no golden example is provided?), and making the async checkpoint pause work correctly across API calls... that took significant effort. LangGraph was the right tool for this, but the learning curve was real.

The human review UI also went through several iterations. Too little information and TAs just rubber-stamp. Too much and they're overwhelmed. I landed on showing the score, a short reasoning summary, and expandable evidence sections.

RedInk: A Multi-Agent Grading Assistant