posts about mistakes rss

Applying Aider-like Grammar Corrections to a Novel

Are you part of the vanishingly small Venn diagram of people who are writing a novel or other long-form text, like writing in markdown, don't like existing grammar-correction tools, are comfortable writing code, and aren't sick of blog posts about AI? I'm writing a fantasy novel, and I wanted to do a round of semi-automated grammar corrections on my rough draft before doing my own pass of revisions.

I started with aider's diff edit format for the LLM to document its suggested changes:

<<<<<<< SEARCH
{original}
======= RATIONALE = {rationale}
{replacement}
>>>>>>> REPLACE

Once I had the suggested corrections, I could go through them one at a time and either accept, reject, or edit them. When I accepted a suggestion, my script would replace that copy with the corrected version.

Screenshot of a terminal showing the script's UI. It has four components: Original, Suggestion, Rationale, and options to accept/reject. The original text on one example is "it found correction specific to my novel." and the suggestion is "it found corrections specific to my novel." (with the added "s" on "corrections" highlighted). The rationale is "Pluralize 'correction' to 'corrections' to agree with 'found'." The options are "Accept," "Reject," or "Custom"

Here's my current prompt:

Suggest GRAMMAR AND SPELLING CORRECTIONS in the format:

<<<<<<< SEARCH
{original}
======= RATIONALE = {rationale}
{replacement}
>>>>>>> REPLACE
        
This is from a chapter in a fantasy novel, so there may be unfamiliar terms and slang. ONLY LOOK FOR GRAMMAR CORRECTIONS.

- Slang includes phrases like "blighted," "cankered," "muddy," and "misty." These terms do not need to be corrected. 
- The Mist, Twisted, and Storm are capitalized when referring to the magical phenomena.
- The Heart, Roots, and Trunk are capitalized when referring to those parts of the Tree.
- "The" does not need to be capitalized when part of names like "the Great Staircase" or "the Conclave"
- There are mistakes where a character will incorrectly call a character "sir" when they should be called "ma'am." Colonel Merrin is a woman. If you are ever unsure of an officer's gender with "sir," flag it as a possible correction and I will fix it manually
- When using speech tags like "said," "announced," or "yelled," be sure to use a comma rather than a period. '"Hello," she said' and NOT '"Hello." she said'
- ONLY include the original text as-is. Don't include "..." or any other indicators of past text. The {original} text will be used in a find-and-replace

ONLY INCLUDE GRAMMAR CORRECTIONS. DO NOT INCLUDE STYLE CORRECTIONS.

I used Cursor to create the script to hit OpenAI and iterate through the suggested corrections. This is exactly the kind of script that AI excels at generating: it's a one-off script in a well-known language that doesn't need to integrate with an existing codebase.

It also reinforced how important it was to have a technical person in the loop:

Main takeaways:

o4-mini did shockingly well! In addition to identifying problems like comma-splices and misused dialogue tags, it found corrections specific to my novel. Things like "bone wyrm needs a hyphen to match its usage elsewhere," "Ithilia is misspelled," and "'twigs-worth' should be 'twig's-worth.'" It's worth calling out that none of those corrections were documented in my prompt and it was able to figure out the correction from context.

It also did a fantastic job at accurately reproducing the SEARCH portion for the replacement. There were relatively few times when a small mismatch meant that it wasn't able to apply a correction.

It did a good job of finding and flagging "opinionated" grammar corrections that tools like Google Docs won't flag as errors. A common example is dialogue tags; one of my most common mistakes was ending dialogue before a dialogue tag with a period rather than a comma:

"Hello." He said.

Google Docs thinks this is perfectly fine, and I honestly don't notice the difference when reading, but I want to have my dialogue end with a comma:

"Hello," he said.

This is the kind of thing that existing tools could catch, and I didn't look around at all to see if Grammarly or something similar would have worked. Most of my errors were the sorts of things that simpler tools would have caught—writing words words twice, adding an extra space between words, forgetting a period or comma or misspelling "taut" as "taught." [sic]

Most of the suggested corrections were correct, but there were quite a few misses in there too. It wanted to change "loose arrows" to "lose arrows," it kept wanting to refer to a character named Barkle as "Sir Barkle" to "show respect,"[1] and it kept trying to add closing quotes to multi-paragraph dialogue tags. I didn't track this, but I think I rejected or rewrote ~1/20 of the suggestions. It was a low enough error rate that it was a bit hard to focus.

Aside from incorrect suggestions, it also missed a decent number of translations. When I went back over the corrected chapters myself, I was still able to find mistakes—mostly incorrectly conjugated verbs and misused dialogue tags. The review was useful, but was far from quality of a copy-editor.

o4-mini initially didn't restrict itself to grammar corrections until I STRONGLY STRESSED that grammar corrections were all I wanted. Aside from wanting to feel like the author of my own work, I disliked the stylistic suggestions it came up. Its suggestions would try to correct text that was intentionally written to be choppy or long or have an odd affect.

Finally, o4-mini did much better with these grammar corrections than the GPT-4.1 models I tested. If you run GPT-4.1 models against this blog-post, they flag things like the intentionally incorrect "Hello." He said. that's added as an example, while not catching the doubled "words words" or the extra space I added in the paragraph above.

The total cost to review my book was under $2, which felt pretty worth it to me to be able to spend less time correcting dialogue tags and misspellings, and more time focused on improving the actual contents.

I'm sure o4-mini and my script missed things! I'm willing to bet that there are plenty of grammar errors still lurking in this post that an experienced copy-editor would be able to track down.

Should you write a script like this?

Should you use a script like this? Maybe? It was pretty easy to write a script that did exactly what I wanted. It's surely worse than other tools out there, but having a tool that does exactly what you want and nothing more or less is quite nice.

One takeaway for me is that AI is now good enough at following suggested output formats that the technique of generating potential diffs to apply to a file is worth exploring for bulk edits.

  1. It was weirdly insistent about this one, and I have no clue why it cared so much about Barkle's missing title. ↩︎