The more you get right in your manuscript before submitting to editors, the less work they'll have to do on mechanical issues—and the more time they can spend on making your writing shine. Plus, learning the conventions of fiction style* can save you money.
*This is Articulate Editing house style, which is based on generally accepted fiction conventions (in the US) and a tiny pinch of personal opinion. Most of what is written here will apply across the board. If a guideline has wiggle room, I'll let you know. And if you're working with a publisher, make sure you follow their specifications as their house style may differ slightly. (This style guide also applies to creative nonfiction.)
When it comes to spelling, most US fiction editors use the Chicago Manual of Style and either Merriam Webster's Collegiate Dictionary or Merriam Webster's Unabridged online dictionary.
But spelling isn't always as cut-and-dried as you may think.
Sometimes the dictionary gives more than one acceptable option (editors usually go with the first option listed in this case). And some fantasy authors like to use the British spelling "leapt" rather than the American spelling "leaped." It's okay for creative writers to make decisions that deviate from the "rules" if they have a good reason to do so. Just make sure you're consistent with whatever spelling you choose.
Make sure to set spell check to the English you're using. If you read a lot, you're probably familiar with both British and American spellings and may not notice a "towards" that should be "toward" (or vice versa). As these aren't words you'd think to look up in the dictionary, spell check is your friend here.
Compound words can be closed, open, or hyphenated.
Check the dictionary. Well-established compounds like the ones above will most likely be in there, and spell check won't catch most misspelled compounds.
For compounds that aren't in the dictionary, get your hands on a copy of the hyphenation guide from the Chicago Manual of Style. It's priceless. And far too detailed to cover in a blog post.
Hyphenation general rules
When compound modifiers come before the noun they modify, hyphenation adds clarity.
Do not hyphenate after an -ly adverb
rapidly-declining grades X
When compound modifiers are predicate adjectives (they follow the noun and a linking verb such as "to be"), hyphenation is usually unnecessary. Even if the compound modifiers are hyphenated in Webster's, like "well-known".
The compounds are well established.
The actress is well known.
Clarity trumps rules every time. If hyphenation clears up ambiguity then by all means hyphenate:
Use multiple hyphens for multiword modifiers, unless they can be left out without ambiguity.
late twentieth-century ideas
When you omit the second part of a hyphenated expression, the hyphen is retained.
line- and copy-editing
fifth-, sixth-, and seventh-grade teachers
But ... not with a single range.
a five- to ten-minute lesson X
a five-to-ten-minute lesson ✓
Contractions and omitted letters
When letters are omitted, use apostrophes to show the omission.
cannot → can't
singing → singin'
it is → 'tis (make sure the curly apostrophe is facing left even though it's in the front)
Some contractions are irregular.
is not, are not, am not → ain't
will not → won't
The articles a and an
Use a in American English before words that begin with consonants and words where the h sound is pronounced.
Use an before vowels and when the h is silent.
With abbreviations, go by the first sound, not the first letter.
an FBI agent (the abbreviation starts with a short e sound: "ef")
a UFO (the abbreviation starts with a consonant y sound: "yoo")
Abbreviations are the shortened form of a word. (Initialisms and acronyms are a subset of abbreviations and are mentioned below.)
Abbreviations are usually spelled out in fiction in both narrative and dialogue.
But social titles are always abbreviated when used with a name. And in American English, take a period.
What should I do, Doctor?
What am I going to do with you, Junior?
The initials AM/PM or a.m. and p.m. are both acceptable in fiction. Use periods with the lowercase abbreviation, but not the uppercase abbreviation. If using the uppercase, small caps are preferable. For more on styling time, see Fiction Style Guide: Numbers.
Initialisms and acronyms, both real and imagined, are common and acceptable in fiction in both narrative and dialogue. They don't necessarily have to be spelled out the first time if the reader's familiar with them, and unlike in nonfiction, you don't need to put the initialism in parentheses after the first spelled-out occurrence of the term. This would take the reader out of the fictional world. For more on how to handle unfamiliar abbreviations in fiction, this article by Carol Saller at Fiction+ is excellent.
Initials for names should be followed by a period and separated by a space, unless two or more initials are used alone.
J. K. Rowling
J. R. R. Tolkien
We call him PJ for short.
Don't use periods with abbreviations that have two or more uppercase letters.
Abbreviations with ampersands don't have spaces around the ampersand.
Johnson & Johnson
Abbreviations for chronological eras are uppercase with no periods. Small caps are preferable.