Let's look at some color names: red, scarlet, crimson, carmine, for instance. (These aren't the same shade of red, but for the moment that doesn't matter--we're considering what other meanings attach to these words for many readers.) I'm not going to cite all the connections for each word, but realize that every time someone has seen a word in print, or heard a word spoken, new associations for that reader/listener are formed or strengthened. Thus every previous writer has helped shape how readers "hear" what you say. If the word was used in well-known works, an old association (from the Bible, from Shakespeare) may have more influence than you'd expect, if you're a stark post-modernist. Thus depth of reading--familiarity with, awareness of--the works most educated people are likely to have read will help you in your search for the best word in a given circumstance.
Red: A friend of mine was scolded in grade school for wearing a red dress on Valentine's day because to her teacher, red meant Red: Communist. A school in this county tried to bar all students from wearing anything red or blue, because some gangs used "colors" and the school wanted to avoid gang action in the schools (a laudable goal but what were they going to do about the American flag? And what about blue jeans?) It's hard for a woman to wear a red hat these days without someone asking if she's a member of the Red Hat club. Politically, someone decided to consider some states "red" and some "blue" (Republican states are "red", which is really odd when you consider the 1950s/1960s concern about "the Reds," Communists. Internationally, a "Red state" is a Marxist government; nationally, a "Red state" votes Republican.) Red is also associated with blood, with anger, and--via both blood and anger--with violence. "Devil" costumes are often red, and you find devils in early religious paintings shown as red. Another traditional meaning of the color was courage (again, via blood, and also The Red Badge of Courage.) Red hair is associated with a hot temper. Red paint is associated with barns, and thus with traditional rural culture and values. Red chestnut, as a horse color, is associated with racehorses (Man o' War, Secretariat, "Big Red") Red is also associated with Valentine's Day, with hearts, and thus red roses (and to some lesser extent other red flowers) are associated with romantic love. Then there's "red-shirting" (in two different but similar associations.)
In some settings, red isn't much affected by its associations. It's a very common, old, standard word, useful enough that it'll be used widely no matter what. You can write about a red apple, or a red vase or a red chair, and of course a red rose or red geranium, and readers will simply visualize the central color "red" attached to that object. But if you say a red *book*, this will bring up specific books--for fantasy readers, the famous Red Book in Wales, and for twitchy politicians, Mao's Little Red Book. If your character wears a red shirt, some readers will immediately think of the college football tactic of "red-shirting" some players, and others will immediately think of Star Trek (where, it's said, wearing a red shirt on an away team is a death sentence.)
Scarlet. Scarlet is forever linked to Scarlett O'Hara in Gone with the Wind. Thus the color name alone is associated with passion, tempestuous temperament, drama, and conflict. The actual color is a bright, "hot" red, which enhances the associational meaning for those who know what the color is. Those who aren't color-conscious will nonetheless imagine a very bright red because of the associations...nobody imagines a dull scarlet. It's important to realize that readers don't need to have read a book or seen the movie to be affected by associational meanings that derived from those and other media....people who have never read The Red Badge of Courage, never read or seen Gone with the Wind, still have some associations to those stories in their response to the words red and scarlet. Celebrities who were named--or chose the name--Scarlett have contributed to the associations to both the name and the color word. "Scarlet" used to describe any item of women's clothing will lead some readers to think of that character as somewhat like "Scarlett." (This isn't all bad, of course. If you want to drop a hint that mousy Miss Millicent is really not that mousy on the inside, a scarlet lace teddy under that high-necked gray frock is all it takes...and a scarlet toadstool in the garden will seem more magical or more sinister than a common white one.)
Crimson. Crimson associates with the universities and athletic teams that have chosen crimson for a color. This is a powerful additional association (both to the specific universities, and everything *their* names evoke, and to masculine sporting images. So Harvard's social and intellectual reputation and Alabama's "Crimson Tide"are both part of crimson's assocations. But crimson has been a favorite literary word for centuries, so there are associational links to Shakespeare (among many others) where crimson is used for the color of blood. Crimson is more associated with blood than scarlet, for instance, because the blood you see in most wounds is more crimson: only arterial blood is scarlet. So crimson has some association with autumn (crimson leaves), with death, with darkness. (Whether crimson is a sad word or a not depends on the words you put around it...it can go either way, and it's always a strong word.) Unlike red, which has many negative associations, and scarlet and carmine, which are dominated by character associations, crimson's multiple associations are easier to handle.
Carmine. Carmine, like scarlet, gets its main association from a fictional character, in this case Carmen-of-the-opera-Carmen, but with shadings from Carmen Miranda. It's actually a dark, purplish red (obtained from cochineal) but you sometimes see it labeling lipsticks that are a hot peach-red, undoubtedly by association. Like Scarlett, the character Carmen is hot-tempered, flamboyant, flirtatious, dramatic...all of which play into its reading by people who may or may not have ever seen the opera or the dancer. Since it's less common than the other three, it's more affected by this single associational link--to women, fictional or real, named Carmen. So, if you need to use that color in a masculine context (color of a tie, of furnishing in a man's room), the chose of carmine is going to create certain associations you may or may not want.
Note: associations vary from region to region, and country to country, and language to language, along with the positive/negative connotations of those associations.
Awareness of the shading that associational meanings give words in specific contexts can help you use those associations to enrich your writing. As
stinabat said in a comment on an earlier post, writers like Charlotte Bronte and other great writers were fully aware of all the meanings (associational and formal) of the words they used and deliberately evoked that range for alert readers. Political writers and advertising writers are adept at choosing language that evokes exactly the response they want, by researching how specific words are understood in their "market," but it does not have to be that crass...you can use the knowledge, as Bronte and Shakespeare and many later writers have done, to enrich your readers' experience of your work.