We all have an intuitive idea of what a genre is. Science Fiction, Fantasy, Mystery... these are all examples of genres. But this intuitive understanding doesn't seem to easily translate into coherent definitions. It's easy to list books that qualify as Science Fiction; it's harder to explain why, and it's even harder to find a coherent definition of Science Fiction that everyone can agree on.
So what are genres? Are they a useful concept for understanding literature?
Let's start with what genres aren't. When people debate the definition of say, Science Fiction, what makes these debates possible is the belief that an objective, correct definition of Science Fiction exists. However, the assumption that genres have objective, coherent definitions doesn't hold up to even the most cursory of scrutiny.
Works can be assigned to multiple genres. Radiance is a decopunk pulp SF alt-history space opera mystery. (Hat tip to BESW for the example). Debates about what genre a book belongs to can only be resolved with the recognition that stories can have multiple genres. Genres are not distinct categories, and genres are not mutually exclusive; genres merge, overlap, and split.
Genres often have little to do with the actual content of a text, and often have everything to do with how readers of a text perceive themselves. Harry Potter is a perfect example of this. Originally marketed as children's literature, the British publisher published "adult editions" of the Harry potter books. The adult editions had the same text but a different cover. It's not hard to read the subtext that some adult readers were not comfortable reading books identified as being for children. And it's not hard to read into the debate over whether Harry Potter is children's literature or adult literature a debate, not about the content of the books, but about who should be reading Harry Potter.
Genres are not fixed in time; any definition of a genre will eventually become out of date. Let's return to the example of Harry Potter. Research suggests that due to the popularity of Harry Potter, the length of children's literature books has increased substantially.
Genres do not merely describe literature; in many cases the existence of a genre is what causes the creation of works of literature. The example of "solarpunk" is highly instructive. (Hat tip to BESW for this example). The genre of solarpunk was literally created when someone wrote a tumblr post proposing the creation of a new genre. Since then, the existence of the genre of solarpunk has caused the creation of new stories.
What all of these case studies suggest is that genres are not objective, descriptive, fixed, and coherent, but are subjective, fluid, and are as much prescriptive as they are descriptive. With this newfound knowledge, let's move on to some actual definitions of what genres are.
If we look at how genres are actually used, genres essentially represent communities of readers. If a book is labeled with a genre, this serves as a signpost for the community of readers that said book is a book that they should read. These communities are both real, in that they have conventions and clubs and websites, and imagined, in that readers of a genre may not have interacted with other readers but still feel that they are a part of the community.
Genres are about marketing. They are labels used by marketers to appeal to readers. Genres are created, defined, and redefined by a complex interplay between marketers and readers. There are too many examples to count of publishers reclassifying books in response to reader pressure, or of readers disagreeing with the classifications used by publishers. It's not a top-down model.
To answer the last part of the question, genres are a powerful force in the world of literature, and the study of genre is useful, necessary, and will lead to important insights. This answer has given many examples of how genres shape literature. But in order for genre to be studied productively, it's important to understand what genre is or is not. There can be no productive study of genre with the assumption that genres are objective, fixed, coherent categories. That will only lead to pointless debates about the classification of books into genre, at the expense of understanding how genre actually effects the world of literature.
I would argue that a genre is a kind of non-spoken contract between creator and reader, an implicit understanding of some of the things that has to be accepted to make the work function as intended.
One of the most important such understanding when it comes to literature is that if something is a labelled "fiction", then it need not have anything to do with reality. Many genres of fiction ascribe further to a certain realism, where it is deemed important that most or all events contained in the work are realistic, and most try for at least a level of psychological realism, where the characters are supposed to act in a believable manner. However, different genres can play more or less loosely with the realism, sometimes replacing it with a set of genre conventions; for example, in most detective fiction, it is expected that some sort of justice is ultimately served, while some science fiction subgenres relax the laws of physics as we know them so that interstellar travel is possible.
Other genres instead are defined mainly by the form that the work takes: poetry places further constraints on language, where the lines classically has had to follow some sort of pattern, but where instead word order or word choices are freer than "ordinary" language.
Of course, genres are not rigid things: it is, for example, possible to write detective fiction where justice is not served. Challenging the conventions has always been a way to create new, interesting works. Even if a creator has intended to place his creation firmly within a certain genre, readers can also derive enjoyment through treating it as if it belonged to another: an excellent example of this is the sherlockians who treats Arthur Conan Doyles stories as "real" retellings of the adventures of an historical figure.
Thus, genres are useful shortcuts that lets us dispense with certain questions about the realism of certain things (such as "how comes this person always stumbles into murder mysteries with unnaturally clever attempts to cover the tracks?"), and instead analyse the qualities of a work with the genre conventions taken as a a given.
When you want to read something, how do you make your choice? When you go into a library, where do you go first? When you go to the movies, what do you want to see?
When we read for recreation, we choose what to read based on the mood we're in. For sentiment, we choose romance. For stimulation, we choose adventure. For fun, we choose humor. For puzzling, we choose mystery. Every story has a theme, and we want to feel it.
Yes, there are subgenres and shades of emotion. Some adventure stories turn into horror. Some humor turns into romance. In the end, though, we get what we came for. If we don't get that satisfaction, that catharsis, we feel cheated.
At its most basic, genre boils down to tragedy and comedy. Will things turn out well for the characters, or do some of them have to die? Will we feel happy or sad when the curtain goes down?
Genre is about evoking an emotion. An abandoned house, for instance, triggers dread and not joy, while a brightly lit garden triggers joy and not dread. Writers use certain tropes to condition the reader, to influence the reader's emotions. Twisting or inverting tropes doesn't invalidate them, it only changes the perspective on them. Tropes have power. That's what makes them tropes.
When we read for criticism, we analyze. We look at themes (among other things). In a short story, there may be only one emotion. In longer works, the emotion may shift or alternate. As a basic story element, theme is as much a part of literature as plot or character. Genre helps to establish the theme, so it is important to analysis, albeit in a secondary role.
