The problem you raise (you call it "malicious externalities") is perhaps better known as the extortion or blackmail problem. Medema (2014) reviews the history of this critique.
Those who raised the extortion critique.
an industrial enterprise may find it profitable to adopt processes which generate much waste in order to be able to accept payment for reducing discharge.
any activity can be turned to profit as long as it is sufficiently annoying to someone else.
The administrative authority would have to stand ready to make payments to industrial plants which never do locate in the area but which would do so if a payment reflecting the costs their effluents would impose were not made. Payments on this basis would of course be an open invitation to extortion.
If external diseconomies against others can be expected to lead to bribes by victims to desist, then the production of negative externalities becomes a valid by-product of primary production. Profitability is enhanced whenever any firm can select from among its inputs and/or output alternatives those which cause substantial damage to third parties. Resource use will tend to become specialized toward much-augmented third-party interference. The new legal industry of selling protection against disturbance will be highly profitable.
Once it became apparent that polluters could extract payments for pollution reduction, there would be an incentive for entrepreneurs to threaten to undertake production activities with external pollution costs simply to be bribed not to undertake them.
Mumey (1971) makes the distinct point that contrary to some defenses of Coase against the extortion problem (below), in a frictionless world with costless communication, the problem of extortion or "malicious damages" is actually exacerbated.
Daly (1974) makes another distinct point:
the simultaneous existence of competition and of small numbers of externally affected and effecting parties are mutually inconsistent phenomena. If the large numbers generally associated with competition are present there will exist the acknowledged prohibitive transactions costs. In the absence of such competition, however, market solutions are burdened with complex, game theoretic bargaining situations which involve a substantial likelihood of counterproductive (costly) activity and for which the achievement of efficient outcomes cannot be assured.
See also Daly and Giertz (1975, 1978).
Those who defended against the extortion critique.
Under competition the price of a good, or of an agreement, is held to its cost. ... In the competitive regime, rivalry between “extortionists” would push to zero the price they receive not to undertake these excessive activity levels. Thus, competing “extortionists” would offer to refrain from extortionist activities for smaller, and ultimately for zero, payment.
Also:
The concept of extortion is legal, not economic. All transactions are transfers of rights involving exactly what Shoup has called extortion with the possible exception of perfectly competitive ones, whatever they are. All transactions are preceded by attempts to gain information about the other party’s reservation price. A variety of techniques are used to do this. From the viewpoint of economic analysis, none of these differ. The use of the legal term “extortion” in economic analysis is likely to result in confusion and to hide implicit value judgments. There are some wealth transfer activities which, as citizens, we would seek to discourage through legal arrangements. But until we have formed a clearer notion of the criterion to be used to separate these from other “approved” wealth distributing activities, the use of terms such as “extortion” in economic discussions is laden with difficulties. Thus, if without any more valuable alternatives, Professor Shoup were to threaten to leave the University of Michigan unless he were paid more, thus threatening to impose real costs on that institution by depriving it of all the advantages of employing him, would that constitute an analytical use of the word “extortion”?
See also Demsetz (1972, 1978).
- Zerbe (1971): extortion "does not occur ... because of high transactions costs, symmetrical possibilities of extortion [One man’s threat to raise chickens is countered by his neighbor’s threat to raise pigs], and the operation of the legal system in holding certain forms of extortion illegal [With costless bargaining, undoubtedly my neighbors would pay more for me not to grow pigs than I could make growing pigs. Hence, the law forbids my raising pigs in Chicago and in most cities]."
Zerbe also raises a separate point similar to Demsetz's second point:
Extortion does occur, however, and it is not necessarily undesirable. The general economic rule governing extortion would seem to be that extortion rights should go to those who are willing to pay the most. This rule illumines the close relationship between extortion and market transaction. Indeed it is difficult to separate extortion from other production activities. Payments required to attract factors of production are also payments to ensure that these factors will not do something else. Consider Shoup’s case of the polluting oil company that threatens towns A, B, and C with location in them. This situation is similar to the case of a desirable company, say a computer company, also choosing among towns A, B, and C. The computer company chooses the town that offers the deal with the least amount of extortion money. In this way both end up in the right location. In this sense extortion is a method for achieving results and may or may not be desirable, depending on what other methods are available and whether or not extortion interferes with more desirable methods. In actuality, towns generally offer companies tax and other incentives and disincentives that reflect the town’s desire for the company. Here the cases become wholly symmetrical. Both the oil and computer companies choose the location for which the location price, ceteris paribus, is the least. At some price, the oil company becomes attractive to a town. It is not surprising then that there is a problem in distinguishing between legal and illegal extortion, and at this point an extensive legal-economic discussion of extortion is called for but is provided neither by this note, by Shoup, nor by my original article.
- Jaffe (1975): With zero transaction costs, we will always arrive at the most efficient allocation:
imagine that B can increase the destructive potential of his resource by physical investment. We further assume that the degree of destructive potential is an increasing function of physical investment. This function is known to both A and B.5 As B can now increase the potential of his resource, he may be able to exact an even greater bribe from A. However, applying the same rationality assumption used by Coase, no additional investment in the resource will actually be undertaken by B. Merely B's potential to create a more destructive resource should be enough to force A into a more generous bargain.
When I referred to the problem of blackmail in "The Problem of Social Cost," it was simply to show that, however rights were defined, there would always be opportunities for this type of blackmail. I pointed this out in the section of the article in which it was assumed that transactions were costless and in which therefore no resources were absorbed in bargaining. In such a world, it is not clear what the objection would be to this type of blackmail. The position is, however, very different if we make the realistic assumption that transaction costs are positive. It is obviously undesirable that resources should be devoted to bargaining which produces a situation no better than it was previously.
There is no such thing as the Coase Theorem.
Or as Coase stated repeatedly, including in this 2012 interview:
"It's not about my work at all!"
Coase's goal was to illustrate that:
institutions matter, and that understanding how and why they matter, and the implications of this for economic analysis and policy may require a re-orientation of the ways in which economists practiced their trade. Here, too, transaction costs occupy center stage. (Medema, 2014.)
Most of all, his goal was to direct economists' attention to the world of positive transaction costs.†
Ironically, economists became fixated with the simplistic world of zero transaction costs, where they could churn out a variety of clever "Coase Theorems" and avoid having to study the messier real world of positive transaction costs.
What I — and probably also Coase, Medema, and McCloskey — would say is this:
The extortion problem is worthy of our attention and analysis. However, we should study it not in the context of the economist's utopian world of zero transaction costs, but in the context of Coase's very real world of positive transaction costs.
We should study how and why, given the very real fact of transaction costs, institutions ameliorate (or fail to ameliorate) the extortion or blackmail problem. I think Coase (1988), Zerbe (1971), and Demsetz (1971) went some way towards initiating this discussion, but there is much more to be done.
† D. McCloskey has also repeatedly made the same point and stated the Coase Theorem (1985, p. 336) as:
In the presence of transaction costs the location of a pollution tax or of other liability for damages does matter for efficiency.
