I agree that the generally-accepted correct comma usage is A. There've been some cases in my own writing where I find sentences that seem to work without the comma after the introductory word, but I've yet to find an exception to the general rule of the Purdue OWL.
Introductory phrases also set the stage for the main action of the sentence, but they are not complete clauses. Phrases don't have both a subject and a verb that are separate from the subject and verb in the main clause of the sentence.
Barking insistently, Smokey got us to throw his ball for him.
(introductory participial phrase, main clause)
This sentence has both an introductory word and an introductory phrase.
Introductory words like however, still, furthermore, and meanwhile create continuity from one sentence to the next.
The coaches reviewed the game strategy. Meanwhile, the athletes trained on the Nautilus equipment.
The only thing I can find about omitting introductory commas comes from here.
Note: A comma is not always needed after short prepositional phrases or subordinate clauses, as long as leaving it out does not cause confusion for the reader. However, using a comma after even a short prepositional phrase or subordinate clause is never wrong, so if in doubt, go ahead and use it.
EDIT: Here's a source focusing on introductory words, or specifically in the case of words like "Finally", conjunctive adverbs.
Often the introductory adverb modifies just the verb, as does the word "often" in this sentence. When that is the case, a comma is usually not necessary, but sometimes the writer may prefer to use the comma for emphasis or to create a dramatic pause. Notice the different effects produced by including or omitting the comma:
Often the introductory adverb modifies just the verb, as does the word "often" in this sentence.
Often, the introductory adverb modifies just the verb, as does the word "often"in this sentence.
There is no unvarying rule for whether or not to use a comma after a conjunctive adverb. Consider the strength or weakness of the conjunctive adverb in relation to the clause that it modifies.
Source
tl;dr Both are technically correct. Since you can omit "Finally" from the sentence completely without changing the meaning, its relation to the 'having' clause is rather weak. Best to set it off with a comma, as in A).