In linking two branches of my family a male sibling relationship in 1705 is somewhat questionable.
Following male lines back to that period, through seven generations, can DNA analysis be reliably used to establish a positive link?
If so, what specific test would give the greatest level of certainty?
Yes, following male-only lines with Y-DNA or female-only lines with mtDNA can establish a positive link going back 7 generations and even more.
Both Y-DNA and mt-DNA passes directly from parent to child without recombination, so that DNA basically passes intact with the exception of a few mutations which happen on average once every few generations. So the connection of two family branches using two direct male-line descendants can be reliably confirmed with Y-DNA tests or female-line descendants with mtDNA tests.
This is not true for autosomal DNA analysis where each parent contributes half to the child. Sixth cousins will have very little, and often zero autosomal DNA in common. So you will not be able to reliably establish a relationship that distant with an autosomal test.
Your problem is exactly what genealogists today mostly use Y-DNA and mtDNA tests for, that is, to establish the connection between two lines. Other than that, they are really only good for ancient genealogy studies.
I think it's possible to make this determination with autosomal DNA, it's just statistically extremely (more like overwhelmingly) difficult.
Even with 4th cousins, there's only about a 50% chance that any two at random share any significant DNA at all. So, if you test and don't match, it doesn't necessarily mean there's no relationship. If you compare your sample against 2 actual 4th cousins, there's a 25% chance of no match, against 4 a 12.5% chance of no match, and against 8 a 1.5% chance of no match. So, you would probably need to compare against about 6 possible 4th cousins to be reasonably certain (97%) that no match indicates no relationship (this presumes that you know with some certainty that those 6 are closely related to each other).
Obviously, the numbers get a lot worse with 7th cousins. The 50% likelihood of a 4th cousin match falls to 1-3% for a 7th cousin match. That translates to optimistically on the order of 100-200 failed matches against possible 7th cousins to be reasonably certain of no relationship. The converse is also true - if you have 100-200 samples in the "known" branch of the tree, and a suspected 7th cousin matches none of them, then most likely there's no biological relationship (I'm ignoring the NPE factor here, but you'd also need a lot of samples in both lines to rule that out).
Even if you do luck out and get a match, you have the problem of pedigree collapse - a match between two individuals, one in the "known" line and one in the "target" line, doesn't provide absolute certainty that your proposed common ancestor is the source - it's possible that there's another common ancestor somewhere in the mix (often typical with ancestral groups who migrated together and/or lived in close proximity for generations, attended the same church, etc.). To rule out (or at least reduce the likelihood) of that possibility, you'd again need to have samples spread out across multiple descendant lines on both sides. I can't guess in your case, but on AncestryDNA (presumably the largest sample database in the industry), my largest "ancestor group" has about 20 members, with perhaps 2 or 3 times that many who are shared matches with one or more of them. In many of these cases, there's more than one common ancestor involved, so it's difficult to draw any conclusions about shared matches who have no listed pedigree.
A chromosome browser and the corresponding ability to map specific DNA segments to specific ancestral lines would obviously help considerably, but it doesn't change the statistics all that much.
As other answers have mentioned, Y-DNA seems to be the way to go with your case, since the seventh cousins you are comparing are – hopefully – descended from two brothers.
DNA is pretty good at proving a negative (i.e. likely no relationship), but you can never be 100% sure that a positive match means a certain relationship.
Keep in mind that all Y-DNA tests are not equal. For example, a 12 marker Y-DNA test would be good for ruling out that you were cousins in the male line. A 9/12 match would be strongly suggestive that you do not share a common male ancestor within the last 7 or 8 generations. A 12/12 match could mean you share a common ancestor in the last 4 generations or 20 generations, there are just not enough markers to make any definitive conclusions.
The more markers you test, the more confident you can be that a positive match is a true positive. Ralph Taylor has written a nice summary of Probabilities in DNA. The data might be a little bit dated now, but it gives a good idea of the relative genealogical "strength" of the levels of Y-DNA testing. For example, for 67 marker tests matching at 67/67 markers, according to Taylor's data you can be 90% confident that your most recent common ancestor is as few as 4 generations ago, and 95% confident that it is as few as 6 generations ago.
With Y-DNA it is also worth keeping in mind that non-paternal events were fairly common. A mismatch on Y-DNA could just indicate that somewhere along one of your paternal lines someone wasn't the father as stated in paper records. So you might be right about the paper genealogy suggesting you and your cousin are descended from brothers, but due to no fault in your research this just may not match the genetic genealogy.
My answer would be: "It depends." What information to you hope to gain from the DNA test, and what role does that information play in your proof argument?
One advantage of joining a surname project: you have an opportunity to find more matches from descendants of those individuals. With a larger group of collaborators, you can get more data, both from the DNA itself, and from having more people to comb genealogical periodicals for case studies that might bear on your problem, and to look through the catalogs of archives and repositories for lesser-used record groups.
If you don't have a clear idea yet exactly what you hope to gain from the DNA test, take a step back and see how other genealogists make their cases. Some articles and presentations to consider are in the links below.
