A few topics in security benefit from being closed-source.
This is not the general case. In cryptography and most of security, Kerckhoff's principle applies. A system should be secure even if its design is fully public, because the adversary can usually learn the system's design anyway. In cryptography, secrecy is concentrated in a very small part: a key, a password, a nonce… The operations that are performed on these secret values are well-known.
Most of security benefits from being open-source. For example, all recommended cryptographic algorithms are public and have undergone years of scrutiny by experts. For software, note that the benefit of being open source only comes into play if someone actually does review the software. This is rarely done because it's expensive. I invite you to read threads on this topic on Security Stack Exchange, such as Open Source vs Closed Source Systems, Security related code in Open Source projects?, How are open-sourced operating systems and software kept secure?, etc.
Nonetheless, there are a few specialized domains where closed source is an advantage for security. These are domains where in order to obtain information about the system design, an adversary has to conduct active attacks that run the risk of detection. Generally, in such domains, the design of the system evolves, so while adversaries can eventually gain the knowledge that they seek, by the time they manage to do so, that knowledge becomes obsolete.
For example, Stack Exchange has measures to combat spam and vote fraud. These measures would not work as well if the detail of how they work was public. For example, it's known that there are filters to block spam, but the details of what makes the system decide that a new user is a spammer and isn't allowed to post are only known to a few Stack Exchange employees. Similarly, it's well-known that there is an automated process to cancel votes between sockpuppets of the same person, but the details of how this works is not public, it is not even known to moderators. In order to learn how to get through, spammers and fraudsters need to carry active probes where they try to post things and see what sticks and what gets shut down. All this probing leaves tracks, which help the defenders make the system more resistant and identity the attackers.
Another example is some of the security countermeasures on smart cards. Smart cards have to maintain secrecy in a hostile environment where the attacker has physical control over the card (e.g. a credit card thief). They are potentially vulnerable to many types of attacks, including side channel leakage and fault injection. Some of the protections on a smart card involve code that erases confidential data if it detects an attack (as well as physical protections such as dissolving part of the card if someone tries to drill a hole in it). These countermeasures work better if the attacker doesn't know exactly what triggers them, so that they'll need to brick many cards before they figure it out. Eventually they will figure it out, but if it takes them a few years and the cards using this technology have expired, the defenders have won.
Both examples are a permanent arms race. Closed source is an advantage only temporarily, for a few years at most. Once ways to break the system are known, the whole design might as well be published. But making the design of these particular subsystems open from the get go would facilitate attacks. Closed source is good to protect a few tricks with expected obsolescence. This doesn't necessarily mean that such systems can't be open-source, but an open-source system would have to be extremely good to compete.
