As can be seen from answers to this question, a piece of open source software cannot be licensed in such a way that it is only usable on one specified operating system.
However, can it be licensed in such a way that it can only be used on open source operating systems? Or on operating systems with a licence compatible with the licence of the software in question?
You cannot do this with an open source license. The reason is the same as in the linked question: you cannot discriminate usage, including platform.
Obviously, you can construct a license that specifically requires running on open source operating systems, but this license won't be open source.
Without using the license, it is possible to construct your software in a manner that makes it impractical to run on closed source operating systems, but without a non-free license, there is nothing stopping a determined programmer from adapting it to run on closed source operating systems.
Historically, there was a lot of software that was de facto tied to open source operating systems, because the only desktop operating systems that almost fully implemented Unix/POSIX standards happened to be open source. This is largely not the case today, as standards compliance and embracement of open source by proprietary software has improved across the board, making it easier to write portable software. Chances are good that if your software is useful, someone will port it to closed operating systems anyway.
Not a whole operating system, but:
Some people - the FSF included - claim that the terms of use of the Apple App Store make distribution of GPL-licensed software not legally possible via this store.
If a similar store is used by other platforms with comparable terms, and software installation to the platform is only possible via this store, then you can choose a license - the GNU GPL - to prevent your software to be legally installed on those platforms - and probably make it very hard for people who do not care about this, too.
But you still haven't restricted the software, the software was restricted by the people who wrote the terms of use of that app store.
You can get close.
The GPL is a good example of a license that is widely acknowledged as open source. The GPL prohibits linking with libraries that are distributed under GPL-incompatible licenses. However, the GPL makes an exception for libraries that are distributed as part of an operating system. (That's the short and perhaps not 100% accurate version.)
If you create license that is roughly the same as the GPL, except not containing any exceptions for system libraries, then you've effectively guaranteed that your own software, although still open source, cannot legally be compiled by the typical system compilers on Microsoft Windows, nor on some other closed source operating systems. Or more precisely, that the output of that compilation cannot legally be distributed.
This does not prohibit anyone from running your software in an emulation environment, though, or with non-standard libraries that provide similar functionality as the system libraries. Prohibiting that goes against fundamental principles of both the Open Source and the Free Software movements and as answered already makes your own application non-free. For example, suppose that your license allows running your application under Cygwin, then the Open Source Definition makes it clear in points 9 and 3 that if someone wishes to distribute your application along with a minimal version of Cygwin pre-configured to just work, and you wish to word your license in a way to stop that, then your license is not Open Source.
