1

I've may have reached a point in my neural network research that I cannot continue without significant financial investment.

I am using neuroevolution to evolve a network on the EMNIST data set. It is embarrassingly parallel (100 individuals * 116,323 records). The program is impossible to accelerate with a GPU. I have tried. This leaves OpenMP, which to my surprise is granting a speedup near-equal to the number of cores. However, even a 72-core Virtual Machine is projected to take 34 days (200 iterations, minimum), and set me back >$2,400.

MPI is on my list of changes to make to the program, but (assuming seamless parallelization) the cores and time required will not change. Several combinations of virtual machines and core-counts (Amazon, Azure, Google) have revealed a cost estimate of $1,500-$2,500. For 1 experiment. Assuming that ALL settings are correct, and I do not need to run further experiments.

I am a sophomore-level undergraduate. My university has NO computing clusters, nor access to any. The best they have are 8-core workstations (that would take a year to run the job). I don't have $1,500 to spare. Are there any resources (sponsors, grants, laboratories, etc) that would help me get the cores I need (cloud, or "home supercomputer")?

Anton Menshov
  • 8,672
  • 7
  • 38
  • 94
  • Do you have an advisor, are you part of a research group, or is this your own independent study? –  Feb 26 '18 at 04:54
  • I have an advisor, but this is ~99% me. –  Feb 26 '18 at 04:55
  • 7
    Have you talked to him about this? If he has contacts or even a second appointment at another university it may be possible to access some of their resources less formally, but otherwise you probably have to write up some kind of proposal. –  Feb 26 '18 at 05:00
  • In any case, the vast majority of funding opportunities would have to go through your advisor, or at least require their recommendation. So I agree that involving your advisor is the first step. –  Feb 26 '18 at 05:01
  • Have you tried searching to see if any existing supercomputers allow free usage? I seem to remember reading something about IBM doing that. –  Feb 26 '18 at 09:00
  • 1
    Do you need to SHOUT at us? –  Feb 26 '18 at 10:19
  • If you are referring to my capitalization of my inability to use a GPU, I have asked about how to accelerate this program before. Every time, someone suggests CUDA, and that I make changes to how my program represents and evaluates netwroks. I have tried. For nearly 2 months. The data structures necessary for my program need to be variable in size, multi-dimensional, and have too much branching. GPU acceleration is impossible. –  Feb 26 '18 at 12:11
  • I am transferring this question to Computational Science.SE. It's a better fit than for this site. – aeismail Feb 27 '18 at 23:32
  • 1
    The big problem I see here is that each experiment is expected to consume almost 60k CPU-hours. On Stampede2, 10 repeats of this experiment would consume roughly 10,000 SUs, or "a pretty good starter allocation." In other words, unless you only need to run one or two runs of this program, you're very unlikely to find anyone with those kinds of resources lying around for free. I was going to point to the Jetstream Cloud trial allocation, but those are pitifully small, unfortunately. – chipbuster Feb 27 '18 at 23:52

1 Answers1

5

You need funding, you need a better algorithm, a better implementation, or you need to change your problem. To get funding, you need to work with your adviser. This could be about using grant money or getting a grant for an XSEDE allocation. If your adviser has neither of those, it may not be a good choice to work with them even if you like their research: science requires funding.

If you want to make it cheaper, then think about a different algorithm. Even if it's doing a lot of branching and stuff like that, there must be some way to utilize linear algebra operations in some parts to make use of GPU (or Xeon Phi) acceleration. Maybe you need to precompute some values, or need a wholly different approach. Only you (and your adviser) would know. In fact, this algorithm development is many times the entire purpose of the research. Your adviser may have given you this project because no simple computer can solve this problem. Your goal isn't to find a bigger computer, it's to find out how to make your computer solve this problem (or at least, find out how to solve it better). No online tutorial or anything can help you with this: it's real research.

You also need to take a look at your implementation. If you're looping and branching in a high level language like Python, this could be the problem. You either need to re-write the code to be vectorized to do a bit better, or to get "production-quality" speeds you need to re-write it in a non-allocating format in a performance language (one which generate strictly typed and interprocedurally optimized compiled code) like C++ or Julia. It's not hard to find non-vectorized Python codes which can be made 1000x faster just by doing a better implementation and being careful with memory use (even vectorization is bad with memory use so that still leaves a lot to optimize).

And lastly, if this is a highly researched area where it's well-known it cannot do any better and you cannot get the funds/allocations to run the algorithms, pick a different project. Sometimes problems currently aren't possible. I highly doubt that your adviser would give you a problem that is in this domain though given that you're an undergraduate.

However, given you're a younger undergraduate, I would double/triple/quadruple check that you are actually correct that this kind of computing power is even required before going to blame anyone else. Embarrassingly parallel and (100 individuals * 116,323 records) is quite small, even to train on a desktop computer without a GPU. You should really make sure everything is as fast as it can be (rewrite it a few times in a few different languages, benchmark and profile each part, get rid of all memory allocations in the inner loops, etc.), make sure there's no better algorithm choices, make sure there is no cluster allocation you can get on, and only then should you consider moving to a new project that is more suitable to your current access to computing power.

Chris Rackauckas
  • 12,320
  • 1
  • 42
  • 68
  • 4
    I have often thought I needed to do compute on a huge dataset, got frustrated with how long it was taking, then found out ~10 seconds of compute on a subset gave essentially the same results as 10 hours. I suspect my experience is not unique. – user14717 Feb 28 '18 at 06:02
  • There are two things I need to point out. [1] I proposed this research project, and my adviser has (at times) trouble following it. [2] The program is written in C++; the problem is that the program is founded on multi-dimensional, variable-size STL vectors that cannot be addressed for use on a GPU. However, I am working on ways to serialize the network to an array at run-time. Work-in-progress, but that is another concern for another site. –  Mar 06 '18 at 03:32