3

I want to stop immediately an MPI program, that has N processors and each processor runs a different algorithm (with different computational complexities). Processors share information to each other using an unidirectional token ring topology. As soon as one processor ends it should notify its neighbor and all processors should stop immediately as soon as they receive the interruption. Here is an sketch of the execution program at each processor:

1. Processor_i(array of n data) {
2.   while (stopping condition is not satisfied) {
3.     Run algorithm_i(data) for the ith processor
4.     Receive information from its source node (at most read 2 messages)
5.     if (the incomming message indicates interruption)
6.       break;
7.     Send its own information to its destiny node (maximum 2 messages)
8.   }
9.   Save local solution
10.  Send interruption message to its destiny node
11. }

Notice that in the step 3, the computational complexity may be, for example, O(n^2), O(n log n) or O(n^3) for processors 1, 2 and 3, respectively.

My problem is that the messages are buffering by MPI and the interruption message is not reading immediately. I was thinking in writing a file with the stopping instruction, but the filesystem is not shared.

How could I deal with this problem?

user5244097
  • 31
  • 2
  • How quick does "immediate" have to be? MPI isn't buffering messages "until it has enough data to send around", but (typically) copies data into its own data structures before sending them out. That can't take more than a few dozen nanoseconds, though. So I'm not sure what your problem is, exactly. – Wolfgang Bangerth Aug 26 '15 at 18:56
  • If by "buffering" you mean that you are using blocking MPI calls, then you need to restructure the code to use non-blocking communication, so that both normal and terminate messages can be received. – mabraham Aug 26 '15 at 20:23
  • MPI_Abort(MPI_COMM_WORLD,0) is one way to implement asynchronous termination. – Jeff Hammond Aug 27 '15 at 02:32
  • I'm using non-blocking communication and after certain number of iterations, each processor sends and reads two messages (that's the way the algorithm works). The buffering is happening since algorithms have different running times. – user5244097 Aug 27 '15 at 18:04
  • I mean "immediately" in less than 1 minute and I don't want to throw errors in the program, since I need to store and write reports locally after the interruption is detected. Thank you. – user5244097 Aug 27 '15 at 18:14
  • I think you are using the word buffering in an unexpected way. One way to do this is by running the computation in a separate thread, so that the main thread can continue listening to messages without blocking on the computation. – Kirill Aug 28 '15 at 05:24

0 Answers0