An MPI implementation of a self-submitting parallel job queue

Research output: Contribution to journalArticle

1 Citation (Scopus)

Abstract

We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.

Original languageEnglish (US)
Pages (from-to)443-464
Number of pages22
JournalInternational Journal of Parallel Programming
Volume40
Issue number4
DOIs
StatePublished - Aug 2012

Fingerprint

Queue
Partial differential equations
Partial differential equation
Bifurcation (mathematics)
Methodology
Bifurcation Diagram
C++
Combinatorics
Data structures
Data Structures
Programming
Scaling
Testing
Vertex of a graph
Demonstrate
Experiment
Libraries
Experiments
Concepts

Keywords

  • Bifurcation
  • Job queue
  • MPI
  • MPQueue
  • Non-attacking queens

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Theoretical Computer Science

Cite this

@article{9d61b392ad9c4d1c9bb5d0be2c35adfd,
title = "An MPI implementation of a self-submitting parallel job queue",
abstract = "We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.",
keywords = "Bifurcation, Job queue, MPI, MPQueue, Non-attacking queens",
author = "Neuberger, {John M.} and N{\'a}ndor Sieben and Swift, {James W.}",
year = "2012",
month = "8",
doi = "10.1007/s10766-012-0195-8",
language = "English (US)",
volume = "40",
pages = "443--464",
journal = "International Journal of Parallel Programming",
issn = "0885-7458",
publisher = "Springer New York",
number = "4",

}

TY - JOUR

T1 - An MPI implementation of a self-submitting parallel job queue

AU - Neuberger, John M.

AU - Sieben, Nándor

AU - Swift, James W.

PY - 2012/8

Y1 - 2012/8

N2 - We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.

AB - We present a simple and easy to apply methodology for using high-level self-submitting parallel job queues in an MPI environment. Using C++, we implemented a library of functions, MPQueue, both for testing our concepts and for use in real applications. In particular, we have applied our ideas toward solving computational combinatorics problems and for finding bifurcation diagrams of solutions of partial differential equations (PDE). Our method is general and can be applied in many situations without a lot of programming effort. The key idea is that workers themselves can easily submit new jobs to the currently running job queue. Our applications involve complicated data structures, so we employ serialization to allow data to be effortlessly passed between nodes. Using our library, one can solve large problems in parallel without being an expert in MPI. We demonstrate our methodology and the features of the library with several example programs, and give some results from our current PDE research. We show that our techniques are efficient and effective via overhead and scaling experiments.

KW - Bifurcation

KW - Job queue

KW - MPI

KW - MPQueue

KW - Non-attacking queens

UR - http://www.scopus.com/inward/record.url?scp=84862135663&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84862135663&partnerID=8YFLogxK

U2 - 10.1007/s10766-012-0195-8

DO - 10.1007/s10766-012-0195-8

M3 - Article

VL - 40

SP - 443

EP - 464

JO - International Journal of Parallel Programming

JF - International Journal of Parallel Programming

SN - 0885-7458

IS - 4

ER -