You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
389 lines
19 KiB
389 lines
19 KiB
<!-------- @HEADER
|
|
!
|
|
! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
|
!
|
|
! Zoltan Toolkit for Load-balancing, Partitioning, Ordering and Coloring
|
|
! Copyright 2012 Sandia Corporation
|
|
!
|
|
! Under the terms of Contract DE-AC04-94AL85000 with Sandia Corporation,
|
|
! the U.S. Government retains certain rights in this software.
|
|
!
|
|
! Redistribution and use in source and binary forms, with or without
|
|
! modification, are permitted provided that the following conditions are
|
|
! met:
|
|
!
|
|
! 1. Redistributions of source code must retain the above copyright
|
|
! notice, this list of conditions and the following disclaimer.
|
|
!
|
|
! 2. Redistributions in binary form must reproduce the above copyright
|
|
! notice, this list of conditions and the following disclaimer in the
|
|
! documentation and/or other materials provided with the distribution.
|
|
!
|
|
! 3. Neither the name of the Corporation nor the names of the
|
|
! contributors may be used to endorse or promote products derived from
|
|
! this software without specific prior written permission.
|
|
!
|
|
! THIS SOFTWARE IS PROVIDED BY SANDIA CORPORATION "AS IS" AND ANY
|
|
! EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
! IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
|
! PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL SANDIA CORPORATION OR THE
|
|
! CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
|
|
! EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
|
|
! PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
|
|
! PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
|
|
! LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
|
|
! NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
|
|
! SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
|
!
|
|
! Questions? Contact Karen Devine kddevin@sandia.gov
|
|
! Erik Boman egboman@sandia.gov
|
|
!
|
|
! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
|
|
!
|
|
! @HEADER
|
|
------->
|
|
|
|
|
|
<HTML>
|
|
<HEAD>
|
|
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
|
|
<META NAME="GENERATOR" CONTENT="Mozilla/4.04 [en] (X11; U; SunOS 4.1.3_U1 sun4m) [Netscape]">
|
|
<META NAME="sandia.approved" CONTENT="SAND99-1376">
|
|
<META NAME="author" CONTENT="karen devine, kddevin@sandia.gov">
|
|
<TITLE>Zoltan Developer's Guide: Quality Program</TITLE>
|
|
</HEAD>
|
|
<BODY BGCOLOR="#FFFFFF">
|
|
|
|
<div ALIGN=right><b><i><a href="dev.html">Zoltan Developer's Guide</a> |
|
|
<a href="dev_dist.html">Next</a> | <a href="dev_intro_coding.html">Previous</a></i></b></div>
|
|
|
|
|
|
<H2>
|
|
<A NAME="SQE"></A>Zoltan Quality Assurance</H2>
|
|
This document describes the Software Quality Assurance (SQA)
|
|
policies and procedures used in the Zoltan project. Zoltan developers
|
|
at Sandia or under contract to Sandia are required to follow these
|
|
software development policies.
|
|
|
|
|
|
|
|
<blockquote>
|
|
<a href="#Quality_Policy">Quality Policy</a>
|
|
<br><a href="#Quality_Definition">Quality Definition</a>
|
|
<br><A href="#Defect_Classification">Classification of Defects</a>
|
|
<br><A href="#Release_Policy">Release Policy</a>
|
|
<br><A href="#Quality_Tools">Software Quality Tools</a>
|
|
<br><A href="#Quality_Processes">Software Quality Processes</a>
|
|
<br><A href="#Zoltan_ASC">Zoltan´s implementation of the ASC Software Quality Engineering Practices</a>
|
|
</blockquote>
|
|
|
|
|
|
|
|
<H3>
|
|
<A NAME="Quality_Policy"></A>Quality Policy
|
|
</H3>
|
|
<p>
|
|
Sandia´s ASC Quality Management Council (AQMC) developed and manages the
|
|
Quality Assurance Program (QAP) for Sandia´s ASC program. The AQMC chartered
|
|
the development of the <I>Sandia National Laboratories Advanced Simulation and Computing
|
|
(ASC) Software Quality Plan, Part 1: ASC Software Quality Engineering Practices, Version 2.0</I>
|
|
document (SAND 2006-5998) as the practical SQA guidance for projects like Zoltan.
|
|
A companion document, <I>Sandia National Laboratories Advanced Simulation and Computing
|
|
(ASC) Software Quality Plan, Part 2: Mappings for the ASC Software Quality Practices</I> (SAND 2006-5997),
|
|
shows how these practices satisfy corporate policies including CPR001.3.6, Corporate
|
|
Software Engineering Excellence, and DOE/NNSA orders 414.1C and QC-1 rev 10.
|
|
<p>
|
|
The Zoltan project is committed to a program of quality improvement in compliance
|
|
with the <I>ASC Software Quality Engineering Practices</I> document. The Zoltan Team Leader is the owner
|
|
of the Zoltan quality system. Zoltan developers at Sandia or under contract to Sandia
|
|
are required to follow these software development practices. The Zoltan team shall
|
|
participate in all reporting processes, audits, and assessments as directed by the AQMC.
|
|
|
|
<H3><A NAME="Quality_Definition"></A>Quality Definition</H3>
|
|
<p>
|
|
QC-1 rev 10 defines quality as "the degree to which customer requirements are met."
|
|
<p>
|
|
The Zoltan project accepts the following definition of quality:
|
|
"the totality of characteristics of a product or service that bear
|
|
on its ability to satisfy stated or implied needs." This is Juran´s
|
|
"fitness for use" definition of quality (ANSI/ASQC A8402-1994.)
|
|
This superior definition of quality fully satisfies the QC-1 rev 10 definition.
|
|
This definition is also more useful in a research environment where the requirements are
|
|
derived from a research proposal rather than directly from customers and end users.
|
|
|
|
<H3><A NAME="Defect_Classification"></A>Classification of Defects</H3>
|
|
The Zoltan project accepts the following system of classification of
|
|
defects:
|
|
<BLOCKQUOTE>
|
|
<b>Critical:</b> A defect that could lead to loss of life,
|
|
significant environmental damage, or substantial financial loss.
|
|
<br><b>Major:</b> A non critical defect that significantly
|
|
impacts Zoltan's fitness for use.
|
|
<br><b>Minor:</b> A (non critical, non major) defect that
|
|
reasonably impacts Zoltan's fitness for use.
|
|
<br><b>Incidental:</b> Any other defect which does not
|
|
reasonably reduce Zoltan's fitness for use.
|
|
</BLOCKQUOTE>
|
|
<p>
|
|
|
|
<H3><A NAME="Release_Policy"></A>Release Policy</H3>
|
|
<p>
|
|
Only the Zoltan team leader may authorize (certify) a release.
|
|
The Zoltan team leader shall not release software with
|
|
any known critical or major defects.
|
|
User registration shall allow the Zoltan team to
|
|
notify all Sandia and ASC users and to recall their
|
|
defective software if a critical or major defect
|
|
is discovered after release.
|
|
The Zoltan team leader may determine that it is
|
|
acceptable to release software with known minor or incidental defects.
|
|
<p>
|
|
|
|
<H3><A NAME="Quality_Tools"></A>Software Quality Tools</H3>
|
|
<p>
|
|
Because of the small scale of the Zoltan Project, only a few, simple tools
|
|
are required for use by Zoltan developers:
|
|
<BLOCKQUOTE>
|
|
<b>CVS:</b> maintains code, documentation, meeting
|
|
notes, emails, and QA program artifacts;
|
|
<br><b>Purify, PureCoverage, Quantify (Rational), Valgrind, gdb:</b>
|
|
for dynamic code testing, coverage measurements, and performance analysis;
|
|
<br><b>Bugzilla:</b> tracks bugs, requests for changes,
|
|
and enhancements;
|
|
<br><b>Mailman:</b> creates email lists to automatically
|
|
notify users by area(s) of interest;
|
|
<br><b>Makefiles:</b> ensures proper compilation and linking
|
|
for all supported platforms; and
|
|
<br><b>Zoltan Test Script:</b> runs
|
|
integration, regression, release and acceptance testing.
|
|
</BLOCKQUOTE>
|
|
|
|
<H3><A NAME="Quality_Processes"></A>Software Quality Processes</H3>
|
|
<p>
|
|
<b>Bug Reporting, Issue Tracking, Enhancement Requests:</b> All of these
|
|
items are now directly entered into Bugzilla by developers and users.
|
|
This "process" is built into the tool. Detailed instructions
|
|
for using Bugzilla are found on the Zoltan web page. Bugzilla also
|
|
provides query and report features for tracking the status of entered items;.
|
|
<p>
|
|
A process is defined as a sequence of steps performed for a given
|
|
purpose (IEEE Std. 610.12.) Zoltan´s other processes are defined as
|
|
checklists because checklists are one of the seven fundamental quality tools.
|
|
These checklists are also the primary artifact created when following a process.
|
|
Currently the following processes are defined:
|
|
|
|
<BLOCKQUOTE>
|
|
<b>Development:</b> (not currently used) defines the software development
|
|
process including
|
|
requirements, design, implementation, testing, reviews, and approvals;
|
|
<br><b>Release:</b> defines the release process including testing requirements
|
|
and creation of the release product;
|
|
<br><b>Request:</b> defines the process of
|
|
capturing user requests for new features;
|
|
<br> Note: this process is now obsolete. Request processes in progress
|
|
may continue until complete but new requests should use Bugzilla;
|
|
<br><b>Requirement:</b> the process of capturing user comments that
|
|
may become requirements after review and approval;
|
|
<br> Note: this process is now obsolete. Requirement processes in progress
|
|
may continue until complete but new requirements should use Bugzilla;
|
|
<br><b>Review:</b> defines the materials reviewed prior to acceptance
|
|
for Zoltan release;
|
|
<br> Note: Developers are encouraged to use Bugzilla to enter the
|
|
specific review process rather than use the Review checklist. At this time this
|
|
is an trial effort and either method may be used.
|
|
<br><b>Third Party Software:</b>defines the steps required to obtain, manage,
|
|
use, and test for software created outside of Zoltan and the ASC program; and
|
|
<br><b>Training:</b> defines the material a new developer must read, required
|
|
skills to demonstrate and computer accounts that must be obtained.
|
|
</BLOCKQUOTE>
|
|
|
|
<p>
|
|
Zoltan's software quality process checklists define how work may be performed,
|
|
including process ownership, authorization to perform, activities and their
|
|
sequence (when sequencing is required), process instructions, metrics, and
|
|
identification of who performed each activity.
|
|
<p>
|
|
The only allowed source for process checklists is Zoltan's CVS repository
|
|
in the SQA_templates directory (under Zoltan_Internal.) A Zoltan developer
|
|
initiates a process by obtaining the current CVS version of the process, renaming
|
|
it, and committing the renamed process checklist back into CVS in an appropriate
|
|
directory on the same day. The process may continue under this committed version
|
|
even if its original process is later superseded unless specifically requested by
|
|
the Team Leader. After one or more activities are completed, the process
|
|
checklist is updated to reflect the results and committed back to CVS (with
|
|
appropriate comments.) A process is completed when all required activities
|
|
are completed including reviews and approvals (as necessary), and committed to CVS.
|
|
The final CVS comment should indicate that the process is complete.
|
|
<p>
|
|
<H3><A NAME="Zoltan_ASC"></A>
|
|
Zoltan´s implementation of the <I>ASC Software Quality Engineering Practices</I></H3>
|
|
<p>
|
|
The following is brief description <b> for Zoltan developers</b> about the Zoltan
|
|
project´s implementation of the <I>ASC Software Quality Engineering Practices</I> (SAND 2006-5998):
|
|
<p>
|
|
<b>PR1. Document and maintain a strategic plan.</b><br>
|
|
The Zoltan web page has a direct hyperlink to the Zoltan Project Description
|
|
defining its mission and philosophy. The Zoltan project has a strong association
|
|
with the Trilinos project to share in the development of common software
|
|
engineering practices and sharing of appropriate tools and experience.
|
|
<p>
|
|
<b>PR2. Perform a risk-based assessment, determine level of formality
|
|
and applicable practices, and obtain approvals.</b><br>
|
|
The Zoltan project has an approved level of formality (medium) for its
|
|
deliverable software. Its biggest technical risk results from providing
|
|
parallel solutions to NP hard partitioning problems. Technical risks are
|
|
mitigated by collaborations within Sandia and internationally. The most
|
|
significant non-technical risk is the conflicting priorities of Zoltan
|
|
developers working on many other projects simultaneously.
|
|
<p>
|
|
<b>PR3. Document lifecycle processes and their interdependencies, and
|
|
obtain approvals.</b><br>
|
|
The Zoltan project follows the <I>Trilinos Software Lifecycle Model</I>
|
|
(SAND 2006-6929). It also follows the ANSI/ASQ Z1.13-1999 standard
|
|
<I>Quality Guidelines for Research</I> which is compatible with the research
|
|
phase in the Trilinos Lifecycle model.
|
|
<p>
|
|
<b>PR4. Define, collect, and monitor appropriate process metrics. </b><br>
|
|
The Zoltan project is committed to comply fully with the new and evolving
|
|
AQMC requirements for collecting and reporting "defect" metrics.
|
|
Other metrics determined by Zoltan´s continual process improvement process
|
|
(<b>PR 5</B>) will be implemented.
|
|
<p>
|
|
<b>PR5. Periodically evaluate quality problems and implement process
|
|
improvements.</b><br>
|
|
The Zoltan project has built the Deming/Shewhart process improvement
|
|
cycle PDCA (Plan, Do, Check, Act) into all of its process checklists. This is
|
|
the most effective process improvement technique known. It is recommended
|
|
by ISO 9001:2000.
|
|
<p>
|
|
<b>PR6. Identify stakeholders and other requirements sources.</b><br>
|
|
The Zoltan project´s primary stakeholders are the ASC applications using
|
|
Zoltan including SIERRA, ACME, ALEGRA/NEVADA, XYCE, and Trilinos.
|
|
<p>
|
|
<b>PR7. Gather and manage stakeholders´ expectations and requirements.</b><br>
|
|
The Zoltan project´s primary input from ASC applications´ expectations and
|
|
requirements are via their communication of Zoltan´s role in meeting their
|
|
ASC milestones. Since Zoltan is an "enabling technology," these requirements
|
|
are broadly stated performance improvement needs. The Zoltan team actively anticipates
|
|
and develops load balancing software for the future needs of the Sandia research community
|
|
before they actually become formal requirements.
|
|
<p>
|
|
<b>PR8. Derive, negotiate, manage, and trace requirements.</b><br>
|
|
Zoltan project requirements normally derive from its funded research proposals
|
|
which state research goals. This is a normal procedure in a research
|
|
environment (see ANSI/ASQ Z1.13-1999). Periodic and final reports document
|
|
the success in meeting these research goals.
|
|
<p>
|
|
<b>PR9. Identify and analyze risk events.</b><br>
|
|
All Zoltan developers should report any new or changed risks via the zoltan-dev
|
|
email target for evaluation by the Team Lead.
|
|
<p>
|
|
<b>PR10. Define, monitor, and implement the risk response.</b><br>
|
|
The Zoltan team will create a corrective action plan whenever any condition
|
|
threatens to adversely impact the Zoltan project resources or schedule.
|
|
<p>
|
|
<b>PR11. Create and manage the project plan.</b><br>
|
|
ANSI/ASQ Z1.13-1999 states that the research proposal is equivalent to a
|
|
project plan in a research environment. The Team Leader assigns responsibilities,
|
|
deliverables, resources, and schedules in order to manage the project.
|
|
<p>
|
|
<b>PR12. Track project performance versus project plan and implement
|
|
needed (corrective) actions.</b><br>
|
|
The Team Leader periodically tracks responsibilities, deliverables, resources,
|
|
and schedules in order to manage the project.
|
|
<p>
|
|
<b>PR13. Communicate and review design.</b><br>
|
|
The Zoltan architecture is fully documented in the Zoltan Developer´s Guide.
|
|
New features are originally documented and reviewed in team discussions to
|
|
the zoltan-dev email target. Prior to release, the design documentation is
|
|
finalized in both the Zoltan Developer´s Guide and the Zoltan User´s Guide.
|
|
<p>
|
|
<b>PR14. Create required software and product documentation.</b><br>
|
|
Developers will follow the Zoltan Development Process Checklist.
|
|
<p>
|
|
<b>PR15. Identify and track third party software products and follow
|
|
applicable agreements.</b><br>
|
|
Developers will follow the Zoltan Third Party Software Process Checklist.
|
|
<p>
|
|
<b>PR16. Identify, accept ownership, and manage the assimilation of other
|
|
software products.</b><br>
|
|
Not applicable since Zoltan does not "assimilate" third party software.
|
|
<p>
|
|
<b>PR17. Perform version control of identified software product artifacts.</b><br>
|
|
All software and process artifact are under maintained CVS as early as reasonable
|
|
after their creation.
|
|
<p>
|
|
<b>PR18. Record and track issues associated with the software product.</b><br>
|
|
Developers will use Bugzilla to record and track issues.
|
|
<p>
|
|
<b>PR19. Ensure backup and disaster recovery of software product artifacts.</b><br>
|
|
Nightly backups, periodic offsite backups, and disaster recovery are services
|
|
provided by the CSRI computer support staff. Disaster recovery has been successfully
|
|
performed from real problems.
|
|
<p>
|
|
<b>PR20. Plan and generate the release package.</b><br>
|
|
Developers will follow the Zoltan Release Process Checklist.
|
|
<p>
|
|
<b>PR21. Certify that the software product (code and its related artifacts) is
|
|
ready for release and distribution.</b><br>
|
|
The Zoltan Team Leader will certify any version of Zoltan for release via an
|
|
email to zoltan-dev target.
|
|
<p>
|
|
<b>PR22. Distribute release to customers.</b><br>
|
|
Zoltan files are released via a download from the Zoltan web site. The Zoltan
|
|
Team Leader will make the download available after certification. (Research
|
|
versions of the Zoltan software are directly available to collaborators for
|
|
development.)
|
|
<p>
|
|
<b>PR23. Define and implement a customer support plan.</b><br>
|
|
(See <b>PR 6</b> for a list of ASC stakeholders.) The Zoltan team provides one-on-one
|
|
training whenever requested and quickly responds to any user complaint.
|
|
<p>
|
|
<b>PR24. Implement the training identified in the customer support plan.</b><br>
|
|
See <b>PR 23</b> above. If additional training is ever requested, the Zoltan project
|
|
will piggy back on the annual Trilinos Users Group meeting with a training
|
|
session on using Zoltan.
|
|
<p>
|
|
<b>PR25. Evaluate customer feedback to determine customer satisfaction.</b><br>
|
|
<p>
|
|
<p>
|
|
<b>PR 26 Develop and maintain a software verification plan.</b><br>
|
|
Developers are expected to create new tests for the Zoltan test suite when
|
|
new features are added to Zoltan.
|
|
<p>
|
|
Currently, a new test framework based on FAST/EXACT is being implemented.
|
|
Documentation about this test framework is under preparation. A process
|
|
checklist will be developed around the steps required to add new tests to
|
|
the suite and to run the suite.
|
|
<p>
|
|
<b>PR27. Conduct tests to demonstrate that acceptance criteria are met and to
|
|
ensure that previously tested capabilities continue to perform as expected.</b><br>
|
|
This practice is a subset of the Zoltan Release Process Checklist.
|
|
<p>
|
|
<b>PR28. Conduct independent technical reviews to evaluate adequacy with respect
|
|
to requirements.</b><br>
|
|
Developers will follow the Zoltan Review Process Checklist. ANSI/ASQ Z1.13-1999
|
|
states that the peer reviewed publications and conference presentations are a normal
|
|
form of technical review in the research environment.
|
|
<p>
|
|
<b>PR29. Determine project team training needed to fulfill assigned roles
|
|
and responsibilities.</b><br>
|
|
New developers will follow the Zoltan Training Process for new team members.
|
|
<p>
|
|
<b>PR30. Track training undertaken by project teams.</b><br>
|
|
Zoltan developers are encouraged to participate in the annual Trilios Users Group
|
|
(TUG) meeting which provides sessions for SQA/SQE training to developers.
|
|
Attendance records are kept for this event and for any Zoltan team meetings that
|
|
provide training. Sandia provides many other opportunities for training including
|
|
formal courses and periodic internal software developers conferences. External
|
|
conferences (e.g., IPDPS and SIAM) are counted as technical training.
|
|
|
|
<p>
|
|
<HR WIDTH="100%">
|
|
<BR>[<A HREF="dev.html">Table of Contents</A> | <A HREF="dev_dist.html">Next:
|
|
Zoltan Distribution</A> | <A HREF="dev_intro_coding.html">Previous:
|
|
Coding Principles in Zoltan</A>
|
|
| <a href="https://www.sandia.gov/general/privacy-security/index.html">Privacy and Security</a>]
|
|
</body>
|
|
|
|
</html>
|
|
|