Software Engineering

In practice, the differences between levels A and B are small:

• Certain objectives of the software design process must be independently verified.
• Source code accuracy, consistency, and compliance with the software architecture must be independently verified.
• Robustness of object code with low-level requirements must be independently verified.
• Test coverage of software structure (modified condition/decision) must be satisfied independently for level A, and is optional for level B and lower.


Different hazard categories require different failure rates and ifferent levels of investment in varying software engineering techniques. For example, a nuclear capable US navy cruiser had ten seperate stages of analysis (eg subsystem analysis, radiation hazard analysis, inadvertant launch analysis) which overlapped and on which the development was based, rather than being added retrofitted.



Hazard Elimination

Many hazards can be eliminated by small changes in design.
• Eg Motor reversing circuit:
  
  If the switches dont move together there is a short circuit, and a fire could occur.
  The solution is to redesign so it is intrinsically safe. Intrinsically safe software is the holy grail, however this normally requires a system level approach.
  Nuclear reactors normally have mechanical fail safes as you can never completely trust software (unless it is a tiny program that you can prove mathematically). Eg pebble bed reactors which are self controlling (as the reactive pebbles and the gas around them heat up, they push each other apart)

  Particular Problems of Large Systems

  In 1988 Curtis,Kramer and Iscoe studied 17 large systems and why they failed. Over 97 interviews team and organistational factors were investigated. The main findings were that large software projects fail because of 1. A thin spread of applciation domain knowledge
• 2. Fluctuating and conflicting requirements
• 3. Breakdown of communication and coordination
  They found the tytpical progression to disaster was 1 -> 2 ->3
  
  

  Thin spread of application domain knowledge

  It is rare to find someone who understands the entire solution. In some areas, eg pilot training, there is a structured effort to overcome this. Otherwise, with luck, you may get a genuine "guru" but even then you can expect specification mistakes.

  Changing requirements

  Competing products, new standards, new equipment and new users all combine to changing requirements.

  Communication problems

  Problems with communications increase with the more layers and more managers. Coping mechanisms such as commitees increase the hierarchy. Managers are often loth to believe bad news, much less pass it on.

  Rates of Software Engineering Failure

  Requirements	Very High
  Specification	Low
  Design	Low
  Implementation	Low
  Installation	High
  Operation	Enormous
  Maintenance	Very High

  Requirements

  The rate of failure is high for requirements as they are developed by two or more groups of people who come from different disciplines and speak different languages.
  
  

  Specification,Design,Implementation

  These activities are done by a group of simliarly disciplined professionals.
  
  

  Installation

  Installation is often done by people who werent involved in the implementation and dont understand the system.
  
  

  Operation

  Operation is often left to people who dont understand the system or what it is meant to achieve.
  Robert Courtney, a New York security consultant, examined thousands of security beaches in both industry and government and found that 68% of them were due to careless operations or incompetent operations.
  
  

  Maintenance

  Maintenance is often done as an afterthought by people who dont understand what they are changing.

  Abstract

  The Software Life Cycle Model consists of the System Development Cycle and the System Maintenance Cycle.
  
  

  Stages in the Software Life Cycle

  The software life cycle (according to IEEE1074) starts with the selection of an appropriate life cycle model followed by recruitment. There are many life cycle models that can be chosen. The following stages are in the sequential order of the Waterfall Model, due to its simplicity and prevalance in software engineering.
  
  
  
• Project planning and a feasibility study are required to determine first of all if a new system is needed, and if it is possible within reasonable costs and time restraints.
  
  
• Requirement analysis is one of the first stages in the waterfall model, in which the requirements of the new system are found through interviews with potential users and management, and feedback on prototypes. It is important that those chosen to be interviewed are technically capable or the interviewer can extrapolate what the user wants form their feedback and spot possible requirement contradictions.
  
  
• From the requirements a formal specification can be written up; normally a strict and mathematical description of what the system must achieve. These come directly from the requirements, and may follow more informal specifications. When the application is further into the development cycle it will be continuously checked against the specifications to ensure the system fulfills the its required purpose.
  
  
• The central stage in the waterfall model is the implementation and (unit & system) testing. Implementation is the most obvious part of creating a system, but often one of the smaller sections time and cost wise. Testing is important to prevent bugs and to test the implementations against the specifications. A failure is generally considered to be when a system doesn't perform as intended, whilst a fault is a programming error that may develop into a failure. It is important testing is thorough,as the earlier a defect is detected the cheaper it is to fix. Testing generally starts during the requirement analysis stage of the software development life cycle. Testing then continues into the development and then execution stages. Unit testing typically follows implementation, followed by integration testing.
  
  
• Acceptance, installation and deployment are the final stages of initial development where the software is put into practice and runs actual business. Choices in deployment time and back up strategies in case of system failure are essential for critical services (eg the failure of the LAS to have a back up ambulance system to switch back to after the failed system deployment).
  
  
• Training and support of the users then increases the likelihood of correct usage of the system.
  
  
• Maintenance can be a very expensive task, as it essentially lasts as long as the system. It may require some of the original programmers to be employed continuously to ensure the correct running of the system and for keeping it appropriately up to date. If requirements change at a late stage it can prove very expensive.
  
  Typically about 40% of development time is spent in the requirements stage of the life cycle, just 20% in the actual implementation and the remaining 40% in testing.
  
  

  Links & Resources

  At the time of writing (Dec 06) the following resources were useful
• Software Develoment Process Wikipedia.org
• Quick Study Notes Computerworld.com
• The Mythical Man Month Fred Brooks

  The Spiral Model

  Iterative Design

  One of the main objections to the waterfall model is that in the real world requirements change during design for a number of reasons: you simply cant create a specification at the start of a project and expect it to be the same when the project is completed.
  The solution to these objections is the notion of iterative development, where there is feedback between prototypes and design.
  
  

  The Spiral Model

  The spiral model (Barry Boehm, 1988) attempts to combine the advantages of the waterfall model with the flexibility of an iterative approach.
  
  "A Spiral Model of Software Development and Enhancement", Barry W. Bohem
  
  

  Advantages

• Estimates (i.e. budget, schedule, etc.) get more realistic as work progresses, because important issues are discovered earlier.
• It is more able to cope with the (nearly inevitable) changes that software development generally entails.
• Software engineers (who can get restless with protracted design processes) can get their hands in and start working on a project earlier.
  
  

  Further reading

• A Spiral Model of Software Development and Enhancement (PDF) - Barry Boehm's original article

  Testing

  Testing can take up to half the cost of development in industry. Bill Gates: "are we in the business of writing software, or test harnesses?"
  Testing takes place at a number of levels:
• Validation of the initial design
• Module test after coding
• System test after itnegration
• Beta test/field trial
• Subsequent litigation
• ...
  Note that testing earlier is far less expensive than testing late.
  
  

  Regression testing

  The major advance in package software in the last ten years has been in testing.
  Regression testing involves checking that the new version of the software gives the same answers as the old version.
  
  A database is continually updated, containing every bug ever found. Unless all previous bugs are tested for bugs have a ~20% chance of reoccuring unexpectedly. It is best to test data given by real users, as this is what is most likely to be noticed.
  
  Reliability growth models help us asses mean time to failure, the number of bugs remaining and the economics of further testing. It is found emprically. Based on these models the rate of software failure drops exponentially at first, then settles down to decrease at k/t.
  Changing testers often makes new bugs visibile, as shown below
  
  Failure to understand the environment in which the system will actually be deployed often leads to incorrect testing and failures. A military approach (a hostile review followed by prolonged field testing) can be very effective, but in industry corporate politics normally prevents a wide uptake of good practices.

  Risk Reduction vs Due Diligence

  Often one can never be certain about how many bugs are left. Organisations are highly averse to such uncertainty and prfer to avoid risk issues. Legal and cultural pressures replace risk reduction with "due diligence": a standard (but often insufficient) checklist is followed to ensure the legal pressures are satisfied but often not the problem itself.
  
  

  Summary

  A lack of consistent testing can cause programs to behave incorrectly, fail or be insecure. For example in March 2002 it was reported that software bugs in Britain's national tax system resulted in more than 100,000 erroneous tax overcharges. The problem was partly attributed to the difficulty of testing the integration of multiple systems.
  Testing is important as there will inevitably be bugs in any program of significant length. These may be caused by a variety of factors at various stages in the development cycle. The earlier a defect is found the cheaper it is to fix -if a product has to be recalled due to a serious fault the cost can be enormous. Inadequate testing may mean no one notices a problem until a vital system crashes. Testing itself however can be very time consuming and expensive and there can never be testing of 100% of real world situations in any program of complexity. More safety critical and military projects will employ greater testing, through large amounts of simulation testing. Often mathematical proofs will be used to show that algorithms work in theory, complementing real world tests.
  There are numerous testing techniques and models, application depends upon who the final system is for. Tests can be broadly split into black box tests (external, based on functionality) and white box testing (based on knowledge of the internal workings of the program, following branches etc.) Unit testing tests particular methods and is useful for the micro scale examination of individual parts of a program.
  If the product is mass produced a company will generally factor the cost of testing against the possible cost of a recall. For such mass market software often beta testing is used, where large numbers of volunteers test the product and report errors before the final product is shipped. This will test common errors as the people testing are fairly normal users, but they lack the expertise to test extreme cases and to notice many bugs and their causes.
  In a “test-driven software development model” unit tests are written first, and as more and more code of the main project is written less and less of the unit tests are failed.
  In a clean room process probabilities are assigned to paths users can take both correct and incorrect, and the most common paths are tested more thoroughly. This method was employed by Ericson in their OS development, cutting errors down to 1 per 1000 lines of code, down from the industry average of about 25 per 1000 (Scientific American, 1994).
  A good test engineer will have the ability to take the point of view of the customer, a desire for quality and attention and good judgment on where to focus testing efforts with scarce time resources.
  
  

  Validation and Verification

  Validation

  In the waterfall model, validation operations provide feedback from Specification to Requirements and from Implementation/unit testing to Specification.
  
• Validation can be thought of as "Are we building the righy system?"
  

  Verificatiom

  Verification operations provide feedback from Integration/ system testing to Implementation/unit testing, and from operations/maintenance back to Integration/system testing.
• Verification asks "Are we building this system right/correctly?" The waterfall model is reportedly employed by both the US D.O.D and NASA as their main software development model, due to its strict design practices and clear progress markers. This is despite being described as “risky and invites failure” by W.W.Royce, who . originally created the term “Waterfall model” and criticized it in favor of an iterative approach.
  The waterfall model is a top down sequential model split into the sections shown in the software life cycle article, and illustrated by the image below
  
  Image by Paul A. Hoadley, Creative Commons license
  Note that:
• Requirements are written in the user's language
• Specification is written in system language
• Unit testing checks units against the spec
• System testing checks the requirements are met
  
  The waterfall model is well understood and time tested but generally considered to be less useful than it once was due to the increasing complexity of systems. It “works well for automating the tasks of clerks and accountants, less well for knowledge works such as experts trying to solve problems” (Information Center Quarterly article, Larry Runge). Another problem is that in the waterfall model users only input is in specifying requirements and that all requirements must be specified at one points before production begins. However, requirements typically change through the process and require more feedback.
  
  Summary of advantages:
• Easy for managers to understand, plan by and to test progress against as it has very clear sequential milestones
• Encourages good design practices such as early clarification of system goals, which in turn save time and money (the earlier a bug is caught, the less harm it can cause)
• Due to lots of up front design planning, should the project be stopped and taken up at a later date, or should project members change, implementation can continue far easier than with more agile development models where there is considerably less design documentation.
• Compatible with a wide range of design strategies
  
  Summary of disadvantages
• Users may change requirements often, at stages beyond the requirements stage
• Users may not be able to state requirements until they see a working prototype that they can criticise
• At the implementation stage programmers may find that there is a near impossible problem to implement, and it may be easier to change the original design than to proceed ahead as planned.
• Constant verification of each stage is required to ensure that the next phase can start on a correct base
• Technology may change during development
  
  
  Most of the problems of the waterfall model are solved by iterative and evolutionary models such as the Spiral Model