BE 1602 Engineering Practice

Artefact Requirements and Testing

By Prof. John. P. Cosmas

1
Week 2
Week 3 Test Plan
Week 5 with - Arash Soleiman Falah
Week 6
 Motivations
 "The hardest single part of building any system is deciding precisely
what to build. No other part of the conceptual work is so difficult as
establishing the detailed technical requirements, including all the
interfaces to people, to machines, and to other systems. No other
part of the work so cripples the resulting system if done wrong. No
other part is more difficult to rectify later.

Therefore the most important function that system builders do for

their clients is the iterative extraction and refinement of the product
requirements.“

 F.P. Brooks, “No Silver Bullet: Essence and Accidents of Engineering”

 The Project Process – Where are we?

Planning Phase Analysis Phase Implement and Test Phase

Integrate Phase

Develop User Derive Develop User Testing

Requirements Functional Mechanical
Requirements Components

Develop Functional
Derive Electric Testing
Technical Components
Requirements
Develop Technical
Software Testing
Components
Design System
Architecture Integrate
Mechanical,
Electric and
Software
 Design Phase Requirements Definition and

 In design phase the following is identified:

 User Requirements – top level definition of how the user
interacts with the system
 Functional Requirements – the functions and interfaces between
them to realise operation of the system
 Technical Requirements - aspects such as performance,
reliability, and availability that must be provided in order to
successfully complete a project.
 In testing phase the design phase requirements are
verified:
 User Testing – tests how the system interacts with the user
against user requirements definition
 Functional Testing – tests how the system functionally
 operates correctly against functional requirements definition
 Technical Testing – tests how the system performs against
performance, reliability and availability technical requirements
specifications
8
 The five themes – First Project
 Writer. Students must construct a robot that can autonomously write a word on a piece of paper. The word must
be at least three letters long. The paper cannot be stuck onto the demonstration surface (floor); instead, its design
must include some means of holding the paper steady while the robot writes on it. A set-up like a printer that can
pick the paper up is permitted.
 Walker. Students must construct a robot that can autonomously walk in as straight a line as possible for a distance
of three metres, without intervention.
 Stacker. Students must construct a robot that can autonomously re-arrange a stack of coloured blocks into a
different order. The blocks are three coloured Lego blocks (not stuck together) arranged into a single vertical stack
and in a particular, pre-defined order (blocks will be different colours, for example, red, blue and yellow). The robot
must re-arrange them into another single stack, but with the colours in a different pre-defined order (whatever the
pre-defined starting and ending arrangements of the blocks, they should be the same for all students and should be
chosen so that it is not possible to achieve the re-arrangement simply by picking up all three blocks at once, rotating
them and then placing them back onto the demonstration surface). The robot must not be in contact with the blocks
immediately before the demonstration starts or at the point the demonstration finishes. It is recommended that the
robot must be at least 10cm away from the blocks at the start of the demonstration. Each team should be given a
set of blocks to use during development.
 Retriever. The students must construct a robot that can autonomously locate an object within a space (a 1m
square area on the demonstration surface/floor), pick it up and transport it to a fixed location, where it must set the
object down without breaking it. The robot must start from a specified location just outside the square. The object
should be placed at a random location inside the square. This should be chosen just before the demonstration starts
and should be varied randomly for each team. The robot must not be in contact with the object at the end of the
demonstration. It is recommended that the object should be a simple wooden cylinder (recommendation: 10cm-
long sections of wooden dowling with a diameter of 1-2cm). Each team should be given an example object to use
during the development of their robot.
 Thrower. The students must construct a robot that can autonomously pick up a small ball (ping pong ball, for
example) and throw it into a bucket or wastepaper bin. The robot must be at least 10cm from the ball at the start of
the demonstration and the bucket must be at least 3m from the ball. Each team should be given a ball to use during
development of their robot.
 Three themes – Design Test Projects
 Sweeper. Robot must hold a broom and sweep a yard of leaves into a corner of the yard.
 Microwave cooker. Robot must pick up a dish and place it into a microwave oven, select a program and when
the microwave completes take the dish out.
 Tea server. Robot must pick up a cup of tea, deliver it to a patient and serve it to the patient on a tray, then
return to pick up the empty cup when patient has finished drinking cup.

10
 User Requirements
 A narrative text (also known as experience blueprint or
customer journey) can be used to describe how the user
interacts with the system at a top level.
 Ideas can be developed using brainstorming techniques
 A Unified Modelling Language Use Case diagram can also
be used to define how the user interacts with the system
visually so the whole project team comes to a common
understanding.
 User Requirements
Narrative Text
 A documented description of a system feature
seen from the end-user perspective. The user
story describes what exactly the user wants the
system to do.
 A <type of user> wants to perform <some goal> so that <some
reason>

 Example: An administrator wants to add

descriptions to products so that users can later
view these descriptions and compare the
products.
 User Requirements
Brainstorming
 Definition
A group creativity technique by which efforts are made to find a conclusion for a specific
problem by gathering a list of ideas spontaneously contributed by its members by
• (a) generate as many ideas as possible;
• (b) prioritize unusual or original ideas;
• (c) combine and refine the ideas generated;
• (d) abstain from criticism during the exercise. 

 Methods
 Brain Writing – separate idea generation from discussion
 Rapid Ideation - a time limit is set for individuals to write down as many thoughts or
ideas around the topic as possible, using any mediums available
 Round Robin - Once the topic is shared, go around the circle one-by-one and have each
person offer an idea until everyone has had a turn
 Starbursting - brainstorming focuses on forming questions rather than answers
 Stepladder - every member in the team to contribute individually before being influenced
by everyone else
 User Requirements
UML Use Case Diagram
Bank
 Provides a high-level overview of the
relationships between actors, different
use cases, and the system
 Use cases. Usually drawn with ovals, use
cases represent different use scenarios
that actors might have with the system
(log in, make a purchase, view items,
etc.)
 System boundaries. Boundaries are
outlined by the box that groups various
use cases in a system.
 Actors. These are the figures that depict
external users (people or systems) that
interact with the system.
 Associations. Associations are drawn
with lines showing different types of
relationships between actors and use
cases.
 Use Cases vs. Functional Requirements
 Consider a cell phone:

Use Cases Internal Functions

 call someone  transmit / receive data
 receive a call  energy (battery)
 send a message  user I/O (display, keys, ...)
 memorize a number  phone-book mgmt.

Point of view: user Point of view: developer /

designer
 From Functional Requirements to
System Design
 Required to get common agreement from all partners on how the system
should be designed:
 Inputs – to system from user input or sensors,
 Outputs – from system to users or actuators
 Functions – derived from functional requirements and subdivided into smaller functions
 Interfaces – the format of the data being transferred between the system functions

Function i/f8 Output

Input i/f1 Function i/f6 5 1
1 1 i/f3
Function i/f5 Function
3 4
Input i/f2 Function i/f4 i/f7 Function i/f9 Output
2 2 6 2
 From Functional Requirements to
Mechanical Design
 Required to get common agreement from all partners on how the system should
be designed:
 Inputs – loads to system in form any power source including human effort, energy from the wind,
water, heat etc., from a chemical reaction or from an electrical, pneumatic or hydraulic device.
 Outputs – forces from change created in the inputs by the mechanism
 Components – of a mechanism converting the input motion and force into an output motion and
force
 Interfaces – Hinge, Weld, Rivet, Adhesive Bond, Screw, Clip

Mechanism Componen i/f8 Output

Input i/f1 Component i/f6
t5
1
1 i/f3
1 Componen i/f5 Component
t3 4

Input i/f2 Component i/f4 i/f7 Componen i/f9 Output

2
2 t6
2
 Technical (or Non-Functional)
Requirements
 Performance is a quality attribute that describes the responsiveness of the system to various user
interactions with it. Poor performance leads to negative user experience. It also jeopardizes system
safety when it is overloaded.
 Availability is gauged by the period of time that the system’s functionality and services are
available for use with all operations. So, scheduled maintenance periods directly influence this
parameter. And it’s important to define how the impact of maintenance can be minimized.
 Scalability requirements describe how the system must grow without negative influence on its
performance. This means serving more users, processing more data, and doing more transactions.
Scalability has both hardware and software implications.
 Reliability defines how likely it is for the software to work without failure for a given period of time.
Reliability decreases because of bugs in the code, hardware failures, or problems with other
system components. To measure software reliability, you can count the percentage of operations
that are completed correctly or track the average period of time the system runs before failing.
 Security requirements ensure that the software is protected from unauthorized access to the
system and its stored data. It considers different levels of authorization and authentication across
different users roles. For instance, data privacy is a security characteristic that describes who can
create, see, copy, change, or delete information. Security also includes protection against viruses
and malware attacks.
 Efficiency of use: the average time it takes to accomplish a user’s goals, how many tasks a user
can complete without any help, the number of transactions completed without errors, etc.
 Intuitiveness: how simple it is to understand the interface, buttons, headings, etc.
 Low perceived workload: how many attempts are needed by users to accomplish a particular task.
 User Testing
 Study goals: The goals should relate to the defined use case requirements.
 Background information: Explain the purpose of the study to participants; tell them
you’re testing the system software, not them; let them know if you’ll be recording the
sessions. Ask them to please think aloud throughout the study so you can understand
their thoughts.
 Tasks and questions: Start by asking participants a couple of background questions to
warm them up e.g. age, technical background, etc. . Then ask them to perform tasks
that you want to test. For example, to learn how well they were able to calibrate a
project, you could have them start on the home page and say, “You are here to
calibrate a robot. Where would you go to do that?” Also consider any follow-up
questions you might want to ask, such as “How easy or difficult was that task?” and
provide a rating scale.
 Conclusion: At the end of the study, you can ask any observers if there are questions
for the participant, and ask if the participant has anything to else they’d like to say.
User Testing
Design of Rating Scale
 The response scale simply represents the distribution of the responses by providing firstly
the number and secondly the type of allowable answers to a question.
 There are various issues to consider, such as the balance, the polarity and the number of
values.
 A scale that has an equal number of positive and negative alternatives is known as
balanced. Such balanced scales are commonly chosen because they tend to produce nearly
normal distributions.

 On the contrary, unbalanced scales are used when it is expected that responses will be
selected from the scale’s extremes.
 Another point to consider is the presence or absence of a scale midpoint. Denying the
ability of neutrality will probably increase the variability towards one extreme or the other.
Additionally, it is sometimes considered impolite to bereft respondents of the choice to be
neutral and force them to select a choice.
 Lastly, the number of response alternatives tends to be based on the degree of
discrimination required. For example, by increasing the number of alternatives, it is possible
to decrease the uncertain responses (neutral). However, this also increases the
questionnaire completion time because the respondent has to examine all alternatives.
Between 5 and 7 is usual.
User Testing
Question Creation and Wording
 Constructing the wording of questions in a
questionnaire requires attention to:
 Vocabulary: It must be clear and precise, avoiding jargon and confusing
terms.
 Negatives: Negative phrases such as "Indicate how often you do not eat
breakfast during a week" may appear confusing and misunderstood by the
reader–the negative word should usually be left out.
 Double-barrelled questions: These are questions involving two questions at
the same time i.e. “Do you like watching cartoons or eating ice cream”. The
respondent may like cartoons but not ice-cream or vice versa and is thus
confusing to answer one way or another.
 Leading questions: These presuppose an event e.g. The statement “Indicate
the sourness of the ice-cream” presumes that the ice cream is de facto
sour. Instead balance the statement “Indicate the level of sweetness or
sourness of the ice-cream”
User Testing
Overall Design of the Questionnaire
 Divide the questionnaire into parts.
 The first part should deal with initial perceptions and how
alternative designs/approaches.
 Following parts should contain questions on the different
identified factors e.g. the storyline, the appeal to human
emotions, the qualities of the characters, the qualities of the set
etc.
 A penultimate part should request the reviewer to assign an
order of importance for each factor.
 Finally, in the last part, the expert enters his/her personal data
such as their specialisation field, time spent on excavations etc.
Most importantly (s)he is asked to give an opinion on the
completeness of the factor list.
User Testing
Example of Questionnaire
User Testing
Participants
 Participants should be recruited taking
into account considerations carefully
depending on what you are trying to
find out:
 Experts/Novices
 Age groups
 Gender
User Testing
Procedure
 Before answering the questionnaire, the participants should
have to read an information page that explains:
 the format of the questionnaire and how to answer questions.
 The background to the artefact that the questionnaire related to.
 In the case of a printed questionnaire, this should be verbally
explained.
 In the electronic version this should be ensured by having to
specifically tick a box in order to accept that they fully
understood the instructions.
 At the end of the questionnaire participants answer questions
about the questionnaire itself such as if they considered the
factor list complete, or if the questionnaire was difficult to fill in.
User Testing
Results
 Pie Charts
 Distributions
 Averages
 Standard Deviations
 Confidence Intervals
 Functional Testing
Typically involves six steps
1. The identification of functions that system is expected to perform from the
functional requirements
2. Create input data based on the function's specifications
3. Determine output based on the function's specifications
4. Execute the test case with test data and then with test equipment (e.g.
robots, sensors and actuators)
5. Compare actual and expected outputs (usually in tabular form)
6. Check whether the application works as per functional requirements
 Example of Functional Testing Table

Sub- Functional Responsible Other participants Input Interfaces Output Interfaces Completions Date Status
System Descriptio Task Leader
n

Function Performs a Person A Person B IP Packets 64 byte frames  1st October 2019 Completed
1 particular
task

Function 2 Performs Person B Person C 64 Byte Frames of IP Bytes 4th October Hardware completed
packet data but waiting for software
another
bug to be identified
task

Etc.
 Technical Experimental Testing
 Aim of the Experiment
 Experimental Setup
 Experimental Procedure
 Presentation of Experimental Results
 Analysis of Results
 Aim of the Experiment
 The aim provides a succinct description of the purpose of your
experiment.
 You will be tasked with verifying a defined technical performance
requirement: How fast, how accurate, how long, how straight, how
high, how far etc.
 For example:
 “The aim of this experiment was to examine how far a robot can throw a ping pong
ball”
 Experimental Setup
 This section outlines the equipment setup and materials
used while conducting your experiment.
 This information is important as it allows for others to
reconstruct your experiment.
 You will need to provide:
 Diagram of the experimental setup
 Enough background that your experiment could be repeated – including
information on: your robot (size weight, height and reach etc.) , what
weight, colour and shape of manipulated objects, what types of pens,
what colours of lines, what type of surfaces etc.
 Experimental Procedure
 Explain what you had to do to perform your experiment:
 Where was your robot positioned?
 Where were you manipulated objects positioned?
 What commands you provided to your program to perform
experiment?
 How many times you repeated the experiment?
 Presentation of Experimental Results
 This section contains the data you obtained during the experiment.
 This data should be presented in a way that makes it easily accessible
to the reader – that is graphically or tabulated where possible.
 Repeat the experiment a number of times to obtain sufficient data to
include confidence intervals.
 If there are multiple experiments to be conducted then these should
be separated into separate sub-sections to increase readability.
 The contents of this section should be observations only; save the
interpretation of your results for the discussion section.
 It is important that any statements in this section are clear, concise
and accurate – avoid using vague terms, instead use specific
language to illustrate your results to the reader.
 For example, say: “The difference in positioning accuracy between using object A and object B
resulted was less than 1%.”
 Rather than: “The difference between the simulation and theoretical results was small.”
 Confidence Intervals
 This fact is known as the 68-95-99.7 rule, or the empirical rule, or the
3-sigma rule:
 About 68% of values drawn from a normal distribution are within one standard
deviation σ away from the mean;
 About 95% of the values lie within two standard deviations;
 About 99.7% are within three standard deviations.
Normal Distribution Confidence Level z*– value

68.2 1.0

80% 1.28

85% 1.44

90% 1.64

95% 1.96

95.4% 2.0

𝜎 E
98% 2.33
𝐶𝐼=𝜇 ± 𝑧
√𝑛 99% 2.58

𝜎 99.7% 3.0
𝐸=𝑧
√𝑛
http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Probability/BS704_Probability12.html
http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Confidence_Intervals/BS704_Confidence_Intervals2.html
http://sphweb.bumc.bu.edu/otlt/MPH-Modules/BS/BS704_Power/BS704_Power2.html
 Plotting Results
with Matlab inputmeasurements.xlsx
clc; clear all;
% Input or Independent Variable
x = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100];
% Output or Dependent Variable
y = xlsread('inputmeasurements.xlsx');
% Calculate mean and Standard Dev
M = mean(y)
S = std(y)
% Select CI by choosing z value
% CI 68%: 2*z*S; z=1;
% CI 95.4%: 2*z*S; z=2 Plot of Target Distance against Actual Distance (CI=95%
120
% CI 99.7%: 2*z*S; z=3
z=2;
% Calculate size of error bar 100
err = 2*z*S/sqrt(10);
% Calculate the margin of error E and the % margin of error
E = (z*S)/sqrt(10); 80
PercentageE= (E./M)*100;
% Plot data with error bars
figure (1) 60

errorbar(x,M,err)
title('Plot of Target Distance against Actual Distance (CI=95%')
40
xlabel('Target Distance')
ylabel('Actual Distance')
% output all your data 20
filename = 'outputdata.xlsx';
xlswrite(filename, [x;y]);
0
0 20 40 60 80 100 120
Target Distance
 Analysis of Results
 The structure and content of your discussion section depends greatly on the
aim of your experiment; thus, there is no “one size fits all” solution to writing
a discussion section.
 Step 1: Link your results to your aim:
 In your aim, you outlined what your experiment set out to examine. Are your results consistent
with the aim outlined at the beginning of your report?
 Step 2: Describe how successful / unsuccessful was experiment. This should
be supported by quantified results. If your experiment failed, you should
analyse how and where it failed and how it could be improved.
 Step 3: Identify any errors or inconsistencies in your results and provide a
rationale or ways to improve the accuracy of your results: What were the
sources of uncertainty or error while conducting your experiment? This can
include things like:
 Equipment tolerances
 Equipment failure
 Measurement errors.
 Conclusions and Recommendations
 Your conclusion should be used to provide a succinct
overview of the experiment, the results achieved and
whether the experiment was successful.
 It may also include any recommendations you have for
further research or refinements to the system design and
the experimental procedure used to obtain your results.
 It is important that you do not introduce any new
material in your conclusion that was not discussed in your
report.