Post on 19-Jun-2020
Jornada sobre
Innovación y Tendencias
en la Gestión de Requisitos
9 de mayo – MadridSede Madrid Network
organizan:
#gestionrequisitos2016
Mejora en la calidad de requisitos:Un caso de éxito
José M. Fuentes
1
Presenter profile
• José M. Fuentes
• jose.fuentes@reusecompany.com
• Co-founder of The REUSE Company
• Current Chief Commercial Officer of The REUSE Company
José M. FuentesCCO
• Member of AEIS Board (Spanish Chapter of INCOSE)
• Member of INCOSE Requirements Engineering WG
• Contributor to INCOSE Guide for Writing Requirements
• PM for TRC in EU Research Projects:
– AUTOSoft project
– CRYSTAL Project
– AMASS Project
– REVaMP22
The REUSE Company is a tool
manufacturer providing Knowledge-Centric
solutions to critical systems engineering
such as requirements quality management
or systems knowledge reuse.
The Requirements Quality Suite (RQS), a
tool to manage the requirements V&V
activity, is the most well-known product.
TRC is a SME based in Madrid (Spain)
Parque Tecnológico Legatec.
The REUSE Company profile
3
Contents
• Introduction
– The impact of poor quality in our projects
– Requirements Quality Analysis
• Practical case study
– Goals, inputs and expected outputs
– Tools benchmark
– The PoC Process
– PoC results
4
The impact of poor quality
The impact of poor quality projects
6
Why Requirements Quality Analysis?
7
Why Requirements Quality Analysis?
Doing the right thing right (verification)
http://www.theguardian.com/world/2014/may/21/french-railway-operator-sncf-orders-trains-
too-big
http://elpais.com/elpais/2015/02/04/inenglish/1423052376_326956.html8
The FOCUS:
Requirements Quality Analysis
Why Requirements Quality Analysis?
Source : INCOSE SE Handbook V3.2
95%
85%
70%
Time
Cu
mu
lati
ve p
erce
nta
ge
Life
cylc
e C
ost
Operations through Disposal
100%Production
and test
50%
8%Design
15% 20%Concept
Commited Costs
3-6x
500-1000x
20-100x
Development
10
Systems and Requirements
Engineering life-cycles
CONOPS
Stakeholders
Requirements
System
Requirements
System
Design
Equipment
Requirements
Equipment
Design Equipment
Verification
System
Equipment
System Verification
Product
Product Verification
11
Systems and Requirements
Engineering life-cycles
Elicitation Analysis Specification Validation
close gapsclarify
rewrite
re-evaluate
confirm and correct
Source: Karl Wiegers
12
Systems and Requirements
Engineering life-cycles
CONOPS
Stakeholders
Requirements
System
Requirements
System
Design
Equipment
Requirements
Equipment
Design Equipment
Verification
System
Equipment
System Verification
Product
Product Verification
Requirements
Verification
Requirements
Verification
Requirements
Verification
Design
ValidationDesign
Verification
Requirements
Validation
13
Practical Application at
• Objectives
– Perform correctness, completeness and consistency analyses of requirements (individually and collectively) to improve the Quality of requirements specifications
– Assess the computer-aided requirements authoring feature to accelerate the learning curve of new practitioners (or improve the capability of current practitioners) in requirements development
• Goal
– Exonerate engineers from format concerns (structure) and allow them to concentrate on content (essence of requirements): technical data useful for design
Quick Proof of Concept on Requirements
Quality Improvement
15
• External audits results
– “… Requirements Characterization is not complete: Derived/uncovered requirements justification, Contribution, Categories (technical vs non-technical), V&V Methods…
– …V&V Plan is not complete: Verification activities, or agreed alternate practices (waivers) and associated deliverables…”
• CMMI for Development
– Requirement Development process area – SG 3 Analyze and Validate Requirements
• “… Analyze requirements to determine whether they satisfy the of higher level requirements.Analyze requirements to ensure that they are complete, feasible, realizable, and verifiable…”
– Verification process area – SG 2 Perform Peer Reviews
• “… Establish and maintain checklists to ensure that the work products are reviewed consistently...Rules of construction , Completeness, Correctness…”
Also, a means to improve current practices
16
Previous results
Most common requirements defects. Source: Gauthier Fanmuy - the RAMP project: - AFIS
17
Requirements quality characteristics vs
quality metrics• Well-known requirements quality characteristics
• IEEE Std. 830:– Correct
– Unambiguous
– Complete
– Consistent
– Ranked
– Verifiable
– Modifiable
– Traceable
• ESA PSS-05,
ISO/IEC 29148, others:
– Pretty much the same characteristics
• SMART:– Specific
– Measurable
– Achievable
– Relevant
– Traceable
"I believe that this nation should commit
itself to achieving the goal, before this
decade is out, of landing a man on the
Moon and returning him safely to Earth"
18
Quality characteristics to measure
• How to… Perform CCC
Define Metrics for the Rules proposed in the
INCOSE Guide + Others
INCOSE Requirements Working Group
Guide for Writing Requirements V4
Characteristic Cxx – Characteristic name
Rationale: xxxx
Strategy: xxxx
Rules that help establish this characteristic:
Rxx - /Section/Rule nameAvoid xxxxRyy - /Section/Rule nameAvoid yyy
19
• Quality of individual requirements
– Correctness Requirement statement is understandable by humans (TRC)
Requirement statement’s structure is agreed with the SE dept. (TRC)
Requirement Statement corresponds to user request (ISO TR 24766)
The requirement contains all the information necessary for design and implementation (Alstom)
• Quality of requirements sets
– Consistency Absence of conflict among a set of requirements (ISO TR 24766)
A set of requirements is consistent if each necessity (requirement) is expressed in one and only one requirement (TRC)
– Completeness The set of requirements represents a complete definition of the product
(ISO TR 24766)
Completeness by comparing project content vs project content (TRC)
Managing Requirements Quality: CCC approach
20
Requirements Quality Analysis Benchmark
INITIAL SOURCE:
• AFIS « RAMP » project 2010-2012
• Evaluations by AIRBUS 2012
21
Requirements Quality Suite
The Requirements Quality Suite (RQS) intends to tackle requirements quality
management by offering a set of tools and processes
Automatic measurement of requirements quality metric
Support to Requirements Authoring
RQS models requirements quality metrics using the CCC approach (Correctness,
Consistency and Completeness)
Requirements Quality Analyzer (RQA):
to setup, check and manage the quality of a requirements specification.
Requirement Authoring Tool (RAT):
to assist authors while they are creating or editing requirements.
Knowledge Manager (KM):
to manage knowledge around a requirements specification: the ontology it is based on, the structure of the requirements to be used in the project, the communication between authors and domain architects.22
• IBM DOORS ©
• PTC Integrity ©
• CATIA Reqtify ©
• OSLC
• VISURE Requirements ©
• Microsoft Excel ©
• XML file
Near future:
• Microsoft Word ©
• Siemens Teamcenter ©
RQS – Requirements Quality Suite: connectors
23
RQS – Requirements Quality Suite: languages
• RQS is highly dependent of the language of the
requirements
• Languages supported so far:
24
• Correctness metrics are quantitative
• Correctness metric values are calculated counting items
– Example: In the Metric Text length in words the metric counts the number of words; then the QF transforms it into a quantitative value
• The process is simplified by using interval (step, discrete) quality functions
• Metrics use one of the following quality functions:
Managing the metrics Quality Functions (QF)
25
textLength()
QHigh
Med
Low
1 5 10 30 50
Quality Management Definition: Maturity
lifecycle
WHITE Belt
Metrics
YELLOW Belt
Metrics
ORANGE Belt
Metrics
BLUE Belt
Metrics
BROWN Belt
Metrics
BLACK Belt
Metrics
GREEN Belt
Metrics
DEMANDINGLEVEL
textLength()
QHigh
Med
Low
1 5 10 30 50
textLength()
QHigh
Med
Low
1 4 6 2526
Specification quality assessment w.r.t. the
Maturity lifecycle
DEMANDING LEVEL
WHITE BeltMetrics
YELLOW BeltMetrics
ORANGE BeltMetrics
GREEN BeltMetrics
BROWN BeltMetrics
BLACK BeltMetrics
BLUE BeltMetrics
CORRELATED LEVELS OF RESULTS
27
Specification quality assessment: The Goal
WHITE BeltMetrics
YELLOW BeltMetrics
ORANGE BeltMetrics
GREEN BeltMetrics
BROWN BeltMetrics
BLACK BeltMetrics
BLUE BeltMetrics
DEMANDING LEVEL
CORRELATED LEVELS OF RESULTS
28
Specification quality assessment: The Path
TIME
WHITE BeltMetrics
YELLOW BeltMetrics
ORANGE BeltMetrics
GREEN BeltMetrics
BROWN BeltMetrics
BLACK BeltMetrics
BLUE BeltMetrics
29
Specification quality assessment: Maturity level by depts.
or Teams
WHITE BeltMetrics
YELLOW BeltMetrics
ORANGE BeltMetrics
GREEN BeltMetrics
BROWN BeltMetrics
BLACK BeltMetrics
BLUE BeltMetrics
TIME
30
The Knowledge Base
31
Terminology layer
Valid terms, forbidden terms, other NL terms, Syntactic clustering types, everything as concepts
Thesaurus layer
Relationships among concepts (hierarchies, associations, synonyms…) as well as semantic clusters and relationship typesPatterns layer
Matching Patterns
Formalization layer
Semantic formalization
Inference layer
For decision making (e.g. consistency, completeness)
• Patterns:
– Represents the structures every correct requirement should
meet
– Different types of requirements different patterns (templates)
– Customizable for every domain, customer and content of each
requirements document
– Libraries with sets of patterns
– Represented as a sequential set of restrictions: placeholders
Examples of requirements metrics: patterns
32
When <Event> <Component> Shall <Action> <Object> Time_constraint
Knowledge Base: Example
33
A380 A350 System Operate Temperature Environment PressureControlled
Vocabulary
A380 A350
<<Aircraft>> “ Greater than (>) “
Operate Work
<<Operation>>
<<Aircraft>> Shall <Operation> <<Minimum>>At Environment Of[MEASUREMENT
UNIT]NUMBER
temperature“ Greater than (>) “
ºC-70
Patterns
Temperature Pressure
Environment
Temperature [-60ºC , +60ºC]“ Operation Range “
Inference
Rules NUMBER “ Lower than (<) “ -60º NUMBER “ Greater than (>) “ +60º||
Thesaurus
Formalizations The aircraft shall be able to operate
at a minimum temperature of -70º C
If ºC ºC
“ Lower than (<) “
Shall
At a minimum
<<Minimum>>
At a minimum Of
• The REUSE Company has developed IT solutions that attempt to
understand, formalize, represent, reason-about and search-for all
kinds of knowledge assets
• Using: Terminology Control, Patterns, Graphs and Natural language
Processing
Knowledge Base: Requirements Patterns
matching
34
UR044 : The Radar shall identify hits at a minimum rate of 10 units per second
The <DEVICE> Shall <ACTION> <ITEMS> <MINIMUM>At Rate of<RATE
VALUE>[NUMBER]
Radar Hits
<<Detect>>
10 Units per Second
<<Minimum Value>>
Hits
• To promote requirements reuse
• To detect duplicates
• To provide quick access to related requirements
Knowledge base: semantic search engine
35
The PoC Process
Process: Work with one Requirements
Specification
37
OriginalReqs.
Specification
INITIAL QUALITY ASSESSMENT
OriginalReqs.
Specification
QUALITY METRICS
DEFINITION
OrganizationKnowledge Base
V1
SPECIFICATION UPDATE
Reqs.Specification
Rev 1
OriginalReqs.
Specification
OrganizationKB V1
OrganizationKB V2
QualityResults
OrganizationKB V0
1st QUALITY ASSESSMENT WITH
BELTS
OriginalReqs.
Specification
OrganizationKB V1
2nd QUALITY ASSESSMENT WITH BELTS
Reqs.Specification
Rev 1
OrganizationKB V2
The PoC Results
Results: Initial Quality Assessment without belts
39
Original Specification
All out of the box metrics enabled 32 Metrics (INCOSE + TRC)
329 Requirements
% of Requirements
Effort centred in metrics for Correctness
Results: Developed Colored Belts
40
WHITE Belt
YELLOW Belt ORANGE Belt
20 Metrics correctness 31 Metrics correctness 51 Metrics correctness
+
2 Metrics completeness
Results: Developed Ontology
Terminology
Patterns
1 381 unclassified terms after first indexation
70 new organization specific patterns
41
Results: First Quality Assessment with belts
42
Original
Specification329 Requirements
% of Requirements
WHITE Belt YELLOW Belt ORANGE Belt
Results: Specification modification (by experts)
43
WHITE Belt
Metrics
329 Requirements 438 Requirements
66.26% Modified Reqs.
Final Specification
% of Requirements % of Requirements
Original Specification
Results: Comparison original vs modified
specification
44
Final Specification
Original Specification
White Belt Original vs. Final Spec.
• Achieved results
– A tool-supported requirements quality process
– A gradual set of quality metrics (belts)
– A Formal Reusable Knowledge Base
– ~70 new Patterns
– One improved Requirements Specification
– An Alstom “Guide for Requirements Authoring”
• Resources
– 2 participants from Alstom, 2 participants from TRC
– 1.5 effective months in calendar: 3.37 PM
Learning and mastering of the tool suiteDefinition of Metrics, Generation of OntologyModifications of requirements
Conclusion
45
0
50
100
150
200
250
White belt Yellow belt Orange belt
PoC Hours
REQUIREMENTSQUALITY
ASSESSMENT
Specificationof Type 2
CustomerKB V2
QualityResults
REQUIREMENTSQUALITY
ASSESSMENT
Specificationof Type 3
CustomerKB V2
QualityResults
• Decision to apply the white belt maturity to
analyze the quality of other specifications
– Fine-tune the metrics
• Decision to enhance the current Requirement
Engineering Process by including
requirements verification activities
• Decision to launch a real-scale pilot project:
real industrial project and staff, IT
infrastructure, measurement of performances
Conclusions of the PoC
46
Detalles de Contacto
José M. Fuentes
jose.fuentes@reusecompany.com
+34 912 17 25 96
@ReuseCompany
Mejora en la calidad de requisitos
47