Post on 11-Feb-2022
Applied Software Engineering
Research Group
Definición y análisis de Indicadores de rendimiento de procesos
Adela del Río Ortega Universidad de Sevilla
www.isa.us.es
University of Sevilla
• 65.000 students • 4.600 teaching staff • 500 years
Computer Engineering School
Amador Durán Joaquín Peña David Benavides Beatriz Bernárdez Octavio Martín Manuel Resinas
Pablo Fernández Sergio Segura Carlos Müller Jose María García Pablo Trinidad Adela del Río
Javier Troya Jose A. Galindo Ana B. Sánchez
Antonio Ruiz
Jose Antonio Parejo Antonio Gámez
Applied Software Engineering Research Group www.isa.us.es
Alfonso Márquez Antonio M. Gutiérrez Bedilia Estrada
ISA in numbers
• > 200 Publica3ons in Journals and Conferences
• 3 Interna3onal Patents • 16 So:ware Tools developed • 5 European Research projects • 10 Na3onal Research projects • 5 Research Networks • 35 Public-‐Private Transfer
Research Contracts • 2 Spin-‐offs
25 members 14 Senior Research Staff 4 Research Assistants, 3 PhD students, 4 Software Developers
(>20 former members)
ISA Research Areas
Business Process
Management Cloud and Services
Variability Management
Software Testing
Experimentation
Creating rich models
Finding techniques to analyse them
• Constraint Satisfaction Problems
• Description Logics and Ontologies
• Metaheuristics
ISA Research Areas
Business Process
Management Cloud and Services
Variability Management
Software Testing
Experimentation
Business Processes (BPs) Conference travel management (THESIS)IS
A Re
sear
ch G
roup
ISA Research Group
SubmitPaper
Fill TravelAuthorizat ion
Send TravelAuthorizat ion
Registerat Conference
Any problem?
MakeReservat ions
Accommodation Transport
TravelAuthorizat ion< < filled> >
Sign TravelAuthorizat ion
TravelAuthorizat ion< < signed> >
TravelAuthorizat ion< < stored> >
CheckResponse
TravelAuthorizat ion
< < approved> >
TravelAuthorizat ion< < rejected> >
Research Vice- chancellorship
NoYes
CRISTINA CABANILLAS MACÍAS 1 of 1 21.11.2012
Conference travel management (THESIS)IS
A Re
sear
ch G
roup
ISA Research Group
SubmitPaper
Fill TravelAuthorizat ion
Send TravelAuthorizat ion
Registerat Conference
Any problem?
MakeReservat ions
Accommodation Transport
TravelAuthorizat ion< < filled> >
Sign TravelAuthorizat ion
TravelAuthorizat ion< < signed> >
TravelAuthorizat ion< < stored> >
CheckResponse
TravelAuthorizat ion
< < approved> >
TravelAuthorizat ion< < rejected> >
Research Vice- chancellorship
NoYes
CRISTINA CABANILLAS MACÍAS 1 of 1 21.11.2012
Perspectives in Business Processes
Human resources
Control flow Access Types Read (R) Write (W) Read/Write (R/W)
Data
Data
Business Process Perspectives
Business Process
Resources/ Organisational
IT systems Data/ Informational
Functional
Performance
Control flow/Behavioural
delays caused by committee
number of RFCs per project
percentage of perfective changes
out of approved RFCs
Examples
Process Performance Indicator (PPI)
Quantifiable metrics that allow the evaluation of the efficiency and effectiveness of business processes. They can be measured directly by data that is generated within the process flow and are aimed at the process controlling and continuous optimization.
[G. Chase et al., 2011]
PPI vs KPIs
KPIs
PPIs number of RFCs per project
percentage of
corrective
changes from RFC
approved
percentage of
satisfied
costumers
Design
Instrumen-tation Computation
Evaluation
Design and Analysis
Configuration Enactment
Evaluation
BPM lifecycle PPIM lifecycle
Define PPIs, Connect with BP,
design-time analysis
Implement measurement
points
Calculate PPIs’ values and
monitor PPIs
Identify PPI correlations,
conflicts and predict future behaviour
The PPI Management Lifecycle
How to define PPIs to support the PPI management lifecycle?
How to extract valuable information?
Problem statement
How to define PPIs to support the PPI management lifecycle?
How to extract valuable information?
Problem statement
• High expressiveness
• Automation
• Traceability with BP elements
• Extensible
Features
More details at: Adela del-Río-Ortega et al. “On the definition and design-time analysis of process performance indicators”. Information Systems 38(4): 470-490 (2013)
Visual PPINOT
PPI
Measures
Aggregated Measure Derived Single-Instance Measure
Derived Multi-Instance Measure
Base Measure
What to measure Time
SUM
SUM
Count
State Condition
Data Property Condition
Data content
Connectors
Aggregates Time connectors
Applies to isGroupedBy Uses
IT D
epar
tmen
t
PLan
ning
& Q
ualit
y M
anag
er
RFCReceived
Analyse RFC
Cancel RFC
Approve RFC
Elevatedecision tocommittee
ReportRFC
cancelled
ReportRFC
approved
RFCregistered
RFCcancelled
RFCapproved
Com
mitt
ee
Analyse incommittee
Requester
PPI5
AVG
Avg time of RFC analysis
PPI4
(a/b)*100
Perfective RFCs out of approved
1 2 3
SUMSUM
from
bape
rfec
tive
chan
ges
to
More details under request (BISE journal article under review)
The PPI value must be greater than [or equal to] <lower bound>
Easier and faster than writing whole paragraphs from scratch
Fills placeholders in prewritten sentences
Sucessfully used in RE
Linguistic Patterns
PPI-005 Average time of RFC analysis Process Request for change (RFC) Goals • BG-002: Improve customer satisfaction
• BG-014: Reduce RFC response time MeasureDefinition The PPI is calculated as the average of the duration between the time
instants when activity RFC analysis becomes active and when activity RFC analysis becomes completed
Target The PPI value must be lower than or equal to 1 working day Scope The process instances considered for this PPI are those in Last 100
instances scope Source Event logs of BPMS Responsible Planning and quality manager Informed Chief Information Officer (CIO) Comments Most RFCs are created after 12:00
PPI Template Example
More details at: Adela del-Río-Ortega et al. “Using templates and linguistic patterns to define process performance indicators”. Enterprise Information Systems 10(2): 159-192 (2016)
• How to define PPIs to support the PPI management lifecycle?
• How to extract valuable information?
Problem statement
Which process elements are measured by this PPI?
Average lifetime of approved
RFCs
How do I instrument this
indicator?
Measured By Relationship
Which process elements are involved in this PPI?
Average lifetime of approved
RFCs
This PPI is too costly. We need
to replace it
Involved In Relationship
Rela9onship Opera9ons
Measured By
MeasuredBPElement: Which process elements are measured by this PPI?
MeasuredPPI: Which PPIs are defined over this process element?
Involved In
InvolvedBPElements: Which process elements are involved in a given PPI?
NotInvolvedBPElements: Which process elements are not involved in any PPI?
InvolvedInAllBPElements: Which process elements are involved in all PPIs?
Associated PPI: Which PPIs are involved in a given process element?
Formulate operations in terms of DL reasoning operations (e.g. satisfiability, subsumption, realization, etc.)
DL reasoners • Hermit • Pellet
• Racer • Etc.
DL-based implementation
DL Knowledge Base
More details at: Adela del-Río-Ortega et al. “On the definition and design-time analysis of process performance indicators”. Information Systems 38(4): 470-490 (2013)
Applied to several use cases
• IT department of the Andalusian Health Service • Company for training health professionals • Information and Communication service of the University of
Seville • A part of the administration of the Andalusian Regional
Government
Ongoing work
• SLA modelling for BPO services
• Automatic establishment of links between natural language PPIs and their implementation
• PPI variability management
• PPI Thresholds determination based on execution data
Ongoing work
• SLA modelling for BPO services
• Automatic establishment of links between natural language PPIs and their implementation
• PPI variability management
• PPI Thresholds determination based on execution data
SLA modelling for BPO services 5
Table 2. Penalties definition (in monthly billing percentage) for the FI Service SLA
AFIP Penalty94% AFIP < 95% -1%93% AFIP < 94% -2%92% AFIP < 93% -3%91% AFIP < 92% -4%90% AFIP < 91% -5%
AFIP < 90% -10%
Cont
ract
or
APC
FI requested
FI request
Plan FI Perform FI
FI documentation
required?Create and submit FI
documentationDocumentation
Accepted
Correctionrequired
FI closed
no
FI documentation Correction request
FI documentationacceptation
Fig. 1. BPMN model of Field Intervention (FI) service
have two main features. On the one hand, it must be expressive, i.e. it must allow the137
definition of a wide variety of metrics. On the other hand, it must be traceable with138
the business process so that it enables their automated computation. In addition, it is139
convenient that the metrics are defined in a declarative way because it reduces the gap140
between the SLA defined in natural language and the formalised SLA and decouples141
the definition of the metric from its computation.142
3.3 Service Level Objectives (SLOs)143
These are the assertions over the aforementioned metrics that are guaranteed by the144
SLA and, hence, must be fulfilled during the execution of the service. For instance, the145
running example defines AFIP > 95% as an SLO for AFIP metric of the FI service.146
In general, SLOs can be defined as mathematical constraints over one or more SLA147
metrics.148
3.4 Penalties and rewards149
They are compensations that are applied when the SLO is not fulfilled or is improved,150
respectively. An example is shown in Table 2, which depicts the penalties that apply for151
4
Resolution Time Elapsed time between the technician arrival and the end and closureof the FI.
Documentation Time If documentation, i.e. reports, is required, it is defined as theelapsed time between the end and closure of the FI and documentation submission.If the APC considers such documentation as incomplete or inadequate, it will bereturned to the contractor and documentation time is again activated and computed.
3 Requirements for Modelling SLAs of BPO Services
The requirements for modelling BP SLAs in the context of SLA–aware PAIS have beenidentified after a study of the state of the art in SLAs for both computational and non–computational services, and the analysis of more than 20 different BPO SLAs developedby 4 different organisations in 2 different countries. The conclusion is that four elementsmust be formalized in SLAs for BPO services, namely: 1) the business process; 2) themetrics used in the SLA; 3) the SLOs guaranteed by the SLA; and 4) the penalties andrewards that apply if guarantees are not fulfilled. Next we describe each of them.
3.1 Business process
An SLA is always related to one or more specific services. The way such services mustbe provided is usually defined by describing the underpinning business process, andthis is often done in natural language. Consequently, the formalization of SLAs forBPO services requires the formalization of the business process itself. Note that it is notrequired for the SLA to detail the low level business process that will be enacted by theprovider’s PAIS since most SLAs do not delve into that level of detail and just focuson main activities and the consumer–provider interaction (cf. Fig 1 for the high–levelbusiness process of the running example). However, it should be possible to link thishigher level business process to the lower level business process enacted by the PAIS.
3.2 SLA metrics
These are the metrics that need to be computed so that the fulfilment of the SLA canbe evaluated. For instance, in the running example, response time, presence time, orAFIP are examples of such metrics. The mechanism used to define these metrics must
Table 1. Committed times by the contractor (in hours) for the FI Service SLA
CriticalityLevel
ResponseTime
PresenceTime
ResolutionTime
Document.Time
Timetable Calendar
Critical 0.5 4 2 4 8:00 – 20:00 LocalHigh 2 8 4 12 8:00 – 20:00 LocalMild 5 30 6 24 8:00 – 20:00 LocalLow 5 60 8 48 8:00 – 20:00 Local
3
reports on work related to the definition of SLAs for BPO services. Finally, conclusionsare detailed in Section 7.
2 Running Example
Let us take one of the BPO SLAs to which our approach has been applied as runningexample throughout this paper. The SLA takes place in the context of the definitionof statements of technical requirements of a public company of the Andalusian Au-tonoumous Government, from now on Andalusian Public Company, APC for short.Statements of technical requirements are described in natural language and include in-formation about the services required as well as their SLA. Although the running exam-ple includes one service only, further information on this or the rest of services, as wellas on other application scenarios, is available at http://www.isa.us.es/ppinot/caise2015.
The statement of technical requirements document of this example is defined forthe Technical Field Support for the Deployment of the Corporative TelecommunicationNetwork of the Andalusian Autonomous Government. It is presented in a 72–page doc-ument written in natural language including the SLAs defined for five of the requiredservices, namely: 1) field interventions; 2) incidents; 3) network maintenance; 4) instal-lations and wiring; and 5) logistics. In particular, we focus on the field interventions (FI)service. The term field intervention makes reference to the fact of requiring the pres-ence of a technician at any headquarter of the APC for different reasons: troubleshootingtechnical assistance, installations supervision or restructure, for instance.
From a high–level perspective, the FI service can be defined as follows: the APC re-quires an FI, which can have different levels of severity, from the contractor staff. Then,the contractor plans the FI and performs it at headquarters. In some cases, it is necessaryfor the contractor to provide some required documentation and, if such documentationis considered incomplete or inadequate by the APC, it needs to be resubmitted by thecontractor until it fulfils the APC’s quality requirements.
For this service, the statement of technical requirements document presents the fol-lowing information: 1) the committed times by the contractor (see Table 1); 2) the gen-eral objective defined for FIs —the SLO of the SLA— represented as AFIP > 95%,where the AFIP (accomplished FIs percentage) metric is defined as:
AFIP =# accomplished FIs
# FIs⇥ 100
and 3), the penalties applied in case the SLO is not accomplished (see Table 2). Thesepenalties are defined over the monthly billing by the contractor for the FI service. Inaddition, the statement of technical requirements document presents the following def-initions for the referred times in Table 1:
Response Time Elapsed time between the notification of the FI request to the contrac-tor and its planning, including resources assignment, i.e. technicians.
Presence Time Elapsed time between resource (technician) assignment and the begin-ning of the FI, i.e. technician arrival.
>95%
5
Table 2. Penalties definition (in monthly billing percentage) for the FI Service SLA
AFIP Penalty94% AFIP < 95% -1%93% AFIP < 94% -2%92% AFIP < 93% -3%91% AFIP < 92% -4%90% AFIP < 91% -5%
AFIP < 90% -10%
Cont
ract
or
APC
FI requested
FI request
Plan FI Perform FI
FI documentation
required?Create and submit FI
documentationDocumentation
Accepted
Correctionrequired
FI closed
no
FI documentation Correction request
FI documentationacceptation
Fig. 1. BPMN model of Field Intervention (FI) service
have two main features. On the one hand, it must be expressive, i.e. it must allow thedefinition of a wide variety of metrics. On the other hand, it must be traceable withthe business process so that it enables their automated computation. In addition, it isconvenient that the metrics are defined in a declarative way because it reduces the gapbetween the SLA defined in natural language and the formalised SLA and decouplesthe definition of the metric from its computation.
3.3 Service Level Objectives (SLOs)
These are the assertions over the aforementioned metrics that are guaranteed by theSLA and, hence, must be fulfilled during the execution of the service. For instance, therunning example defines AFIP > 95% as an SLO for AFIP metric of the FI service.In general, SLOs can be defined as mathematical constraints over one or more SLAmetrics.
3.4 Penalties and rewards
They are compensations that are applied when the SLO is not fulfilled or is improved,respectively. An example is shown in Table 2, which depicts the penalties that apply for
Metrics
Penalties & Rewards
Service Level Objective (SLOs)
SLA modelling for BPO services
Agreement
Terms
Name
Context
Service Terms
Guarantee Terms
Service description terms
Service properties
Service references
Business Process
SLA Metrics
SLO, Penalties &
Rewards
Structure: WS-Agreement
From Computational SLAs
Published at: Adela del-Río-Ortega et al. “Modelling Service Level Agreement for Business Process Outsourcing Services”. CAiSE 2015: 485-500
Ongoing work
• SLA modelling for BPO services
• Automatic establishment of links between natural language PPIs and their implementation
• PPI variability management
• PPI Thresholds determination based on execution data
Automatic establishment of links between natural language PPIs and their implementation
• The formulation of PPIs is typically a managerial concern • Monitoring PPIs requires a technical perspective on a
process
Natural language PPI description
Process model
Alignment
Automatic establishment of links between natural language PPIs and their implementation
Process model
Alignment Approach
Decision tree
1. Type classification
2. PPI parsing
3. Alignment to model
Classification
Typeindicators
Extractedphrases
Measuretype Alignment
Unstructured PPI description
Published at: Han van der Aa et al. “Narrowing the Business-IT Gap in Process Performance Measurement”. CAiSE 2016: 543-557
Ongoing work
• SLA modelling for BPO services
• Automatic establishment of links between natural language PPIs and their implementation
• PPI variability management
• PPI Thresholds determination based on execution data
PPI variability management
DIM-2
DIM-1
Dimensions: result of analysis Performance perspective subject to variation
VAR
Needs of models and tools
VAR
VAR
First Step
• Avoid redundancy • Reduce efforts • Increase clarity
Published at: Bedilia Estrada-Torres et al. “Identifying Variability in Process Performance Indicators”. BPM (Forum) 2016: 91-107
Ongoing work
• SLA modelling for BPO services
• Automatic establishment of links between natural language PPIs and their implementation
• PPI variability management
• PPI Thresholds determination based on execution data
PPI Thresholds determination based on execution data
Lagging Indicators Measures goal
accomplishment, easy to measure but hard to
influence.
Leading Indicators Predict goal achievement,
you can influence them but can be more difficult to
measure.
Method for PPI Threshold Determination
Data Preprocessing
Relationship checking
Threshold extraction: ROC Curve
Error probabilities:
Bender method
Thresholds validation
Lag indicator
Lead indicators
Threshold
To be published: Adela del-Río-Ortega et al. “Enriching decision making with Data-Based Thresholds of Process-Related KPIs”. CAiSE 2017: TBD
Data