ISO/IEC JTC 1 SC 42 Artificial Intelligence - Working Group 4
Use Cases & Applications
   03/29/2024

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 13 ID: Use Case Name: AI solution to identify automatically false positives from a specific check for "untranslated target segments”"from an automated quality assurance tool
Application
Domain
Other
Deployment
Model
Cloud services
StatusPoC
ScopeThe scope of this use case is limited to automated linguistic quality assurance tools, but the outcome of this use case could be applicable to other areas, such as for example: Machine Translation, automated post-editing, Computer Aided Translation Analysis and pre-translation, etc. This use case will be relevant for contents across any domain.
Objective(s)To reduce the number of false positive issues for check for untranslated target segment for bilingual content with in-house automated quality assurance tool.
Short
Description
(up to
150 words)
In the future, we aim to build an AI solution that could automatically identify likely false positives issues from the results of the "check for untranslated target segments" following an approach where we could use machine learning based on already identified false positives by our users. The expected outcome would be to increase end user’s productivity when reviewing automated quality assurance findings and to change user behaviour to pay more attention to this type of issues by reducing the number of false positives in 80%. In addition, we would like to reduce the amount of time, we spent on a yearly basis on refining this check manually based on users' feedback.
Complete Description Untranslated target segments contain characters, symbols, and words that remain the same in source and target language. These segments can contain, numbers, alphanumeric content, numbers, code, e-mail addresses, prices, proper nouns, etc. or any combination of those. On a yearly basis, this check produces over 1 Million potential issues across over 50 different languages. Refining this check manually based on annotated false positive data for each specific customer and product and for specific language pairs is very costly, and the coverage is never sufficient, as new content is constantly produced and there are always new opportunities for refining this check via code. In addition, because of the high proportion of false positives over (95.5%) our translators tend to ignore the output from this valuable check and in many cases, we suspect that valid relevant issues for situations when there are real forgotten translations are missed. There are typically three types of false positives for this type of check: 1) Language specific false positives, for example for situations where source and target segment need to be the same as the words from these segments are "cognates" with the same meaning. For example:
Fig.1
2) Customer profile specific false positives, for example situations where certain segments are to be left untranslated based on specific guidelines from the customer, for example for segments that jut consist of Company names, Product Names or specific words and segments that have been determined as not to be translated by our customer:
Fig.2
3) Segments that remain the same in source and target, because they act as special type of entities with some special meaning, for example: alphanumeric segments, for example part numbers, placeholders, code.
Fig.3
The idea is to create an AI solution that can automatically identify results from the "check for untranslated target segment" that are likely to be a False Positive. With this solution, we expect to reduce the number of potential issues presented by this check to our end users in 80%. This way our end users can focus their efforts on those potential issues that are more likely to be valid corrections because there could have been a forgotten translation. In addition, we will be able to increase the productivity of our end users when reviewing automated quality assurance potential issues from their bilingual content evaluation, and we will be able to save costs internally as we won't have to manually implement code changes in this check based on manual analysis of our data based on user's annotation.
StakeholdersCustomers, Translation partners, end users of the translated content.
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Bias from changes in requirements on the customer’s end or inappropriate training data.
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Recognition
Method(s)Machine Learning
Hardware
Topology
Terms &
Concepts Used
Machine Learning
Standardization
Opportunities
Requirements
Challenges
& Issues
Challenges: Try to achieve eventually 80% of the accuracy of linguists when identifying false positives for untranslated target segments, preventing as much as possible false negatives. Issues: segmentation of false positive data by Customer and Product profile could be challenging.
Societal Concerns Description Not applicable
SDGs to
be achieved
Data Characteristics
Description Data from end user identification of false positives and valid corrections for the "untranslated target segment check" results of Moravia QA Tools.
Source RWS Moravia Analytics Portal (https://analytics.moravia.com/Dashboard/459 )
Type Structured content in a table with additional metadata fields (source segment, target segment, source language, target language, valid correction, false positive, customer and product profile, frequency)
Volume (size) Data for last 18 months
Velocity Every hour
Variety Data types will be the same but there would be different variables to be considered (source language, target language, customer and product profile)
Variability
(rate of change)
No changes
Quality End-user dependent

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 45 ID: Use Case Name:  Anomaly Detection in Sensor Data Using Deep Learning techniques 
Application
Domain
 Maintenance & support 
Deployment
Model
 Hybrid or other (Cloud or on premise deployment) 
Status PoC 
Scope Temporal Data captured from sensors 
Objective(s) Identify Anomalies and Events by learning the temporal patterns of sensor data, based on Deep Learning techniques 
Short
Description
(up to
150 words)
 Mechanical devices such as engines, vehicles, aircrafts, etc., are typically instrumented with numerous sensors to capture the behaviour and health of the machine. The sensors temporal data has several complex patterns that are very hard to identify with traditional methods. We have proposed the use of Deep Learning algorithms for analysing such temporal patterns for anomaly/event detection, diagnosis, root cause analysis. Algorithms proposed so far are LSTM-AD, EncDec-AD, online RNN-AD. We used industrial datasets wherever possible and publically available datasets in other scenarios. In most of the cases, our algorithms were significantly better than other methods. 
Complete Description Mechanical devices such as engines, vehicles, aircrafts, etc., are typically instrumented with numerous sensors to capture the behaviour and health of the machine. However, there are often external factors or variables which are not captured by sensors leading to time-series which are inherently unpredictable. For instance, manual controls and/or unmonitored environmental conditions or load may lead to inherently unpredictable time-series. Detecting anomalies/events in such scenarios becomes challenging using standard approaches based on mathematical models that rely on stationarity, or prediction models that utilize prediction errors to detect anomalies.
LSTM-AD
Our Work started with Stacked LSTM network which is trained on non-anomalous data and used as a predictor over a number of time steps. The resulting prediction errors are modeled as a multivariate Gaussian distribution, which is used to assess the likelihood of anomalous behavior. The efficacy of this approach was demonstrated on four datasets: ECG, space shuttle, power demand, and multi-sensor engine dataset.
EncDec-AD
As an extension to the prior work we proposed a Long Short Term Memory Networks based Encoder-Decoder scheme for Anomaly Detection (EncDec-AD) that learns to reconstruct normal time-series behavior, and thereafter uses reconstruction error to detect anomalies. We experimented with three publicly available quasi predictable time-series datasets: power demand, space shuttle, and ECG, and two real-world engine datasets with both predictive and unpredictable behavior. We had shown that EncDec-AD is robust and can detect anomalies from predictable, unpredictable, periodic, aperiodic, and quasi-periodic time-series. Further, we showed that EncDec-AD is able to detect anomalies from short time-series (length as small as 30) as well as long time-series (length as large as 500).
Online-AD
The common approach of training one model in an offline manner using historical data is likely to fail under dynamically changing and non-stationary environments where the definition of normal behavior changes over time making the model irrelevant and ineffective. We described a temporal model based on Recurrent Neural Networks (RNNs) for time series anomaly detection to address challenges posed by sudden or regular changes in normal behaviour. The model is trained incrementally as new data becomes available, and is capable of adapting to the changes in the data distribution. RNN is used to make multi-step predictions of the time series, and the prediction errors are used to update the RNN model as well as detect anomalies and change points. Large prediction error is used to indicate anomalous behaviour or a change (drift) in normal behaviour. Further, the prediction errors are also used to update the RNN model in such a way that short term anomalies or outliers do not lead to a drastic change in the model parameters whereas high prediction errors over a period of time lead to significant updates in the model parameters such that the model rapidly adapts to the new norm. We demonstrate the efficacy of the proposed approach on a diverse set of synthetic, publicly available and proprietary real-world datasets.
StakeholdersMaintenance and support functions, Monitoring, Procurement 
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Prediction
Method(s)Deep Learning 
Hardware
Topology
Terms &
Concepts Used
Deep Learning, LSTM, encoder-decoder, Temporal data 
Standardization
Opportunities
Requirements
Challenges
& Issues
  • Sensor data collection
  • Noisy Data
  • Data with missing temporal features
  • Rarity of Anomalous Data
Societal Concerns Description
SDGs to
be achieved
Industry, Innovation, and Infrastructure 
Data Characteristics
Description Multiple datasets(publically available, real industrial) were used 
Source
Type Temporal data 
Volume (size)
Velocity
Variety Space shuttle, ECG, Engine, Power demand 
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 19 ID: Use Case Name:  AI to understand adulteration in commonly used food items 
Application
Domain
Wellness 
Deployment
Model
StatusInitial validation done 
ScopeUnderstand the patterns in hyperspectral / NIR or visual imaging specifically for adulteration in milk, banana and mangoes 
Objective(s)To device a simple , cost effective tool to identify the adulteration in food items at point of purchase 
Short
Description
(up to
150 words)
Food adulteration is one of the big evil of modern society. Hyperspectral technology was evaluated to find out adulteration in food items 
Complete Description Food adulteration is becoming menace specially with adulterants that are either carcinogenic or harmful to body parts like kidney. To give few examples, Milk is adulterated with Soda, Urea and detergents. Whereas mangoes and bananas are quickly ripened by calcium carbide and so on. Common man cannot live without these items. There is no frugal way to identify these type of adulterations. Experiment of controlled adulteration was done and hyperspectral reflectance reading were taken. AI helped to find the patterns in hyperspectral signature and was able to reliably classify ( 90% ++) samples that were unadulterated and adulterated.
Stakeholders
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Prediction 
Method(s) Machine learning 
HardwareHyperspectral camera
Topology
Terms &
Concepts Used
Machine Learning 
Standardization
Opportunities
Requirements
Challenges
& Issues
 Challenges: Large scale data collection, Miniaturization of frugal NIR / Hyperspectral senso 
Societal Concerns Description Adulterated milk is hazard for children, many aliments including cancer / kidney failures due to consumption of adulterated food. If the AI system is rolled out and taken as reliable then it should be able to perform in all cases and scenarios.
SDGs to
be achieved
Data Characteristics
Description Hyperspectral signatures ( 300 nm to 1300 nm @ 30 nm band) 
Source Hyperspectral camera 
Type
Volume (size) about 500 samples 
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 48 ID: Use Case Name: Value-based Service
Application
Domain
Manufacturing
Deployment
Model
Hybrid deployment: Cloud and on-premise deployment in the production field
StatusPoC
ScopeProcess and status data from production and product use sources are the raw materials for future business models and services.
Objective(s)The objective of this use case is the provision of remote services for product and production based on (generic) service platforms. This use case can be seen as a fundament for the deployment of arbitrary AI remote services.
Short
Description
(up to
150 words)
Service platforms collects data from product use – for example machines or plants – and analyses and processes this data to provide tailor-made individualized services, e.g. optimized maintenance at the proper time, or the timely provision of the correct process parameters for a production task currently being requested. Companies offering these services (service providers) occupy the interface between the product provider and the user.
Complete Description Use Case description taken from [1,2,3]. In the consumer area, the increased interconnectivity of users which has made it possible to collect user data has made a whole new range of services possible. For example, navigation systems in our cars not only determine the shortest route, but also the quickest, as the traffic situation is assessed in real time based on movement data from other users. Entertainment media is no longer purchased rather made available as needed using streaming services. The services offered extend beyond simply making the products available. The individual customer receives optimized offers, based on user data: the quickest route during rush hour, or music tailored to that customer’s taste.
Similar developments are occurring in an increasingly interconnected industrial environment. Services that go significantly beyond simply providing a production unit – a contemporary example is leasing – are gaining in importance and are changing the classic value-added processes and business models.

Key aspects
At the heart of this application scenario are IT platforms that collect data from product use – for example machines or plants for production purposes – and analyze and process this data to provide tailor-made individualized services. This could include for example optimized maintenance at the proper time, or the timely provision of the correct process parameters for a production task currently being requested. The collected data could be product parameters, for example the machines and plants required for manufacture, the product status information, or data from the production process or the upstream supply process. Even the characteristics of the processed raw materials or the parts of the product could be included. The goal is to use this data as a raw material for optimizing products and production processes and for new services. This can help to not only improve existing value chains but also perhaps create new value-added elements.

Effect on value chains
The industrial environment today is influenced in principle by two actors – the product provider (i.e. manufacturers of production facilities and service providers) and the customer (product users, i.e. production facility operators), who work together with varying degrees of intensity.

With the introduction of Value-Based Services an additional actor enters the scene, operating IT platforms that it uses to provide new services to both classic partners. This platform operator could be a new element of the value chain, that is, an autonomous company. However, this role could be taken on by product providers by increasing their value added compared with the current situation.

Product providers make their product data and parameters available. On the basis of all of this user data, new services can now be developed, such as individual optimized maintenance or specific operating and process parameters that optimize or even expand production capabilities of the existing infrastructure. The companies offering these services (service providers) occupy the interface between the product provider and the user. The result is that the share in the value chain spanning from the product provider to the user can be shifted significantly, compared with the situation today. The user can then distinguish between the products by considering the accompanying services or the possibility of expanding those services even after purchasing the product, and no longer primarily by the (physical) specifications mandated by the product provider. This makes it very attractive for the product provider to use such platforms and to offer new services on them.

Value added for participants
In this application scenario the value added for the product provider stems from the availability of a multitude of process data from various application scenarios, which the user can apply to further development of its product port-folio. As an operator of related IT platforms, the product provider can offer new services. In this way, it strengthens customer loyalty and increases its portion of value added.

StakeholdersCustomer (product user), platform provider, service provider, product provide
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Reasoning and autonomous problem solving in the platform, services based on the platform use AI features, e.g. for predictive maintenance, data semantics (cf. [5,6] for an overview)
Method(s)
Hardware
Topology
Terms &
Concepts Used
Standardization
Opportunities
Requirements
Standardization needs for setting up this use case is currently under further investigation. Some initial intentions on standardization needs are the following: For this use case, standardization can be seen as enabler because an agreement on a (small set of) communication protocols would facilitate to connect to the platform and use this protocol also for device2device communication. Since services running on a platform are not aware of an implicit sematic of data sources (machines, sensors, actuators, …), an explicit semantic or a common vocabulary is need describing data and enable reasoning about machine states on premise (on the machine/edge) as well as on the cloud. For cloud2cloud communication and cloud federation, further interoperability standards are required on communication level as well as on data semantics level.
Challenges
& Issues
Societal Concerns Description
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 49 ID: Use Case Name: AI solution for traffic signal Optimization based on multi-source data fusion
Application
Domain
Transportation
Deployment
Model
Cloud services
StatusIn operation
ScopeGenerate traffic signal timing plans by analyzing traffic flow status and patterns based on fusing internet data, induction coils data and video data, and control the traffic signal with the generated timing plans in a real-time, self-adaptive and cooperative way
Objective(s)To find an effective and efficient solution to improve the road utilization efficiency by increasing traffic flow speed and reducing traffic flow waiting time.
Short
Description
(up to
150 words)
An AI solution was developed that could recognize real-time traffic flow status and abstract traffic flow patterns by fusing internet data, induction coils data and video data, and could generate optimized traffic signal timing plan by self-adaptively responding to real-time traffic flow fluctuation and with regards to traffic flow coordination among multiple intersections within a given region.
Complete Description By far, traffic administrator produces traffic signal timing plans by observing traffic flow situation on-site at intersections or through videos, and relies on her/his personal experience. Then, the timing plans are input into and executed by the traffic signal control system. The disadvantages of this manual traffic signal timing plan generation approach are as follows: 1. Low computing efficiency, it consumes very long time for traffic administrator to observe and analyze traffic patterns. 2. Low computing precision, traffic administrator only cares about the macro traffic flow tendency at intersections without computing detailed traffic parameters such as speed, queue length in each lane, etc. 3. Slow response to traffic flow fluctuation, it is hard for traffic administrator to produce adaptive timing plan in time with respect to real-time traffic flow fluctuation, due to her/his limited computing ability, not mention to coordinate traffic flows among multiple intersections by controlling the traffic signal in real-time. 4. Experienced traffic administrators are severely in short for cities with the scale of thousands intersections.

For solving the above problems, the AI provider applies a multi-source data fusion approach to recognize the traffic flow status and generalize the traffic flow pattern by analyzing the internet data (i.e., vehicle driving trajectory data provided by internet service supplier), detector data collected by induction coils, and structured data recognized from videos. Furthermore, the AI provider develops an optimization method to figure out optimized traffic signal timing plan by self-adaptively responding to real-time traffic flow fluctuation and with regards to traffic flow coordination among multiple intersections.

The developed methods have been applied in practice within a given region from a large city. It generates traffic signal timing plans for all the intersections in the region according to their real-time traffic flow fluctuation with an updating frequency of 5 minutes per time. Compared with the manual traffic signal timing plans form the traffic administrators, the plans generated by the new method have increased the average vehicle driving speed by 9%, and reduced the average vehicle waiting time by 15%.

StakeholdersDOT, DOP
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
new privacy threats, new security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Optimization
Method(s)Deep learning, Bayesian network, Time series analysis, Operational research optimization method (i.e., Mixed integer linear programming, etc.)
HardwareECS
TopologyCloud Service
Terms &
Concepts Used
Traffic signal self-adaptive and coordinative control for a large number of intersections. Issues: 1. Not all intersections are equipped with detectors such as induction coil or video. 2. The detectors may output abnormal values which need data clean processings
Standardization
Opportunities
Requirements
Challenges
& Issues
Challenges:Traffic signal self-adaptive and coordinated control for a large number of intersections. Issues: 1. Not all intersections are equipped with detectors such as induction coil or video. 2. The detectors may output abnormal values which need data clean processing.
Societal Concerns Description Relieve urban road congestion
SDGs to
be achieved
Sustainable cities and communities
Data Characteristics
Description Internet data, Induction coil data, Video data
Source Internet, Detector
Type Structured text and number, Structured text and number, Unstructured video
Volume (size)
Velocity Internet data updated daily, Induction coil data updated every 5 minutes, Video data updated in real-time
Variety From multiple domains
Variability
(rate of change)
Dynamic
Quality Exists missing values or abnormal values

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 50 ID: Use Case Name: AI solution to quality control of Electronic Medical Record (EMR) in real time
Application
Domain
Healthcare
Deployment
Model
Cloud services
StatusIn operation
ScopeDetecting defects in EMR by inspecting unstructured data based on Natural Language Processing(NLP) ability
Objective(s)To insure the completeness, consistency, punctuality and medical-compliance of EMR written by physicians
Short
Description
(up to
150 words)
This AI solution in ET Medical Brain Medical service support system was developed that could simultaneously detect mistakes while physicians wrote EMR (Electronic Medical Record).

Using NLP (Natural Language Processing) ability, it can process a large amount of unstructured text and judge the accuracy according to recognized medical reference.

It achieved 80% coverage of all the EMR quality control requirements issued by Chinese government, and human labour of EMR QC (Quality Control) was reduced 60%, which translated into cost savings, and enhanced physician education.

Complete Description This AI solution in ET Medical Brain Medical service support system was developed that could simultaneously detect mistakes while physicians wrote EMR (Electronic Medical Record).

Using NLP (Natural Language Processing) ability, it can process a large amount of unstructured text and judge the accuracy according to recognized medical reference.

It achieved 80% coverage of all the EMR quality control requirements issued by Chinese government, and human labour of EMR QC (Quality Control) was reduced 60%, which translated into cost savings, and enhanced physician education.

Medical records are the records of the occurrence, development and prognosis of patients' diseases, as well as the medical activities such as examination, diagnosis and treatment.

A high-quality medical record has great value at medical and legal level.

When medical records are converted from handwritten to electronic input, delayed, uncompleted writing and copying are endangering the quality of medical records.

Once the medical record data does not meet the requirements, it will greatly affect the health of patients, the development of medicine and the judgment of responsibility in medical accidents.

Nowadays, hospital has a Medical Records Department to control medical records quality manually. However, as the number of medical records increases, the inspection requirements become more complex, and the medical professional knowledge requirements are improved, so the medical records quality inspection becomes harder.

The intelligent electronic medical record quality control system is based on NLP. When a doctor writes medical records, it can analyze unstructured medical record text, and control the quality based on government requirements, ensure the integrity, consistency, timeliness and compliance of medical records.

ET (Evolutionary Technology) Medical Brain Medical service support system has learning ability to learn more medical knowledge including clinical pathway, drug compatibility taboo etc. it can learn the habits and rules of doctor’s manual review to inspects records profoundly.

The current system has covered 189 medical records quality inspection requirements, saved 60% review time for medical record department, which greatly saved the cost of the hospital, reduced the inspection time and repeated work, and will help doctors put more energy into the education and training.

StakeholdersDoctor, Hospital, Patient
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
New privacy threats, new security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Natural language processing
Method(s)SimHash
HardwareECS
TopologyCloud Service
Terms &
Concepts Used
Jaccard index
Standardization
Opportunities
Requirements
Challenges
& Issues
Challenges: Achieve all EMR QC requirements in different disease areas

Issues: 1) Lack of medical reference data 2) Lack of medical knowledge graph

Societal Concerns Description Achieved 80% coverage of all the EMR quality control requirements issued by Chinese government, and human labour of EMR QC (Quality Control) was reduced 60%, which translated into cost savings, and enhanced physician education.
SDGs to
be achieved
Good health and well-being for people
Data Characteristics
Description EMR text data
Source EMR system
Type Text data from EMR system vendor
Volume (size)
Velocity Real time
Variety Multiple datasets
Variability
(rate of change)
Static
Quality High (depending on EMR system)

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 46 ID: Use Case Name: Adaptable Factory
Application
Domain
Manufacturing
Deployment
Model
Cyber-physical System, Embedded System
StatusPoC
Scope(Semi-)Automatic change of a production system’s capacities and capabilities from a behavioral and physical point of view
Objective(s)The objective is to enable flexible production resources which enable fast reconfiguration and adaptation to changing situations, context, and requirements which facilitate optimized resource usage under uncertainty.
Short
Description
(up to
150 words)
Rapid, and in some cases completely automated, conversion of a manufacturing facility, by changing both production capacities and production capabilities. This use case describes the adaptability of an individual factory by (physical) conversion and/or adaption of a factory’s and its machines behavior in order to adjust to changing situations like disruptions, material quality variation, production of new products, etc.
A prerequisite is a modular and thereby adaptable design for manufacturing within the factory. The result is a need for intelligent and interoperable modules that basically adapted to an altered configuration on their own, and standardized interfaces between these modules.
Complete Description Use Case description taken from [1,2,3]. Plug & Play – using a home computer and a USB cable, it is easy to connect new devices and use them almost immediately without any additional effort. The flexibility that has been available for quite a while on desktop computers is now gaining importance for industrial production. Demands on adaptability of production infrastructure are already rapidly increasing. Shorter and shorter product and innovation cycles require investment decisions for new production facilities that reflect future demand for production and process changes, where possible. In addition, the growing volatility of orders is hindering the optimal utilization of manufacturing lines with increasing frequency. Flexibility and adaptability will become increasingly important criteria in decisions regarding construction and operation of new production facilities.

One example is product labeling. Various printing technologies are available, for example tampon printers (transferring ink from the printing form to the product using an elastic tampon), inkjet printers and/or laser printers. In an adaptable factory this type of operating equipment can be connected directly to the automated production process. Simply put, the material to be printed says: “Print me”, and the tampon printer will ask: “Is the material to be printed greaseless?” The ink jet printer will then ask about the material characteristics, because it uses heat for the drying process, for example. A laser printer will ask about the material receiving the label to ensure sufficient contrast.

Use Case description taken from [1,2,3]. Plug & Play – using a home computer and a USB cable, it is easy to connect new devices and use them almost immediately without any additional effort. The flexibility that has been available for quite a while on desktop computers is now gaining importance for industrial production. Demands on adaptability of production infrastructure are already rapidly increasing. Shorter and shorter product and innovation cycles require investment decisions for new production facilities that reflect future demand for production and process changes, where possible. In addition, the growing volatility of orders is hindering the optimal utilization of manufacturing lines with increasing frequency. Flexibility and adaptability will become increasingly important criteria in decisions regarding construction and operation of new production facilities.

One example is product labeling. Various printing technologies are available, for example tampon printers (transferring ink from the printing form to the product using an elastic tampon), inkjet printers and/or laser printers. In an adaptable factory this type of operating equipment can be connected directly to the automated production process. Simply put, the material to be printed says: “Print me”, and the tampon printer will ask: “Is the material to be printed greaseless?” The ink jet printer will then ask about the material characteristics, because it uses heat for the drying process, for example. A laser printer will ask about the material receiving the label to ensure sufficient contrast.

Key aspects
The application scenario for adaptable factories describes the rapid, and in some cases completely automated con-version of a manufacturing facility, by changing both production capacities and production capabilities. The key concept for implementation is a modular and thereby adaptable design for manufacturing within the factory. Intelligent and interoperable modules that basically adapted to an altered configuration on their own, and standardized interfaces between these modules allow for quick and simple conversion to adapt to changes in the market and customer demands. Whereas the application scenario Order-Controlled Production emphasizes flexible use of existing manufacturing facilities by means of intelligent connectivity, this scenario describes the adaptability of an individual factory by (physical) conversion.

Today, when creating a production line, the focus is usually not only on quality, but also maximization of productivity and profitability of a pre-conceived product range. Individual components are connected statically and are capable of producing the pre-conceived functionalities and projected volumes. Frequently, a system integrator takes care of coordinating the individual components and developing a control system for the entire facility. However, if the order level is driven by strong product individuality or high fluctuation in demand, companies can no longer rely on the advantage of particular production lines. In this case, modular, order-oriented and adaptable manufacturing configurations become more attractive: For example, they increase overall utilisation or ability to deliver products. At the same time, however, the demands on individual machines or manufacturing modules increase. Even more important than high variance of specific manufacturing steps will be the ability to combine individual modules with ease and in any situation. In order to achieve this, the modules must contain a self-description regarding their ability to be combined or converted into a machine or plant very rapidly and robustly. The following examples illustrate these requirements:

  • A new network-enabled field device, for example a drive with a new version of firmware, is hooked up to the production line. The new device must be provided automatically with network connectivity and be made known to all online subsystems. The participating systems must correspondingly be updated.
  • An unconfigured field device is introduced to production, for example to quickly replace another defective device. The field device now must be individualized and parameterized due to the information located in the software components.
  • A production facility is converted or modified because a new product variation is planned. The control and software related changes must be detected and automatically transmitted to all participating systems.
  • After conversion of a plant, it should be possible to move software components for process management around the decentralized control units, while observing certain criteria, such as output or availability.
  • A (new) function of the Manufacturing Execution System (MES) is inserted or altered, for example the visualization of a situation not previously required. The visualization should be done automatically and access to the necessary information from the field level should also be automatic.

    This requires the mechanical engineer to design the internal development processes accordingly. Modular machines require “modular” engineering, based on libraries of re-usable modules (“platform development”). Machine architecture must be designed such that combinable mechatronic modules are created, including the Plug & Produce capability of production modules using interoperable interfaces and adaptive automation technology. This requires development of concepts for “services” across manufacturer boundaries, such as archiving, alerting or visualising, as well as a low-cost integration of MES functions.

    Effect on value chains
    Value added is shifted from the system integrator to the machine provider or its supplier, because the machines or components are enhanced so that they are easier to integrate. The type and quality of system integration change. The present focus on (production) technology shifts to a stronger focus on organization and business processes related to production processes. In extreme cases, the system integrator could become obsolete if intelligent, self-configuring and interoperable manufacturing modules can be created at the level of the machine suppliers.

    Value added for participants
    For manufacturing companies, a quick, inexpensive and reliable conversion of manufacturing becomes possible, so that they can react quickly to changes in customer and market demands. Increasing standardization and modularization also expand the possibilities for combining manufacturing entities of various providers and therefore realizing the most economic solution for each individual module.

    Machine modularization opens up new areas with scale effects for machinery manufacturers.

StakeholdersComponent suppliers (sensors, actuators), Machine builders, system integrators, plant operators (manufacturer)
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Automatic reasoning (e.g. [7,8]), AI (task) planning (e.g. [4,6]), distributed coordination and negotiation (e.g. [5])
Method(s)
Hardware
Topology
Terms &
Concepts Used
Standardization
Opportunities
Requirements
Standardization needs for setting up this use case is currently under further investigation. Some initial intentions on standardization needs are the following: a vocabulary with formal semantic for symbolic reasoning about production capabilities across different vendors, standardized negotiation mechanisms, standardized autonomy classes of components, machines, etc. Quality model for trustful learned models and automatic behavior resulting from it.
Challenges
& Issues
Societal Concerns Description Enabling flexible and autonomously reconfigurable production systems ease human-machine configuration, facilitate optimized machine use, reduce failures through autonomous compensation, optimized product quality through prediction techniques.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 47 ID: Use Case Name: Order-Controlled Production
Application
Domain
Manufacturing
Deployment
Model
Cloud Services
StatusPrototype
ScopeAutomatic distribution of production jobs across dynamic supplier networks
Objective(s)The objective is to enable automatic supplier contracting for optimized utilization of manufacturing capabilities at suppliers, novel degrees of flexibility in contract manufacturing, and enable (mass) customized customer ordering
Short
Description
(up to
150 words)
A network of production capabilities and capacities that extend beyond factory and company boundaries allows for a quick order-controlled adaption to changing market and order conditions. The result is a largely fragmented and dynamic value chain network that change as required by the individual order, and thereby make the best use of capabilities and capacities of existing production facilities. The goal is to allow for automated order planning, allocation and execution, thereby considering all production steps and facilities required to facilitate linking external factories into a company’s production process, as automated as possible.
Complete Description Use Case description taken from [1,2,3]. Many contemporary products are changing at an ever-in-creasing rate. Whereas up until just recently, smartphone displays were flat, the first curved displays are already on the market. The array of materials used in the automotive sector is also continually expanding – from aluminum, to high-strength steels and even fiber-reinforced plastics, today many types of materials are used.

Innovation and product cycles are getting shorter all the time, and new production technologies are putting pressure on manufacturing companies to react more and more rapidly and make quick investment decisions regarding both consumer goods and investment goods. In order to confront this trend and avoid lengthy investment decisions, companies are starting to increase the network of their production capabilities beyond their own company boundaries.

Key aspects
The Order-Controlled Production application scenario describes a flexible manufacturing configuration. Owing a network of production capabilities and capacities that extend beyond factory and company boundaries, this company can quickly adapt to a changing market and order conditions, and thereby make the best use of capabilities and capacities of existing production facilities. In this way the potential provided by a network to other factories out-side of the company’s own facilities is used to align the company’s own portfolio - and especially its production - to quickly changing customer and market demands. Specifically, manufacturing chains are optimized for various parameters, such as cost and time.

At its core, order-controlled production is based on standardization of the individual process steps on the one hand and the self-description of production facility capabilities on the other hand. This standardization allows for auto-mated order planning, allocation and execution, thereby considering all production steps and facilities required. This helps to combine individual process modules much more flexibly and earlier than previously possible, and to make use of their specific capabilities.

In this respect, companies offer their available production capacities to other companies and thereby increase the utilization of their own machinery. Other companies may access these capacities as needed, thereby temporarily expanding their own production spectrum. In so doing, available production capacities are utilized better and order fluctuations can be smoothed out. The goal is to facilitate linking external factories into a company’s production process, as automated as possible. In particular, the order placement process required for this should be executed automatically.

Effect on value chains
Today’s relatively rigid and separately negotiated relation-ships between companies along the value chain will be transformed into a largely fragmented and dynamic value chain network that changes as required by the individual order. This applies both horizontally over the entire manufacturing process as well as vertically, with regard to production depth. Manufacturing companies focus on value-added steps that distinguish them significantly from other competitors. The possibility of creating fast and global client-manufacturer relationships can lead to unexpected competitive situations, because companies may change their role from order to order. Dynamically integrating production capacities will lead to better machine utilization and, as a result, diminishing demand for machinery suppliers.

Value added for participants
On the one hand, manufacturing companies will be able to automatically expand their production capabilities and capacities ad hoc in line with demand, by utilizing external production modules. No investment is required. This enables companies to react very flexibly to changing market and customer demands. On the other hand, companies offering their machines on the market can optimize their utilization rates.

StakeholdersCustomer, Producing companies, Broker
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Automatic reasoning, AI (task) planning, distributed coordination and negotiation (cf. [5-8] for details and overview)
Method(s)
Hardware
Topology
Terms &
Concepts Used
Standardization
Opportunities
Requirements
Standardization needs for setting up this use case is currently under further investigation. Some initial intentions on standardization needs are the following: Standardization of data formats and semantic for exchanged data is enabler for this use case where multiple companies and institutions are involved (formal semantics for reasoning about 3d models, task decomposition and planning), standardization of interaction protocols between participants (esp. coordination and negotiation) enables automatic cross-company contracting.
Challenges
& Issues
Societal Concerns Description Enabling mass-customized production in global dynamic supply chains, and by that, ease production of small lot sizes for customized products.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 43 ID: Use Case Name: Deep Learning Based User Intent Recognition
Application
Domain
Retail
Deployment
Model
On-premise systems
StatusIn operation
ScopeRecognizing users’ intent to solve their problems in e-commerce fields
Objective(s)To recognize and understand users’ intent by AI and deep learning technologies and apply such technologies to build chat bot systems to further reduce labor cost and to be applied in various fields.
Short
Description
(up to
150 words)
Intelligent customer service chat bot is mainly used to categorize users’ questions, recognize users’ intents and answer users’ questions intelligently for different business jobs. Currently, this chat bot has been used to handle 90% of online customer service and has enabled JD.com to save over 100 million labor costs every year.
Complete Description JD.com has been committed to using technology to drive business growth and improve user experience in all customer service fields. Based on the improvement of customer consulting experience and the developing trend of artificial intelligence technology, as early as 2012, JD had decided to develop intelligent chat bots to fulfill the needs of continuous expansion of business, to save customer service costs and increase service capability. Intent recognition is a key and core technology to build such an intelligent customer service chat bot. By applying natural language processing technologies, deep learning technologies, traditional machine learning algorithms, intent recognition accuracy has reached to 95%. Based on accurate intents, and a series of solution finding algorithms, our chat bot can solve the user’s problems to a great extent and give the user a high quality consulting experience. Finally, in order to provide diversified and personalized customer services, we are continuously improving the accuracy of intent recognition, personalized solution generation, sentiment recognition, and image recognition. So far, intelligent customer service has revolutionized the traditional customer service consulting business.
Stakeholdersusers
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
high semantic ambiguity, Multiple language expressions in one sentence
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Natural language processing
Method(s)Machine learning and deep learning
HardwareGPU and CPU
TopologyTensorFlow
Terms &
Concepts Used
Natural language processing, deep learning, CNN, HAN, logistic regression
Standardization
Opportunities
Requirements
Process Standardization will Improve Quality and Productivity
Challenges
& Issues
Current challenges of deep leaning and intent recognition:
  • high semantic ambiguity, similar sentences can deliver different meanings.
  • Unclear classification rules caused by complicated business logics
  • Hard to answer reasoning questions
Societal Concerns Description
  • Solve problems intelligently to increase efficiency
  • Free labors from repetitive work to save large amount of resources for the society
SDGs to
be achieved
Decent work and economic growth
Data Characteristics
Description Question answering data from the JD.com online dialogue log
Source Customer's dialogue log at JD.com
Type Text
Volume (size) Millions
Velocity Real time
Variety various scenarios, various business, various categories of products
Variability
(rate of change)
Non-linear
Quality good

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 44 ID: Use Case Name:  Chromosome Segmentation and Deep Classification 
Application
Domain
Healthcare
Deployment
Model
Hybrid or other 
StatusPoC 
Scope
Objective(s)
  • Automating Karyotyping of the chromosomes in cell spread images.
  • Segmentation of chromosomes in the images using non expert crowd
Short
Description
(up to
150 words)
Karyotyping of the chromosomes micro-photographed under metaphase is done by characterizing the individual chromosomes in cell spread images. Currently, considerable effort and time is spent to manually segment out chromosomes from cell images, and classifying the segmented chromosomes. We proposed a method to segment out and classify chromosomes for healthy patients using a combination of crowdsourcing, preprocessing and deep learning, wherein the non-expert crowd from external crowdsourcing platform is utilized to segment out the chromosomes, which are then classified using deep neural network. Results are encouraging and promise to significantly reduce the cognitive burden of segmenting and karyotyping chromosomes.
Complete Description Metaphase chromosome analysis is one of the primary techniques utilized in cytogenetics. Observations of chromosomal segments or translocations during metaphase can indicate structural changes in the cell genome, and is often used for diagnostic purposes. Karyotyping of the chromosomes micro-photographed under metaphase is done by characterizing the individual chromosomes in cell spread images. Currently, considerable effort and time is spent to manually segment out chromosomes from cell images, and classifying the segmented chromosomes into one of the 24 types, or for diseased cells to one of the known translocated types. Segmenting out the chromosomes in such images can be especially laborious and is often done manually, if there are overlapping chromosomes in the image which are not easily separable by image processing techniques. Many techniques have been proposed to automate the segmentation and classification of chromosomes from spread images with reasonable accuracy, but given the criticality of the domain, a human in the loop is often still required. In this paper, we present a method to segment out and classify chromosomes for healthy patients using a combination of crowdsourcing, preprocessing and deep learning, wherein the non-expert crowd from CrowdFlower is utilized to segment out the chromosomes from the cell image, which are then straightened and fed into a (hierarchical) deep neural network for classification. Experiments are performed on 400 real healthy patient images obtained from a hospital. Results are encouraging and promise to significantly reduce the cognitive burden of segmenting and karyotyping chromosomes.
Stakeholders
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Recognition 
Method(s)
Hardware
Topology
Terms &
Concepts Used
Deep learning, crowd sourcing, non-expert crowd, segmentation, karyotyping
Standardization
Opportunities
Requirements
Challenges
& Issues
  • Crowd’s job satisfaction
  • Spamming in annotated data
Societal Concerns Description
SDGs to
be achieved
Good health and well-being for people
Data Characteristics
Description The dataset comprised of 400 stained images with varying degrees of overlap between chromosomes, out of which 200 were kept for testing and the remaining for training and validation
Source Partner hospital
Type Images
Volume (size) 400
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 38 ID: Use Case Name: Machine learning driven approach to identify the weak spots in the manufacturing of the circuit breakers.
Application
Domain
Manufacturing
Deployment
Model
Prototype
StatusOn-premise system
ScopeDetecting the issues in manufacturing process that leads to early failures of the circuit breakers through the data mining of the manufacturing process.
Objective(s)To generate actionable intelligence to improve the manufacturing process of circuit breakers through mining of manufacturing related data.
Short
Description
(up to
150 words)
An approach was developed that can mine the manufacturing data of circuit breakers through multiple machine learning algorithms. The approach could successfully identify the weak spots in the manufacturing where failure rate jumped from 0.2% to 7% (35 fold more probability of failure) and hence candidates for improvement in the manufacturing process.
Complete Description High voltage circuit breakers are critical component of an electric circuit and it has a normal lifespan of 30-40 years. However, due to various reasons few circuit breakers fail within 0-5 years of operation. As a manufacturer of these circuit breakers, lots of data related to manufacturing aspects are present with the manufacturer. Such data has information about production lot size, material of production, design voltages for sub-components, heater voltages, date of failure etc. In general data related to 49 variables are captured for close to 56000 circuit breakers over a lifespan of several years. The manufacturer is interested to know if there are any weak spots in the manufacturing process which leads to higher failure rates.
Circuit breakers can fail not only due to manufacturing defects but also due to wrong operation of the circuit breaker in the field e.g. applying voltages higher than design values. However, operational data of the circuit breakers was not available with the manufacturer.
Therefore, the key challenge of this project was knowledge discovery with partial data set using machine learning algorithms.
The data scientists applied various machine learning algorithms such as decision tree, random forest, support vector machine, Naïve Bayes classifier, logistic regression and neural network and compared the results of one algorithm verses the other algorithm. Through multiple numerical experimentations on data selection and algorithm hyper parameter tuning, the data scientist team selected the best algorithms and deduced the key weak spots in the manufacturing that are generally associated with high failure rates. In conclusion, the work provided a set of 5 actionable rules, where the failure rates jumped drastically from 0.2% to 7% leading to 35-fold higher chance of failure.
StakeholdersBatch manufacturer such as milk pasteurization, pharmaceutical, paint manufacturing, etc.
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Incorrect use of AI/ML
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Classification
Method(s)Decision trees, SVM, ANN, Logistic Regression, Random Forest and Naïve Bayes
Hardware64 GB RAM Windows server
TopologyNA
Terms &
Concepts Used
Classification, Actionable Rules, HV Circuit breakers
Standardization
Opportunities
Requirements
Standardization of data representation models comprising of both manufacturing related data and end-use related data.
Challenges
& Issues
Discovering actionable insight with partial data set and managing bias in ML models due to limited number of failed cases
Societal Concerns Description Safe and reliable power delivery
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 39 ID: Use Case Name: Machine Learning Driven Analysis of Batch Process Operation Data to Identify Causes for Poor Batch Performance
Application
Domain
Batch Manufacturing
Deployment
Model
On-premise systems
StatusPrototype
ScopeDetecting the issues in batch manufacturing process that leads to bad quality products or longer cycle times of batch processing
Objective(s)Provide insight to the operation team to improve the productivity of batch manufacturing through machine learning on historical operation data
Short
Description
(up to
150 words)
An approach was developed that can use machine learning models to identify issues in batch manufacturing.
Complete Description Batch operation is generally quite complex involving dynamics in the operation and interplay of various process variables. Due to this, sometimes, few batches end up running slower than nominal batch time and few batches also yield bad quality end products resulting in significant production loss. Additionally, often in the industrial context, data size and variety are limited and to develop a robust machine learning model from limited available data sets is a challenging task.
Due to transient nature of batch operation data, the traditional PCA algorithm fails in analyzing the batch data and hence MPCA was applied as logical extension of PCA algorithm. As MPCA naturally considers the dynamics in the data and inter-correlations among the process variables, it provides a valuable insight on the batch data.
The approach was successfully demonstrated on milk pasteurization process data where only 4 batches were provided for modelling. Using such 4 seed batches, the algorithm synthetically creates 50 batches of data and introduction of anomalies in some batches. Concept of design of experiments and stochastic perturbations are used in synthetic generation of the data set. The work was able to successfully build a robust MPCA model with such data and isolate the bad batches of data from good batches of the data. Additionally, through contribution plots, the algorithm identifies when a certain batch drifted from nominal operation and which variables are the root causes for the bad batch operation.
Stakeholders
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Incorrect use of AI/ML; New Security Threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Classification
Method(s)Multiway Principal Component Analysis
Hardware64 GB RAM Windows server
TopologyNA
Terms &
Concepts Used
Classification, MPCA, Anomalies
Standardization
Opportunities
Requirements
  • Standard data representation models for AI relevant batch data handling
  • Standard GUI for AI relevant result presentation
Challenges
& Issues
Discovering actionable insight with limited industrial data set, handling dynamics in the process variables
Societal Concerns Description Consistent batch operation lead to enhanced productivity
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 40 ID: Use Case Name: Empowering Autonomous Flow meter control- Reducing time taken to -proving of meters-
Application
Domain
Manufacturing
Deployment
Model
Cloud services
StatusIn operation
ScopeCalibration of control devices
Objective(s)Reduce the time taken for trial & error methods to set the VFD and FCV setpoints
Short
Description
(up to
150 words)
The customer had to set VFD and FCV % manually to achieve desired flowrate using trial & error methods, which could take about 3-4 hours. Efficiency for the proving of the meters was very less & improvement was needed to remove any aberration in reading as it was time consuming.
Complete Description Cerebra was integrated with the system considering the flow of the fluid. The customer can choose between the available options of high flow rate, low flow rate or multi viscous flow. Then, with the master meter in the loop of testing, the meter from the field was introduced to analyse how much of aberration is there and then proving it more efficiently. Since it took more time for them to get the exact values of VFD & FCV % to achieve the desired flow rate, Cerebra’s Prognostics Engine was introduced. Purely based upon machine learning algorithms, the data models for the VFD & FCV % was used to predict the values to be chosen with an accuracy of about 98%. Since there was a presence of a closed-loop system, this predicted value was automatically registered on the valves’ monitors which only required small tweaking in the end, thus reduced human efforts.
StakeholdersProcess Industries; Humans
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Challenges to accountability, security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Prediction
Method(s)
HardwareApplication Server: 64 GB RAM/ 16 Core / 500 GB HDD; Data Server: 128 GB RAM/ 16 Core, 3 TB HDD
Topology
Terms &
Concepts Used
ISO 13379, 13381, 13374, 14224, 17359 , ISA-95
Standardization
Opportunities
Requirements
  • Mandate of the key sensors based on the type of equipment
    Based on the type of equipment, the makers need to have the basic set on sensors imbibed onto the system. E.g. for a pump – it is important to measure the input flow and output flow rates, vibrations, rotation speed, lube oil temperature and pressure. This will guide the equipment manufactures to provide their customers and their data products to capture the minimum required data and understand the equipment performance
  • Mandate for the organizations to expose the minimum and key parameters
    The equipment owners need to enable the basic set of sensors for the equipment health and performance which are required for monitoring the asset from any failures
  • Standards for Data Formats
    Each organization has a different way of capturing data and storing them in different formats. Due to which the solutions are not scalable across organizations though the product behind them is same. It takes customised efforts each time.
  • Guidelines for deciding the sampling frequency based on the type of data
    We see a need to have a specific set of guidelines to capture data at a minimum required sampling frequency. For example, a vibration sensor should capture data at least at 1 ms or less.
  • Guidelines for Feature Engineering
    There must be guidelines as to how the features need to be engineered for AI models. Lack of this would lead to more black box models not explaining how the models behave the way they do.
  • Guidelines for Standardization of event types and codes
    There are multiple events which occur for an asset or in a manufacturing plant. Guidelines would help people capture the data in a similar fashion helping the industry to benchmark against one another and at industry level we can understand, which events are the most critical.
    Guidelines for standardization of Fault and Error Codes for an equipment or process
    Similar to events, it is also useful to capture fault, failure and error codes in a standard way.
  • Process Guidelines for event related data (Maintenance and Work Orders)
    Guidelines would help people capture the data in a similar fashion helping the industry to benchmark against one another and at industry level we can understand, which events are the most critical
  • Guidelines for Training AI models
    A defined set of guidelines for AI models would be useful for the data scientists to follow. It will also aid the consumers of AI models to understand how the outcome has been deduced
  • Guidelines around AI model explainability
    With so many black box models floating around in the industry, it is difficult for consumers of AI models to understand then and their output. And with engineers and domain experts, coming into the picture, it is very much required to make these models more explainable.
  • Process Guidelines and methods for model evaluation (retraining)
    Before deployment and post deployment, it is very critical to have standard methods for models. And also post deployment, we must set guidelines for retaining the model on a periodic basis or based on data volatility. This is increasingly becoming important as AI models are being involved in more strategic and operational decision making.
  • Guidelines for disaster recovery and autonomous operations
    With the aid of AI models, the operations of an equipment or manufacturing plant are becoming more and more autonomous and self- sufficient. But the human monitoring is also important as any kind of inaccurate prediction can lead to a disaster and it is must to have some standard to recover from this situation and to assess the conditions to go for autonomous operations.
Challenges
& Issues
Societal Concerns Description Promoting sustainable industries, and investing in scientific research and innovation, are all important ways to facilitate sustainable development.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 41 ID: Use Case Name: Improving Productivity for Warehouse Operation
Application
Domain
Logistics
Deployment
Model
On-premise systems
StatusPoC
ScopeBig data analysis for enhancing productivity
Objective(s)To improve productivity of warehouse operation by detecting and changing controllable factors
Short
Description
(up to
150 words)
AI-driven operating system that uses big data from work performance information to issue appropriate work instructions has been developed. In PoC, picking operation improvement was conducted in a distribution warehouse. As the result, 8% work reduction was performed.
Complete Description Attempts are being made to increase the efficiency of work improvements through more widespread application of IT to work systems. However, as each new improvement is added or improvements are made with respect to environmental changes, it requires manual changes to the system, leading to increases in work improvement costs. This case has developed an AI system that uses big data such as work performance information, to understand worksite improvements and environmental changes and issue appropriate work instructions. It has conducted a demonstration test, which confirmed the effectiveness of this system for improving distribution warehouse work. In the future, we will continue to work on expanding the AI system to a wide range areas such as manufacturing and distribution.
Stakeholderswarehouse manager
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
possibility of back action
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Optimization
Method(s)modeling of relationship between explaining variables and outcome, and optimization
HardwarePC, wearable sensor
Topology
Terms &
Concepts Used
Human big data analysis, regression analysis
Standardization
Opportunities
Requirements
standardization of data format, sensors to be used, and API of IT and mechanical systems
Challenges
& Issues
understanding of workers' human factors (privacy, additional work etc.)
Societal Concerns Description solving labor shortage problem and improving labor related issues with aiming improving productivity.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 42 ID: Use Case Name: Emotion-sensitive AI Customer Service
Application
Domain
Retail
Deployment
Model
On-premise systems
StatusIn operation
ScopeExtracting sentiment and its intensity from customers’ input, and responding with appropriate attitude in order to improve the quality of customers’ inquiry.
Objective(s)To design an efficient solution for customers’ sentiment and intensity detection, especially in the situation of limited training dataset.
Short
Description
(up to
150 words)
The emotion-sensitive AI customer service of JD.com Int., is supported by AI technology and deep learning method. It is developed for ameliorating accuracy of customer sentiment and intensity. In sentiment classification, it has achieved 74% accuracy and 90% recall score while in intensity detection, it has accomplished 85% accuracy and 85% recall. During the special sale of -618-, it has increased customer satisfaction by 57%.
Complete Description JD’s customer service representatives need to handle millions of requests on a daily basis. Regular AI customer service systems, 24/7 online, are capable of offering instant assistance, which alleviates the labor resources to a large extent. However, it is quite challenging, if not impossible, for those systems to interpret emotions from customer input and respond as friendly as human.
Under this background, based on huge data set of customer comments and rich experience of Natural Language Processing, our system can automatically detect sentiments like happy, angry, anxious, etc. Moreover, this system can also detect the intensity of customer sentiment. Furthermore, we adapt Convolutional Neural Networks, a widely used techniques in visual computing, to interpret the semantic meaning of customer’s expression. It can improve the system’s performance for sentiment classification and intensity detection. Moreover, with the adoption of transfer learning, the system can also be applied into various types of data. To overcome the difficulty of limited training data, we also use data augmentation method such as reverse translation and data noise to increase the variability of training data.
Up to now, the system has reached 90% recall and 74% accuracy rate for sentiment classification over 7 categories. The overall recall and accuracy for sentiment intensity are also around 85%?it has increased customer satisfaction by 57%.
StakeholdersCustomers targeted for the Customer Service system
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
The low degree of humanization, and lack of semantic diversity for response. Reducing the number of human customer service.
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Natural language processing
Method(s)Deep learning, transfer learning, data augmentation
Hardware
Topology
Terms &
Concepts Used
Deep learning: a class of machine learning algorithms use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation.
Transfer learning: we adopt multi-task learning method in this system. Jointly training different annotated data in same domain, this method improves the model performance for classification problems.
Data augmentation: we apply reverse translation to firstly translation Chinese into English and then translate it backward. We also use data noise to improve the data diversity.
Standardization
Opportunities
Requirements
The system can be promoted to as many customer cervices companies as possible once provide with enough training data for the specific Application scenario
Challenges
& Issues
Challenge: the system’s performance should be as good as the human customer server.
Issues: 1) limited training data; 2) sentiment classification among seven categories.
Societal Concerns Description Improving the corresponding efficiency of customer service, improving customer service experience: Reducing labor costs, and reducing operating costs.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description For sentiment classification: conversation data from after-sales customer services. It’s annotated by professional annotators into 7 categories of sentiments.
For sentiment intensity: Only including sentiment data with “anger” and “anxious”; it’s annotated into 3 degrees of intensity: “low, medium, high”.
Source Conversation data from JD.com real-time customer services
Type Text
Volume (size) Around 60,000 sentences for sentiment classification and 20,000 for sentiment intensity.
Velocity Batch Processing
Variety Real-time data from JD.com, including various categories of products.
Variability
(rate of change)
Static
Quality High

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 32 ID: Use Case Name: AI solution to help mobile phone to have better picture effect
Application
Domain
Mobility
Deployment
Model
Hybrid or other
StatusIn operation
ScopeBetter understanding the image and improving image effect on smartphone by using DL model which is trained in the cloud or offline.
Objective(s)To find an efficient solution to Increase camera image quality on smartphone without Increasing too much operation and power burden for mobile phone.
Short
Description
(up to
150 words)
An AI solution was developed that could increase smartphone camera image quality. Using deep learning, smartphone can Identify more scenarios and objects than before. Based on the identified scenarios and objects, smartphone can better understand the image and improve image effect.
Complete Description At present, there are 1.4 billion smart phone shipments in the world every year. Photography is one of the most important functions of smart phones. The industry has been trying to improve the picture quality of mobile phone photography. It hopes to reach even the quality of the professional SLR camera. The traditional image processing algorithm is currently facing the ceiling, many scenes traditional algorithms can ot be used, just because the effect is very poor.

Deep learning algorithm provides a turning point for solving the above problems. By using the AI solution, smartphones can better "understand" the pictures they take. Based on the deep learning algorithm, the smart phone can analyze the shooting scene in real time and intelligently identify various scenes in the shooting process, such as blue sky, flowers, green plants, night view, snow scene, etc. And the smart phone can also intelligently detect the shooting objects in the scene. Base on scene recognition and object detection,the smartphone can automatically adjust and set parameters for different pictures, so as to get better photo effects.
Now the mobile phone can recognize 100 kinds of scenes and can reach hundreds in the future. By using the depth learning algorithm, the mobile phone can now detect the 20 types of subjects, and the future can be detected by hundreds of subjects. Object detection can be used for SmartZoom (auto focus on targets), and portrait segmentation can be used for background blur or light efficiency.

Stakeholdersmobile phone manufacturer, end user, third party testing and evaluation agency
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
new privacy threats (hidden patterns).
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Recognition
Method(s)Deep learning
HardwareNPU, GPU, CPU etc.
TopologyNo Need
Terms &
Concepts Used
Deep learning, 'Understand'
Standardization
Opportunities
Requirements
The standardized content includes:
  1. 1the format of training picture data;
  2. the format of deep learning model generated offline or cloud, which will be transplanted to smart phones;
  3. the platform to support the transplanted model in the smart phone;
  4. API which can be used by others applications, such as: picture classification, security.
Challenges
& Issues
Challenges: Achieve the same level as professional SLR camera for pictures.
Issues:
  1. Lack of data for certain scene;
  2. Lack of computing ability on terminal side;
  3. Users can feel the improvement of image quality, but may not know that it is brought by AI.
Societal Concerns Description For the wrong object detection, it may lead to racial prejudice or privacy protection problems
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description Annotated pictures
Source Public picture library /Self collection picture library /Web crawling pictures /Automatic synthesis of pictures
Type Picture format supported by a training platform and smart phone
Volume (size)
Velocity
Variety Single source
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 33 ID: Use Case Name: Automated defect classification on product surfaces
Application
Domain
Manufacturing
Deployment
Model
On premise system
StatusPoC
ScopeImage Analytics for water taps in sanitary industries.
Objective(s)Image analytics using a combination of feature extraction and classification of defects on shining surfaces in sanitary industries.
Short
Description
(up to
150 words)
A vision system that inspects and identifies the defects on water taps in sanitary industries. The system uses a combination of features for an automatic defect classification on product surfaces. All defects (15 types are identified) are classified into two major categories, real-defects and pseudo-defects. The pseudo-defects cause no quality problem; while the real-defects are critical as they might malfunction the final products.
The AI system uses Support Vector Machine (SVM) classifier along with the combined features to identify the defect types. With the vision system in place, the quality control process is fully automated without any human intervention.
Complete Description The proposed vision system has two parts: the hardware part and the software part. The hardware captures the images of product surfaces under a constant illuminating condition. The software is developed to perform image processing tasks and identify defects on product surfaces.

The steps of proposed system include image acquisition, preprocessing, segmentation, feature extraction, classification and post-processing. The system presents two software components: Feature Extraction and Classifier Design. These two modules are implemented independently which can be developed in offline platform and can be integrated into vision system and work online.

As a first step, the feature extraction is critical and guides the extent to which a classifier can distinguish the defects from one class to another. A combination of features is used like geometry (shape, texture), and statistical features of the segmented images. In the second step, a support vector machine classification model is trained to identify the defect types. The classification results obtained by combining Gabor features, Statistical features, and grayscale features showed comparable performances with human evaluations.

Overall, the vision system is modularized with capabilities to self-learn and future extensions.

StakeholdersSanitary Industries
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Incorrect AI System use (AI system affecting quality control); New Security Threats.
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Recognition
Method(s)Classification; Feature Extraction
HardwareIP Camera and Work Station
Topology
Terms &
Concepts Used
Classification, Feature Extraction, Defect Identification
Standardization
Opportunities
Requirements
  1. Quality acceptance criterion from AI systems: What is the acceptable standard for AI output related to quality? How that can be independently validated?
  2. Standards for dealing with AI failures: How/Can standards facilitate dealing with AI failures, w.r.t., quality, productivity criteria?
Challenges
& Issues
Real time implementation, accurately identify the nature of defects
Societal Concerns Description Promoting sustainable industries, and investing in scientific research and innovation, are all important ways to facilitate sustainable development.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 34 ID: Use Case Name: Robotic task automation: Insertion
Application
Domain
Manufacturing
Deployment
Model
Embedded systems – Cloud service
StatusPoC
ScopeRobotic assembly
Objective(s)
  1. Simple programing/instruction and flexibility in usage
  2. Automation of tasks lacking analytic description
  3. Reliability and efficiency
Short
Description
(up to
150 words)
Assembly process often includes steps where two parts need to be matched and connected to each other through force exertion. In an ideal case, perfectly formed parts can be matched and be assembled together with predefined amount of force. Due to imperfection of production steps, surface imperfection and other factors such as flexibility of parts, this procedure can become complex and unpredictable. In such cases, human operator can be instructed with simple terms and demonstrations and perform the task easily, while a robotic system will need very detailed and extensive program instructions to be able to perform the task including required adaptation to the physical world. The need for such a complex program instruction will make use of automation cumbersome or uneconomical. Control algorithm that are based on machine learning, especially those including reinforcement learning can become alternative solutions increasing and extending the level of automation in manufacturing.
Complete Description The case described here is a common step in assembly processes in manufacturing industry and includes matching and properly connecting two parts when one needs to be inserted into another. Successful and efficient insertion usually needs action by feeling. It is difficult to describe in terms of mathematical algorithms and therefore is difficult to program. Complexities in programming, or high degree of operational failure make usage of robots, or automation unattractive. Use of machine learning and artificial intelligence is one of promising methods to overcome such difficulties.

As will be described below, there are several different phases in the process, where different methodologies can and should be used. To make the methodology usable in a practical case, it should be utilizable by operators without deep technical knowledge with an effort that can be accepted on a production line. Ultimately, such methods must remove the need for programing completely.

The assumption here is that the parts to be assembled are properly localized, such that they can be manipulated by a robot in the desired way. The problem concerns the following steps:

  1. Identification and picking the first part (A).
  2. Moving A to the vicinity of the second part (B).
  3. Alignment of the two parts.
  4. Exertion of force with simultaneous movement for smooth insertion.
  5. Termination of the task when complete insertion is complete.

The above task, with all possible challenges, can easily be performed by a human operator. An operator in majority of cases needs very limited amount of information. Using prior knowledge and experiences and the sensory system the task can be completed and all possible exceptions can be handled. With time, a human operator becomes constantly more efficient and performs the task faster and more reliably.

The topics to be handled in this use case are how a machine can be instructed, trained, perform and improve to a high level of reliability and efficiency. The process can be divided into following steps:

  1. Localization of parts: Image processing, object identification, classification and localization.
  2. Alignment of parts: Control and optimization with (mainly) vision inputs.
  3. Insertion through exertion of forces: Control and optimization with (at least) vision and force sensor feedback
  4. Sensing the termination of the process: Pattern recognition in time series.
  5. Continuous improvement: Reinforcement learning.

Vision and force sensors are most commonly used sensors in such processes. The objects and environment need to be observed at moderate as well as in very close distances. Force sensors are needed but have the weakness of not being active before a complete contact. Therefore, use of other sensors could be helpful.

The method is used for assembly tasks with the target of reducing the programming effort and increasing flexibility. For that to be achieved, the effort necessary to teach, train and use the system should be minimum and the reliability should come high at short time. This implicitly means that the system should become useful with limited amount of data and at limited amount of time. After an initial relatively stable state is reached, reinforcement can be used to improve the efficiency of the system.

The solution will become more attractive if transfer learning is utilized to further reduce the initial training time.

For benchmarking purpose a specific set of objects to be assembled together should be defined and performance of the methods can be measured by necessary training time, need for computing power and memory as well as time for completion of the task. The objects in the tests can be geometrically relatively simple. Special features such as rough surfaces, tight fitting or flexibility of the objects can be considered for different classes of problems.

StakeholdersDiscrete manufacturing industries; Operators
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Incorrect AI system use; New security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Recognition, classification, control, optimization
Method(s)Deep learning, image processing, control, Optimization
HardwarePC equipped with GPU accelerators
TopologyNA
Terms &
Concepts Used
Reinforcement learning
Standardization
Opportunities
Requirements
  • Standardization of definition of KPIs;
  • Standardization of fail-safe options w.r.t. safety and quality;
  • Standardization towards “Human-Co-working”
  • Minimum acceptable standards for commercialization;
  • Standard data set to independently validate the claims
Challenges
& Issues
  • Complex and unpredictable assembly process due to imperfection of production steps, surface imperfection and other factors such as flexibility of parts.
  • Accuracy of sensing
  • Coworking with humans
Societal Concerns Description Promoting sustainable industries, and investing in scientific research and innovation, are all important ways to facilitate sustainable development.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 35 ID: Use Case Name: Causality-based Thermal Prediction for Data Center
Application
Domain
Other
Deployment
Model
On-premise systems
StatusPrototype
ScopeData Center
Objective(s)Minimize energy usage in managing data center
Short
Description
(up to
150 words)
Data centers tend to be overcooled to prevent computing machines from failing due to heat. A reliable fine-grained control that could regulate air control unit (ACU) supply air temperature or flow is needed to avoid overcooling. Methods that are based on correlation-based techniques do not generalize well. Hence, we seek to uncover the causal relationship between ACUs supplying cool air and temperature at the cabinets to prioritize which ACUs should be regulated to control a hot-spot near a cabinet
Complete Description First, we perform experiments in 6SigmaRoom for the layout of the data center being studied. We collect time-series data for supply air temperature and flow per ACU, and for inlet temperature at the cabinets. Next, we test the recorded time series for checking if Granger-causality (G-causality) can be established between the supply air temperature from an ACU to a cabinet. G-causality establishes the unidirectional temporal precedence for data center control actions from ACUs that leads to changes in specific cabinet temperatures. A variable X is said to Granger-Cause Y if, including data about past terms from X, leads to a better prediction of the future value of Y (i.e., Yt+1) than predicting Yt+1 based solely on past terms from Y.

We show by way of simulation that the ACU flows that Granger-Cause reduction in temperature at a cabinet provide a larger share of influence (based on Zone of Influence/Thermal Correlation Index from the simulation) on the cabinet. This could allow an operator to come up with a better control strategy to control hotspots in a data center by regulating ACU supply air temperature/flows.

StakeholdersData center owner; Data center users; Environment
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Incorrect AI system use; Security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Prediction
Method(s)Regression
Hardware64 GB RAM Windows server
TopologyNA
Terms &
Concepts Used
Granger Causality
Standardization
Opportunities
Requirements
  • Standardization towards testin
Challenges
& Issues
Data sufficiency
Societal Concerns Description Promoting sustainable industries, and investing in scientific research and innovation, are all important ways to facilitate sustainable development.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 36 ID: Use Case Name: Powering Remote Drilling Command Centre
Application
Domain
Manufacturing
Deployment
Model
Cloud services
StatusIn operation
ScopeOil and Gas Upstream (Deployed in 150 Oil Rigs and 2.5 Billion+ Data Points each)
Objective(s)Automatic generation of Daily Performance Report, reduction in overall drilling time, cut down Invisible Loss Time and improve rig asset management
Short
Description
(up to
150 words)
It is important for a drilling contractor to have real time monitoring of rig parameters to optimize operations. The customer lacked granular insights during drilling, could not ascertain the root cause of non-productive time, and manual interpretation of signals led to missing of anomalies further degrading performance.
Complete Description Cerebra product extracted and ingested different types of signals from surface and downhole sensors to perform near real-time processing. More than 170 vital signals every second from each oil rig were processed by Cerebra to provide near real time insights into drilling operations. This was achieved by handling Data Format and Data Extraction standards and Cerebra’s Visualization Studio provides the flexibility of generating customized asset utilization reports, thus helping the oilfield engineers to understand the root causes of non-productive time and better utilize the assets on field. Rig specific utilization reports, and weekly and monthly utilization reports helped to plan drilling operations improving drilling efficiency.
StakeholdersOil and Gas Upstream sector; Environment, Humans
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Challenges to accountability, security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Knowledge processing & discovery
Method(s)Utilization and Performance Evaluation
HardwareApplication Server: 64 GB RAM/ 16 Core / 500 GB HDD, Data Server: 128 GB RAM/ 16 Core, 3 TB HDD
Topology
Terms &
Concepts Used
ISO 13379, 13381, 13374, 14224, 17359
Standardization
Opportunities
Requirements
  • Mandate of the key sensors based on the type of equipment
    Based on the type of equipment, the makers need to have the basic set on sensors imbibed onto the system. E.g. for a pump – it is important to measure the input flow and output flow rates, vibrations, rotation speed, lube oil temperature and pressure. This will guide the equipment manufactures to provide their customers and their data products to capture the minimum required data and understand the equipment performance.
  • Mandate for the organizations to expose the minimum and key parameters.
    The equipment owners need to enable the basic set of sensors for the equipment health and performance which are required for monitoring the asset from any failures.
  • Standards for data formats
    Each organization has a different way of capturing data and storing them in different formats. Due to which the solutions are not scalable across organizations though the product behind them is same. It takes customised efforts each time.
  • Guidelines for deciding the sampling frequency based on the type of data
    We see a need to have a specific set of guidelines to capture data at a minimum required sampling frequency. For e.g. a vibration sensor should capture data at least at 1 ms.
  • Guidelines for feature engineering
    There must be guidelines as to how the features need to be engineered for AI models. Lack of this would lead to more black box models not explaining how the models behave the way they do.
  • Guidelines for standardization of event types and codes
    There are multiple events which occur for an asset or in a manufacturing plant. Guidelines would help people capture the data in a similar fashion helping the industry to benchmark against one another and at industry level we can understand, which events are the most critical.
  • Guidelines for standardization of fault and error codes for an equipment or process
    Similar to events, it is also useful to capture fault, failure and error codes in a standard way.
  • Process guidelines for event related data (maintenance and work orders)
    Guidelines would help people capture the data in a similar fashion helping the industry to benchmark against one another and at industry level we can understand, which events are the most critical.
Challenges
& Issues
Compliance of organizations
Societal Concerns Description Promoting sustainable industries, and investing in scientific research and innovation, are all important ways to facilitate sustainable development.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description Data from an Oil & Gas Rig
Source Drilling Equipment
Type Time-Series Sensor Data
Volume (size)
Velocity 2.5 Billion+ Data Points each day
Variety Machine Data
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 37 ID: Use Case Name: Leveraging AI to enhance adhesive quality
Application
Domain
Manufacturing
Deployment
Model
On-premise systems
StatusIn operation
ScopeBatch/Continuous/Discrete Manufacturing (Deployed in 75+ manufacturing lines in 10+ countries; Specifically identified the contributors to quality; predict potential quality failures).
Objective(s)Enhance Adhesive Quality, Performance Benchmarking
Short
Description
(up to
150 words)
Cerebra IOT signal intelligence platform provides the ability to have a holistic perspective and understanding of the sensitivity of the key parameters affecting output quality and ability to monitor and control the process in real-time. This will avoid variations in yields, build-up of inventories and missed customer deadlines.
Complete Description Cerebra IOT signal intelligence platform ingested 3+ years of process data and sensor data regarding plant operations from temperature, rpm, torque and pressure sensors which were strapped on to industrial mixers. These are the mandatory sensors for the operations. Cerebra used its episode detection algorithms (deep learning) to filter signal from noise and specifically identify the contributors to quality (anomaly signatures) that can then be used as signals to predict quality. It used its proprietary N-dimensional Euclidian distance-based scoring algorithms to normalize and present a unified score to the business team. This unified health score provided the process team a different lens to benchmark, specifically target and radically improve process efficiencies. Cerebra then leveraged its sophisticated ensemble models to predict potential quality failures allowing the operations team to take real-time actions to control process deviations. The signals identified in the earlier steps provide Model Explainability to the end-user for reasons behind Quality deviation.
StakeholdersManufacturing industries; Suppliers and Buyers; Environment
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Challenges to accountability, New Security Threats.
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Prediction
Method(s)N-dimensional Euclidian distance-based scoring algorithms
HardwareApplication Server: 64 GB RAM/ 16 Core / 500 GB HDD, Data Server: 128 GB RAM/ 16 Core, 3 TB HDD
Topology
Terms &
Concepts Used
Deep learning; Anomaly Signatures
Standardization
Opportunities
Requirements
  • Mandate of the key sensors based on the type of equipment.
    Based on the type of equipment, the makers need to have the basic set on sensors imbibed onto the system. e.g. for a pump – it is important to measure the input flow and output flow rates, vibrations, rotation speed, lube oil temperature and pressure. This will guide the equipment manufactures to provide their customers and their data products to capture the minimum required data and understand the equipment performance.
  • Mandate for the organizations to expose the minimum and key parameters.
    The equipment owners need to enable the basic set of sensors for the equipment health and performance which are required for monitoring the asset from any failures.
  • Standards for Data Formats
    Each organization has a different way of capturing data and storing them in different formats. Due to this, the solutions are not scalable across organizations though the product behind them is same. It takes customised efforts each time.
  • Guidelines for deciding the sampling frequency based on the type of data.
    We see a need to have a specific set of guidelines to capture data at a minimum required sampling frequency, e.g. a vibration sensor should capture data at least at 1 ms or less.
  • Guidelines for Feature Engineering.
    There must be guidelines as to how the features need to be engineered for AI models. Lack of this would lead to more black box models not explaining how the models behave the way they do.
  • Guidelines for Standardization of event types and codes.
    There are multiple events which occur for an asset or in a manufacturing plant. Guidelines would help people capture the data in a similar fashion helping the industry to benchmark against one another and at industry level we can understand, which events are the most critical.
  • Guidelines for standardization of Fault and Error Codes for an equipment or process.
    Similar to events, it is also useful to capture fault, failure and error codes in a standard way.
  • Process Guidelines for event related data (Maintenance and Work Orders):
    Guidelines would help people capture the data in a similar fashion helping the industry to benchmark against one another and at industry level we can understand, which events are the most critical.
  • Guidelines for Training AI models:
    A defined set of guidelines for AI models would be useful for the data scientists to follow. It will also aid the consumers of AI models to understand how the outcome has been deduced.
Challenges
& Issues
Patented process if any, security restrictions
Societal Concerns Description Promoting sustainable industries, and investing in scientific research and innovation, are all important ways to facilitate sustainable development
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 51 ID: Use Case Name: Machine Learning Tools in Support of Transformer Diagnostics
Application
Domain
Performance evaluation and diagnostics
Deployment
Model
Prototype
StatusUnder development
ScopePower Transformers operation and maintenance
Objective(s)Use of Machine Learning (ML) algorithms as supporting tools for the automatic classification of power transformers operating condition
Short
Description
(up to
150 words)
The successful use of ML tools may find multiple applications in the industry such as providing fast ways of analysing new data streaming from online sensors, evaluating the importance of individual variables in the context of transformer condition assessment and also the need or adequacy of data imputation in the so widely common problem of missing data
Complete Description The work consists of training 12 ML algorithms with real data from 1,000 (one thousand) transformers that were individually analyzed by human experts. Each transformer in the database is scored with a ‘green’, ‘yellow’ or ‘red’ card depending on the data, the interpretation of human experts, or even after some calculations carried out by the company’s internal algorithms frequently utilized by the experts to identify units with technical operational issues.
The ML algorithms, however, do not utilize or are given any of the engineering tools employed by the human experts. The algorithms only employed the raw data in a supervised learning process in which a column named ‘Class’ was added to the transformer information with the classification red, yellow or green provided by the human expert.
StakeholdersTransformers end users
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Lack of enough data to perform the analysis
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Statistical learning
Method(s)12 ML methods used for the comparison exercise [1]:
Linear Algorithms
  1. General linear regression (logistic regression) - GLM
  2. Linear discriminant analysis - LDA
Non-linear Algorithms
  1. Classification and regression trees (CART and C5.0)
  2. Naïve Bayes algorithm (NB)
  3. K-Nearest Neighbor (KNN)
  4. Support Vector Machine (SVM)
Ensemble Algorithms
  1. Random Forest (stochastic assembly of a large number of CART algorithms)
  2. Tree Bagging (Tree Bagging)
  3. Extreme Gradient Boosting Machine (xGBM1 and xGBM2)
  4. Artificial Neural Networks (ANN)
HardwareStandard laptop [5]
TopologyNA
Terms &
Concepts Used
Machine Learning Algorithms, Transformer Diagnostics, Condition Assessment, Automated Tool
Standardization
Opportunities
Requirements
Standardization of asset performance data format and analysis [7]
Challenges
& Issues
Data availability, missing data, imbalanced classes [7]
Societal Concerns Description Safe and reliable power delivery [6]
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 52 ID: Use Case Name: Automated Travel Pattern Recognition using Mobile Network Data for Applications to Mobility as a Service
Application
Domain
Transportation
Deployment
Model
Activity- based Modelling for New mobility Services
StatusPoC
ScopeDetect automatically travel pattern recognition from anonymized and aggregated Mobile phone Network Data
Objective(s)Phase 1: Attribute trip purpose and mode of transport to multimodal door-to-door journeys from Mobile phone Network Dataset using AI and machine learning techniques (Activity based model)
Phase 2: Generate daily activities for static agents in the Agent Based Model
Phase 3: Optimisation of New Mobility services in integration with mass transit
Short
Description
(up to
150 words)
Activity- based modelling has the capability to exploit big data source generated by smart cities to create a digital twin of urban environments to test Mobility as a Service schemes. MND data have been used to create activities for an Agent Based Model. AI is used to automatically detect purpose and mode of transport in multimodal round trips, obtained by anonymized and aggregated MND trip-chains dataset. Data fusion techniques and SQL queries were also used to consider land use and facilities in the urban area of interest.
Complete Description Activity- based modelling has the capability to exploit big data source generated by smart cities to create a digital twin of urban environments to test Mobility as a Service schemes. Given the rise of location- based data and Mobile phone Network Data (MND) for transport modelling purpose, Agent based modelling has become a viable tool to explore a sustainable introduction of mobility services, exploring the integration with mass transit. AI is used in detecting purpose and mode of transport in multimodal round trips and assign purpose and mode of transport to trip- chains dataset coming from MND. The methodology has been developed for the Innovate UK funded Mobility on Demand Laboratory Environment (MODLE) project and will undergo a validation process during the Demand Modelling and Assessment through a Network Demonstrator (DeMAND) project for the Department for Transport (UK)
Stakeholders
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Assign purpose of each trip in the chain, assign model of transport for each trip in the chain, generate daily activity plans, generate static agents (users), generate dynamic agents (service)
Method(s)Agent Based Models with Activity based approach
HardwareNA
Topology
Terms &
Concepts Used
Data fusion, machine learning techniques
Standardization
Opportunities
Requirements
Challenges
& Issues
Societal Concerns Description
SDGs to
be achieved
Data Characteristics
Description
Source
Type
Volume (size)
Velocity
Variety
Variability
(rate of change)
Quality

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 29 ID: Use Case Name: Enhancing traffic management efficiency and infraction detection accuracy with AI technologies
Application
Domain
Transportation
Deployment
Model
Cloud services and on-premise systems
StatusIn operation
ScopeUtilizing AI technologies in traffic monitoring and management
Objective(s)To increase the accuracy and efficiency of infraction detection, traffic monitoring and flow analysis, while minimizing the human effort and the overall solution cost.
Short
Description
(up to
150 words)
Big data enabled AI technologies are applied to monitoring and managing the traffic in a large municipality in China. Multi-sourced data (traffic flow, vehicle data, pedestrian movement, etc.) is monitored, from which illegal operation of vehicles, unexpected incidents, surge of traffic etc. are detected and analysed with machine learning (ML) methods. ML tasks (including training and deployment) are carried out on a platform supporting the integration of various ML frameworks, models and algorithms. The platform is based on heterogeneous computing resources. The efficiency and accuracy of infraction detection, and the effectiveness of traffic management are significantly improved, with much reduced human effort and overall solution cost.
Complete Description With the population and the number of vehicles growing in large cities, managing the heavy traffic in urban areas has become a challenging yet essential task for the municipality. Addressing this issue has become particularly urgent for big cities in China, where millions of people live and commute every day.
In this use case, big data based AI technologies are applied to monitoring and managing the heavy traffic in a metropolitan in south China. Previously, significant human resources were involved in the vehicle and road monitoring, and large investment was made to the computing infrastructure specific to certain functionalities. To increase the efficiency of urban transportation, reduce the traffic jam and air pollution, as well as minimize the human effort, machine learning techniques (e.g. deep learning) are applied to image and video analysis, such as traffic flow analysis, infraction detection and incident detection. Example applications include but not limited to 1) detection of traffic rule violation, e.g. over-speeding, wrong driving lanes or parking. AI-enabled detection produces much faster and more accurate result, and helps in enforcing the traffic regulation. 2) traffic light optimization. Based on the modelling and analysis of multi-sourced traffic information (both real-time and historical data), traffic lights are dynamically configured to divert the flow, increase the passing speed of cars and reduce the traffic jam in major junctions.
The use of AI has obtained remarkable results: The infraction detection efficiency gets 10X increase, and the detection accuracy is greater than 95%. The urban area traffic jam is much alleviated, with vehicles’ passing speed through major junctions increases by 9%-25%.
StakeholdersUrban citizens (drivers and pedestrians), government, car companies, traffic administrative bureaus, logistics companies, etc.
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
Low quality pictures, insufficient processing capability
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
1  Features related to adulterants in radio spectrum  Intensities around NIR range
1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
1 Number of labors reduced % of labors improvement of productivity
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
2 Number of complaints reduced % of labor's complaint improvement of productivity
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
3 Lead time time from order to shipment improvement of productivity
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
6 Success rate Very high success rate is required for the solution to be accepted.
AI Features Task(s)Recognition
Method(s)Machine learning, Deep learning
HardwareHeterogeneous computing platform (CPU plus heterogeneous accelerators such as GPU, FPGA etc.)
Topology
Terms &
Concepts Used
Heterogeneous resource pooling, on-demand resource scheduling
Standardization
Opportunities
Requirements
  • Requirement of computing infrastructure to empower AI applications in the transportation domain, e.g. the integration of acceleration units (GPU, FPGA, etc.), dynamic scheduling and on-demand allocation of heterogeneous resources
  • Support of mainstream ML frameworks, and the algorithms and models from different vendors, to prevent vendor lock-in
  • Challenges
    & Issues
  • Constant improvement in hardware architecture to increase the performance and efficiency of running ML/DL tasks
  • Consistent interfaces between applications, ML engines and heterogeneous resource pools
  • Support of new models and emerging algorithms for growing functionalities
  • Societal Concerns Description AI’s application in urban transportation significantly improves the quality of life for urban citizens, reduces the time wasted in heavy traffic and the air pollution from vehicles.
    SDGs to
    be achieved
    Sustainable cities and communities
    Data Characteristics
    Description Traffic data (vehicle, road, and pedestrian data)
    Source Traffic camera
    Type Image, video
    Volume (size) approx. 100TB/day
    Velocity Stream and batch
    Variety Traffic flows, vehicle information, pedestrian information, etc.
    Variability
    (rate of change)
    Subject to random surge (rush hour, accident, etc.)
    Quality Vary (depending on the weather condition, environment etc.)

    Editor's comments and enhancements are shown in green. [ Reviewed]

    The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

    • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
    • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
    • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
    • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
    • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
    • [6] Functional aspects, trustworthiness, and societal concerns
    • [7] AI life cycle components include acquire/process/apply.
    These characteristics are identified in red in the use case.

    No. 30 ID: Use Case Name: Autonomous network and automation level definition
    Application
    Domain
    ICT
    Deployment
    Model
    Cyber-physical systems
    StatusPoC
    Scope
    Objective(s)To define autonomous network concept and automation level for the common understanding and consensus
    Short
    Description
    (up to
    150 words)
    With the goal of providing common understanding and consensus for autonomous self-driving network, this use case delivers a harmonized classification system and supporting definitions that:
  • Define the concept of autonomous network
  • Identify six levels of network automation from “no automation” to “full automation”.
  • Base definitions and levels on functional aspects of technology.
  • Describe categorical distinctions for a step-wise progression through the levels.
  • Educate a wider community by clarifying for each level what role (if any) operators have in performing the dynamic network operations task while a network automation system is engaged.
  • Complete Description The telecom CSPs have a dual challenge – to increase agility while reducing network operating cost.
    1. The exponential growth of network complexity e.g. 5G will make the traditional network O&M model unsustainable;
    2. Digital transformation accelerates service innovation but requires automation capabilities.
    As CSPs start to evaluate their digital transformation strategies, automation is a central concern. Some operators are already introducing automation to some of their network processes, most commonly O&M, planning and optimization. According to Analysys Mason, in 2018, 56% of CSPs globally have little or no automation in their networks. But by 2025, according to their own predictions, almost 80% expect to have automated 40% or more of their network operations, and one-third will have automated over 80%. The introduction of AI/ML (artificial intelligence/machine learning) will be an important part of that process for many CSPs, helping to make the network more intelligent, agile and predictive. The autonomous self-driving network has two essential elements in common with the autonomous self-driving car:
  • There are different levels of automation, relating to different timescales and scenarios
  • Intensive use of artificial intelligence (AI) is essential

    With the goal of providing common understanding and consensus for autonomous self driving network, this use case delivers a harmonized classification system and supporting definitions that set out six levels of automation for the network.

    Level

    Name

    Definition

    Execution

    (Hands)

    Awareness

    (Eyes)

    Decision

    (Minds)

    Experience

    (Hearts)

    System

    Capability

    0

    Manual

    Operation & Maintenance

    Even with auxiliary tools, O&M personnel perform all dynamic tasks.

    P

    P

    P

    P

    n/a

    1

    Assisted

    Operation & Maintenance

    Under the applicable design scope, the system can execute a sub-task repeatedly based on rules.

    P/S

    P

    P

    P

    Sub-task

    level

    2

    Partial

    Autonomous Network

    Under the applicable design scope, the system continuously completes the control task of a unit based on the model.

    S

    P

    P

    P

    Unit level

    3

    Conditional

    Autonomous Network

    Under the applicable design scope, the system can implement complete closed-loop automation of single-domain scenarios. Users can respond to the requests in a timely manner when the system fails.

    S

    S

    P

    P

    Domain level

    4

    Highly

    Autonomous Network

    Under the applicable design scope, the system can automatically analyze and execute cross-domain and service close-loop automation.

    S

    S

    P

    P

    Service level

    5

    Full

    Autonomous Network

    The system can perform complete dynamic tasks and exception handling in all network environments. O&M personnel do not need to intervene.

    S

    S

    S

    P/S

    All Modes


    P=Personnel (Manual), S=System (Automated)
    -Level 0 - manual O&M: The system delivers assisted monitoring capabilities, which means all dynamic tasks have to be executed manually.
    -Level 1 - assisted O&M: The system executes a certain sub-task based on existing rules to increase execution efficiency.
    -Level 2 - partial autonomous network: The system enables closed-loop O&M for certain units under certain external environments, lowering the bar for personnel experience and skills.
    -Level 3 - conditional autonomous network: Building on L2 capabilities, the system can sense real-time environmental changes, and in certain domains, optimize and adjust itself to the external environment to enable intent-based closed-loop management.
    -Level 4 - highly autonomous network: Building on L3 capabilities, the system enables, in a more complicated cross-domain environment, predictive or active closed-loop management of service and customer experience-driven networks. This allows operators to resolve network faults prior to customer complaints, reduce service outages and customer complaints, and ultimately, improve customer satisfaction.
    -Level 5 - full autonomous network: This level is the ultimate goal for telecom network evolution. The system possesses closed-loop automation capabilities across multiple services, multiple domains, and the entire lifecycle, achieving autonomous driving networks.

    The lower levels can be applied now and deliver immediate cost and agility benefits in certain scenarios. An operator can then evolve to the higher levels, gaining additional benefits and addressing a wider range of scenarios.
    Network automation is a long run objective with step-to-step process, from providing an alternative to repetitive execution actions, to performing perception and monitoring of network environment and network device status, making decisions based on multiple factors and policies, and providing effective perception of end user experience. The system capability also starts from some service scenarios and covers all service scenarios.

  • Stakeholders
    Stakeholders'
    Assets, Values
    Systems'
    Threats &
    Vulnerabilities
    Performance
    Indicators (KPIs)
    Seq. No. Name Description Reference to mentioned
    use case objectives
    1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
    1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
    1  Features related to adulterants in radio spectrum  Intensities around NIR range
    1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
    1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
    1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
    1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
    1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
    1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
    1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
    1 Number of labors reduced % of labors improvement of productivity
    1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
    1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
    1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
    1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
    1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
    1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
    1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
    1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
    1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
    1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
    2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
    2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
    2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
    2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
    2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
    2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
    2 Number of complaints reduced % of labor's complaint improvement of productivity
    2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
    2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
    2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
    2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
    2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
    2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
    3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
    3 Lead time time from order to shipment improvement of productivity
    3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
    3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
    3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
    3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
    4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
    4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
    5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
    5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
    6 Success rate Very high success rate is required for the solution to be accepted.
    AI Features Task(s)All
    Method(s)
    Hardware
    Topology
    Terms &
    Concepts Used
    Autonomous network, self-driving network
    Standardization
    Opportunities
    Requirements
    Challenges
    & Issues
    Societal Concerns Description
    SDGs to
    be achieved
    Industry, Innovation, and Infrastructure
    Data Characteristics
    Description
    Source
    Type
    Volume (size)
    Velocity
    Variety
    Variability
    (rate of change)
    Quality

    Editor's comments and enhancements are shown in green. [ Reviewed]

    The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

    • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
    • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
    • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
    • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
    • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
    • [6] Functional aspects, trustworthiness, and societal concerns
    • [7] AI life cycle components include acquire/process/apply.
    These characteristics are identified in red in the use case.

    No. 31 ID: Use Case Name: Autonomous network scenarios
    Application
    Domain
    ICT
    Deployment
    Model
    Cyber-physical systems
    StatusPoC
    ScopeCommunications network
    Objective(s)Clarification and showcases of autonomous network usage
    Short
    Description
    (up to
    150 words)
    Multiple scenarios of autonomous network enabled by AI is addressed for improving operational efficiency, customer experience and service innovation, including wireless network performance improvement, optical network failure prediction, data center energy saving etc.
    Complete Description The leading reason to adopt AI-assisted network automation is to reduce the cost – almost 80% operators placed this in their top three drivers, followed by:
  • improvement to customers’ network quality of experience
  • efficient planning and management of dense networks
  • part of an end-to-end automation strategy spanning the network and IT operations
    While OPEX reduction is the most important cost-related driver, others include better alignment of network costs to the revenue that is generated; and the ability to defer some capital expenditure (CAPEX) by using existing assets more efficiently.

    Obviously, the autonomous self-driving network needs to move from an O&M approach that is focused on network elements, to one based on usage scenarios. This means that process changes relate directly to a particular result, defined by the operator, and with a business value. Progress will be accelerated if a core set of scenarios is defined, which will be of value to all operators. Development of the related autonomous self driving network solutions can then be prioritized accordingly.

    The criteria for the selection of scenarios as follows:

  • Extent of digitalization: Reflects the technical readiness of the scenarios. Digitalization is the foundation of automation, and the extent to which it is supported determines the extent to which automation can be achieved immediately;
  • TCO contribution: Reflects OPEX savings and the improvement to CAPEX efficiency in the given scenario;
  • O&M life cycle: Reflects the ability to build differentiation in each phase of the life cycle in order to achieve full autonomous driving across many scenarios. The O&M life cycle spans planning, deployment, maintenance, optimization and provisioning of the network and scenarios have been identified for each one.
    Based on those three criteria, we selected six typical key scenarios for the purpose of illustration and clarification.

    Scenario 1: Base Station Deployment

    1. Definition and Description of Scenario
      The base station deployment scenario refers to the entire process after site survey, including network planning and design, site design, configuration data preparation, site installation, site commissioning and site acceptance.
    2. Automation Classification

      Level 1: The O&M tool helps some elements of the process to be automated, but configuration and site acceptance have to be done manually.

      Level 2: Some hardware can be detected and configured automatically, and configuration data is simplified based on rules.

      Level 3: E2E automation: radio parameter self-planning, hardware self-detection and self-configuration, self-acceptance without dialing test.

      Initial outcomes: Upon the usage of AI, some initial results are achieved as follows:
      -Site Deployment Time Shortened by 30%
      -Feature Deployment Time Shortened by 60%
      -Performance Converging Shortened by 85%

    Scenario 2: Network Performance Monitoring

    1. Definition and Description of Scenario
      The mobile network has entered the stage of very precise planning sites and resources: on the one hand, to identify and forecast high traffic areas, and allocate resources precisely to support business goals; on the other hand, to identify and forecast high-frequency temporary traffic, scheduling resources to meet business objectives.
    2. Automation Classification
      Level 1: Network quality is consistent, and network anomalies can be discovered by tools;
      Level 2: 3D presentation of network quality and anomalies, and network planning is self-generated;
      Level 3: E2E closed-loop monitoring and planning: predicting network development according to historical network information, finding value areas and hidden problems, recommending the best network planning and estimating the gain automatically.

    Scenario 3: Fault Analysis and Handling

    1. Definition and Description of Scenario
      The security and reliability is the most important mission of the network, so quick alarm detection and quick fault healing are important. The fault analysis and handling scenario comprises several steps, including alarm monitoring, root cause analysis, and fault remediation.
      Monitoring: Real-time monitoring of network alarm, performance, configuration, user experience, and other information.
      Analysis: By analyzing the correlation between alarms and other dimensions data, root cause of fault and fault repairing can be achieved quickly.
      Healing: Repair fault remotely or by site visiting based on the repairing suggestions.
    2. Automation Classification
      Level 1: Some tools are used to simplify alarm processing, but thresholds and alarm correlation rules are set manually based on expert experience.
      Level 2: Automatic alarm correlation and root cause analysis.
      Level 3: Closed-loop of alarms analysis and handling process: Based on the intelligent correlation analysis of multi-dimensional data, accurate location of alarm root cause, precise fault ticket dispatching, and fault self-healing could be reached successfully.
      Level 4: Proactive troubleshooting: Based on the trend analysis of alarms, performance, and network data, alarms and faults could be predicted and rectified in advance.

      Initial outcomes: Upon the usage of AI, some initial results are achieved as follows:

      -Reduction of alarms: 90%

    Scenario 4: Network Performance Improvement

    1. Definition and Description of Scenario
      Wireless networks are geographically very distributed, and activity varies significantly in different places and at different times of day. This makes the network very dynamic and complex. That complexity is further increased by the diversity of services and of terminal performance, and by the mobility of users. If the network cannot achieve the benchmark KPIs or SLAs (service level agreements), or enable good user experience, it must be adjusted to meet or exceed those requirements.

      This is the function of network performance improvement or optimization.

      The complete process of network performance improvement or optimization includes several stages:

      • network monitoring and evaluation
      • root cause analysis of performance problems
      • optimization analysis and optimization decision-making
      • optimization implementation
      • post- evaluation and verification

    2. Automation Classification
      Level 2: Drive test evaluation is not required for coverage optimization. Adjustment suggestions are provided automatically.
      Level 3: Closed-loop of network performance improvement:

      Automatic identification of network coverage and quality problems, automatic configuration of performance parameters, and automatic evaluation.

      Level 4: Dynamic adjustment is implemented based on the scenario awareness and prediction to achieve the optimal network performance. Network prediction capability is available: scenario change trends could be perceived, and network configuration could adjusted real-time to achieve optimal performance.

      Initial outcomes: Upon the usage of AI, some initial results are achieved as follows:

      -Capacity increase: 30%,

      -Delivery duration: 2 weeks, non-manual

    Scenario 5: Site Power Saving

    1. Definition and Description of Scenario
      T Site power consumption cost accounts for more than 20% of network OPEX. Although network traffic declines greatly during idle hours, equipment continues to operate, and power consumption does not dynamically adjust to the traffic level, resulting in waste. It is necessary to build the "Zero Bit, Zero Watt" capability.
    2. Automation Classification
      Level 2: Tool aided execution;
      Level 3: Power-saving closed-loop: Based on the analysis of traffic trends, self-adaptive generation of power-saving strategies, effect and closed-loop KPI feedback;
      Level 4: Real-time adjustment of power-saving strategies based on traffic prediction. Through integration with third-party space-time platforms, the operator can also add predictive perception of traffic changes, smooth out the user experience, and maximize power-saving.

      Initial outcomes: Upon the usage of AI, some initial results are achieved as follows:

      -Power saving: 10~15%

    Scenario 6: Wireless Broadband Service Provisioning

    1. Definition and Description of Scenario
      WTTx has become a foundational service for mobile operators because of its convenient installation and low cost of single bit. Rapid launch of WTTx service, accurate evaluation after launch, and network development planning have become important supports for new business development.
    2. Automation Classification
      Level 1: Blind launch;

      Level 2: Automation tools to assist the launch, check the coverage and capacity of the user's location before the business hall, and experience evaluation;

      Level 3: Closed-loop for business launch: Integrated with BOSS system to achieve one-step precise launch, remote account launching, CPE installation, fault self-diagnosis and complaint analysis;

      Level 4: Auto-balancing of multi-service, automatic value areas identification and network planning recommendation.

  • StakeholdersCommunications Service Providers, Suppliers, Industrial and consumer users
    Stakeholders'
    Assets, Values
    Systems'
    Threats &
    Vulnerabilities
    incorrect AI system use
    Performance
    Indicators (KPIs)
    Seq. No. Name Description Reference to mentioned
    use case objectives
    1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
    1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
    1  Features related to adulterants in radio spectrum  Intensities around NIR range
    1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
    1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
    1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
    1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
    1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
    1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
    1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
    1 Number of labors reduced % of labors improvement of productivity
    1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
    1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
    1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
    1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
    1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
    1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
    1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
    1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
    1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
    1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
    2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
    2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
    2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
    2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
    2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
    2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
    2 Number of complaints reduced % of labor's complaint improvement of productivity
    2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
    2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
    2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
    2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
    2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
    2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
    3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
    3 Lead time time from order to shipment improvement of productivity
    3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
    3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
    3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
    3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
    4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
    4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
    5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
    5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
    6 Success rate Very high success rate is required for the solution to be accepted.
    AI Features Task(s)All
    Method(s)Machine learning, deep learning, Knowledge graph, decision making & reasoning, analytics
    HardwareAI training and inference system, and network management system
    TopologyEnd-to-end
    Terms &
    Concepts Used
    Autonomous network, self-driving network
    Standardization
    Opportunities
    Requirements
    None
    Challenges
    & Issues
    Data usage and sharing, human expertise & competence
    Societal Concerns Description None
    SDGs to
    be achieved
    Industry, Innovation, and Infrastructure
    Data Characteristics
    Description
    Source
    Type
    Volume (size)
    Velocity
    Variety
    Variability
    (rate of change)
    Quality

    Editor's comments and enhancements are shown in green. [ Reviewed]

    The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

    • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
    • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
    • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
    • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
    • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
    • [6] Functional aspects, trustworthiness, and societal concerns
    • [7] AI life cycle components include acquire/process/apply.
    These characteristics are identified in red in the use case.

    No. 62 ID: Use Case Name: A judging support system for gymnastics using 3D sensing
    Application
    Domain
    ICT
    Deployment
    Model
    On-premise systems
    StatusPoC
    ScopeSkeleton recognition for gymnastics
    Objective(s)To support judgement of difficult element by high-level and high-speed.
    Short
    Description
    (up to
    150 words)
    We have been developing a judging support system for artistic gymnastics to enhance accuracy and fairness in judging. We developed a skeleton recognition technique using the learned model that we trained using a large amount of depth images of gymnastics created from CG in advance. With this technology, it is possible to recognize a human 3D skeleton from depth image.
    Complete Description In gymnastics, wrong scoring is a problem, when it is difficult to judge by high-level and high-speed. Therefore, 3D sensing technology is required to reduce burden of referee by recognizing skeleton of gymnast. We developed a technique to recognize heatmaps of body parts using the learned model that we trained using a large amount of depth images of gymnastics created from CG in advance. We calculate 3D skeleton position using heatmaps of body parts. With this technology, it is possible to recognize a human 3D skeleton from depth image.
    StakeholdersFederation International Gymnastics(FIG)
    Stakeholders'
    Assets, Values
    Systems'
    Threats &
    Vulnerabilities
    Performance
    Indicators (KPIs)
    Seq. No. Name Description Reference to mentioned
    use case objectives
    1 Coverage Ratio of potential issues which are "of interest" for human evaluation. Ideal target is to reduce the current volume by 80%. Improve accuracy
    1 Precision  Correctly Predicted Anomalous scenarios/ Total Anomalous scenarios predicted 
    1  Features related to adulterants in radio spectrum  Intensities around NIR range
    1 Average vehicle driving speed Average vehicle driving speed on all the road sections in a given region Improve the road utilization efficiency
    1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
    1 Accuracy The number of correctly recognized users’ intent over total number of users. Currently, accuracy reaches 95%. Improve accuracy of recognizing users’ intent
    1 Classifier Accuracy  Without straightening and pre-processing, the average classification accuracy obtained was 68.5%. However, with preprocessing, the classification accuracy improved to 86.7%. These results are very likely to improve with more annotated training data for classification. 
    1 Closeness to Golden Batch How close a process is to the best possible batch Helps in isolation of bad batches from good batches by identifying combination of process variable trajectories that lead to good or bad batch operation.
    1 Model Accuracy Accuracy of the prediction model The extent to which the setpoints have correctly predicted
    1 Ratio of ML discovered failure rate to nominal failure rate What combination of manufacturing processes/decisions leads to higher failure rates compared to nominal failure rate Actionable intelligence to improve the manufacturing process of HV circuit breakers
    1 Number of labors reduced % of labors improvement of productivity
    1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
    1 MIoU (Mean Intersection over Union) The intersection of prediction area and actual area divided by the union of the predicted area and the actual area. Ideal target is 100%. Improve accuracy
    1 Classification Ratio Real to Pseudo wrong classification Establishes the quality of identification
    1 Ease of use Simplicity and efficiency during initial learning. Teaching process should be easy.
    1 Zone of Influence/ Thermal Correlation Index Extent of influence of ACUs on data center racks. Helps in improved control.
    1 Invisible Loss Time Indicates the lost time of the asset in being idle or off or unplanned downtime Asset Utilization Reports indicate the effectively utilized time there indicating the lost time and their causes
    1 Prediction Accuracy To what extent has the model been able to predict correctly Provided ability as to % of times the quality complied
    1 Algorithm accuracy Output when compared to the human expert analysis of the same data See Reference
    1 Generation of Activities (land use information and time of travel) Purpose of activities is assigned based on land use information and time of travel. Census data and national/ local travel surveys will provide validation for the process Phrase 1
    1 accuracy The accuracy of infraction and incident detection from traffic pictures/videos To increase the accuracy of traffic monitoring and inspection
    2 Split Proportion of the potential issues which are "more likely to be a valid issue" for our end users. Improve efficiency
    2 Recall Correctly Predicted Anomalous scenarios /Total Anomalous Scenarios 
    2 Average vehicle waiting time Average vehicle waiting time at all the intersections in a given region Improve the road utilization efficiency
    2 Resolution The number of answers solved over total number of questions asked Improve the resolution of questions from users
    2 Annotation Completeness  35.9 chromosomes segmented out after crowd annotation, for 50 images having 46 chromosomes 
    2 % Reduction in Calibration Time The amount of time saved from manually setting the calibration
    2 Number of complaints reduced % of labor's complaint improvement of productivity
    2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
    2 FAR (false acceptance rate) Negative samples are identified as positive samples / Total number of negative samples.The low FAR, the more smartphone will get correct scenes and objects Improve accuracy
    2 Training efficiency Amount of necessary data for training might lead to practical obstacles in application.
    2 Overall drilling time The time spent on one drilling job inclusive of the all downtimes Real Time visibility into operations gives the operations early warnings to take actions immediately.
    2 Generation of agents (travel times, speed on link) Agents generated will build up in the network creating realistic conditionsw of congestion. Speed on links. Phrase 2
    2 split Proportion of images requiring human inspection. The less the split, the higher the efficiency. To minimize the human effort in inspection
    3 Satisfaction The number of users who are satisfied with customer service over total number of users Improve user experience
    3 Lead time time from order to shipment improvement of productivity
    3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
    3 Initial success rate After initial training, the success rate needs to be acceptable such that the system can be put in the production line.
    3 Opeartion of service (number of users for the service) Optimisation of route and operation time in the day. Validation provided using data collected by Mobility service operators during the operation of service Phrase 3
    3 resource utilization ratio Achievable resource utilization ratio in the hardware infrastructure ( the higher the utilization ratio, the lower amount the required resource) To reduce the infrastructure investment and overall solution cost
    4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
    4 Speed of improvement Higher convergence speed of the reinforcement algorithm is making the solution more attractive.
    5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
    5 Operational efficiency Cycle time is the primary measure in manufacturing industry.
    6 Success rate Very high success rate is required for the solution to be accepted.
    AI Features Task(s)Recognition
    Method(s)Deep learning
    Hardware
    TopologyCNN
    Terms &
    Concepts Used
    Deep learning, Convolution neural network, training, training data set
    Standardization
    Opportunities
    Requirements
    Challenges
    & Issues
    Challenges: Recognize skeleton of all gymnastics element. Issues: Recognize 3D skeleton in gymnastics that are complex movements from depth image.
    Societal Concerns Description Positive: Fairness of scoring, reducing burden of referee, and technical improvement of gymnast. Negative:
    SDGs to
    be achieved
    Industry, Innovation, and Infrastructure
    Data Characteristics
    Description Depth images, 2D data of skeleton
    Source Motion capture
    Type Images
    Volume (size)
    Velocity Non-real time
    Variety Single dataset
    Variability
    (rate of change)
    Static
    Quality High
    Scenario Conditions
    No. Scenario
    Name
    Scenario
    Description
    Triggering Event Pre-condition Post-Condition
    1 Training Train multiple models (deep learning, Bayesian network, Time series analysis) for recognizing traffic flow volume and abnormal values in the input data
    1 Training Train a model (deep neural network) with training samples
    1 Training Based on millions of labeled streaming data, train a model using diversified algorithms, such as a deep learning neural network or a traditional machine learning algorithm
    1 Data Augmentation Using reverse translation and noise processing to increase the size and diversity of data. Increase the performance of model training.
    1 Training Train a model (e.g. neural network) with training samples
    1 Training Train a model with training data set Evaluation
    2 Optimization Based on the data processed by the trained models, optimize the period length, split, and key phase offsets among multiple intersections for traffic signal timing plans Completion of missing values or abnormal values processings
    2 Evaluation Evaluate whether the trained model can be deployed
    2 Evaluation Evaluate the performance of the model on online dialogue data Each requirement must be satisfied or exceeded to reach the condition of 'success' (e.g. the accuracy should be more than 95%)
    2 Model Training Based on the large training data, with deep learning method, to develop model for sentiment classification (7 categories) or sentiment intensity (3 categories).
    2 Evaluation Evaluate whether the model is properly trained for the detection Meeting KPI requirements (e.g. accuracy, split) of the particular case
    2 Evaluation Evaluate whether the trained model can be deployed cg data Training/Retraining Execution
    3 Evaluation Pre-evaluate the execution effects of the optimized traffic signal timing plans, which include the period lengths, splits, and key phase offsets among multiple intersections Input prediction of traffic flow situation in the next period The pre-evaluated execution effects of the optimized traffic signal timing plan is superior to the current one
    3 Execution Detect defects (regions including defects) using the trained model The trained model has been evaluated as deployable
    3 Execution Apply the trained model to predict user’s intent
    3 Evaluation Evaluate data performance on open dataset and specific data.
    3 Execution Deploy the model for infraction detection and traffic analysis The model has been evaluated as properly trained.
    3 Execution Recognize real data gained 3D laser sensor Evaluation Retraining
    4 Execution Execute the optimized traffic signal timing plan The pre-evaluated execution effects of the optimized traffic signal timing plan is superior to the current one
    4 Retraining Retrain a model with training samples
    4 Retraining Take a training sample from online dialogue to retrain the model and compare it with the old one by AB test The requirement is that the new model must be better than the old one
    4 Execution Apply the trained model on real-time AI customer service. The trained model has been evaluated as deployable
    4 Retraining Retrain a model with training samples
    4 Retraining Retrain a model with added training data set. Execution
    Trainng Scenario Name:  
    Step No. Event Name of
    Process/Activity
    Primary
    Actor
    Description of
    Process/Activity
    Requirement
    1 Dataset is ready Transform video data into structured data AI provider Transform video data into structured data by deep learning
    1 Training Train a model (deep neural network) with training samples Sample raw dataset is ready
    1 Raw data stored in the database Data extraction Database engineer Extract related data from the database to generate the raw dataset
    1 Complete data augmentation Design model for training AI algorithm engineers Using CNN for sentiment classification and intensity.
    2 Completion of Step 1 Data clustering AI provider Recognize abnormal value patterns and label them in internet data, induction coil data, and structures video data by data clustering
    2 Evaluation Evaluate whether the trained model can be deployed Completion of training/retraining
    2 Completion of Step 1 Generating training samples Data labeling team Label the raw dataset of step one with 300 categories
    2 Complete model designing Transfer learning AI algorithm engineers Multi-task learning with different data in same domain.
    3 Completion of Step 2? Processing of missing value and abnormal value AI provider Recognize abnormal value and process them, and fill missing values by data clustering, time series analysis and Bayesian network
    3 Execution Detect defects (regions including defects) using the trained model Completion of deployment in EMR system The trained model has been evaluated as deployable
    3 Completion of Step 2 Pre-process AI engineer Segment the sentence into words and convert those words into vectors
    4 Completion of Step 3 Data fusion AI provider Compute traffic status parameters such as traffic volume, vehicle driving speed, etc. by fusing internet data, induction coil data and structured video data
    4 Retraining Retrain a model with training samples Certain period of time has passed since the last training/retrainig
    Specification of training data
    Scenario Name Evaluation
    Step No. Event Name of
    Process/Activity
    Primary
    Actor
    Description of
    Process/Activity
    Requirement
    1 Completion of optimization Construct the evaluation model of the traffic signal timing plan AI provider Construct the evaluation model of the traffic signal timing plan based on traffic engineering theory
    2 Completion of Step 1 Evaluate the effect of the computed traffic signal timing plan Traffic administrator Pre-evaluate the effect of the computed traffic signal timing plan with the evaluation model
    1 Completion of training/retraining Preparation AI solution provider Transform sample raw data from EMR system to server on cloud
    2 Completion of Step 1 Detection AI solution provider Given the image data from Step 1, detect defects (regions including defects) using the deep neural network trained in the scenario of training
    3 Completion of Step 2 Evaluation Manufacturer Compare the result of Step 2 with that of human inspection
    1 Certain period of time has passed since the last training/retraining Data Extraction Database engineer Randomly take a sample from streaming data to form a test sample
    2 Completion of Step 1 Prediction AI engineer Predict the test sample in step 1 by the trained model
    3 Completion of Step 2 Evaluation Data labeling team Compare the result of predicted with the result of labeling
    1 Complete model training Evaluation on open dataset AI algorithm engineers Evaluate different models’ performance on open dataset
    2 Complete model training Evaluation on own dataset AI algorithm engineers Evaluate different models’ performance on own dataset
    Input of Evaluation
    Output of Evaluation
    Scenario Name Execution
    Step No. Event Name of
    Process/Activity
    Primary
    Actor
    Description of
    Process/Activity
    Requirement
    1 Completion of evaluation Execute the computed traffic signal timing plan Traffic administrator Execute the computed traffic signal timing plan The pre-evaluated execution effects of the optimized traffic signal timing plan is superior to the current one
    1 Acquire the user’s query pre-process AI engineer pre-process The trained model has been in operation
    2 Completion of Step 1 Text classification AI engineer Text classification
    3 Completion of Step 2 Response AI trainer Response
    1 Finish model training Application AI engineers Application
    2 Given customer’s input Data processing AI algorithm engineers Data processing
    3 Finish data processing Model prediction AI algorithm engineers Model prediction
    4 Completion of Step3 Making response AI algorithm engineers Making response
    Input of Execution
    Output of Execution
    Scenario Name Retraining
    Step No. Event Name of
    Process/Activity
    Primary
    Actor
    Description of
    Process/Activity
    Requirement
    1 Certain period of time has passed since the last training/retraining Data extraction Database engineer
    2 Completion of Step 1 Labeling the sample Data labeling team
    3 Completion of Step 2 Model training AI engineer
    4 Completion of Step 3 AB Test AI engineer The performance of the new model is better than results of the old one
    5 Completion of Step 4 Online active of new model AI engineer
    1 Certain period of time has passed since the last training/retrainig Improve architecture of model AI algorithm engineers
    2 Certain period of time has passed since the last training/retrainig Collecting new data AI algorithm engineers
    3 Completing Step1&Step2 Model retraining AI algorithm engineers
    Specification of retraining data
    References
    No. Type Reference Status Impact of
    use case
    Originator
    Organization
    Link
    1 Conference
    Proceedings
    Published in SPIE Proceedings Vol. 9860: Hyperspectral Imaging Sensors: Innovative Applications and Sensor Standards 2016 David P. Bannon, Editor(s)  Tata Consultancy Services Limited  Link
    1 Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, Gautam Shroff, LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection,  Link
    1 Working Group on Research and Innovation of the Plattform Industrie 4.0. Aspects of the Research Roadmap in Application Scenarios, Working Paper, German Federal Ministry for Economic Affairs and Energy, 2016. Link
    1 patent ZHANG MAOLEI;WEI LIXIA;CHEN XIAOMING;LI JIN.?”Crossing traffic jam judging and control method and system based on sensing detectors ”.CN201310395431.2013 QINGDAO HISENSE TRANS TECH CO Link
    1 Link
    1 Paper Convolutional Neural Networks for Sentence Classification New York University Link
    1 Sharma, Monika & Saha, Oindrila & Sriraman, Anand & Hebbalaguppe, Ramya & Vig, Lovekesh & Karande, Shirish. (2017). Crowdsourcing for Chromosome Segmentation and Deep Classification. 786-793. 10.1109/CVPRW.2017.109. 
    1 Working Group on Research and Innovation of the Plattform Industrie 4.0. Aspects of the Research Roadmap in Application Scenarios, Working Paper, German Federal Ministry for Economic Affairs and Energy, 2016. Link
    1 Working Group on Research and Innovation of the Plattform Industrie 4.0. Aspects of the Research Roadmap in Application Scenarios, Working Paper, German Federal Ministry for Economic Affairs and Energy, 2016. Link
    1 Conference Kumar, S., K., Jamkhandi, A., G., and Gugaliya, J., K., Achieving Manufacturing Excellence through Data Driven Decisions, IEEE International Conference on Industrial Technology, Melbourne Australia PP 1267-1273 Presented in Feb 2019 Use case taken from this reference ABB Link
    1 Conference Jeffy, F., J., Gugaliya, J., K., and Kariwala, V. Application of Multi-Way Principal Component Analysis on Batch Data, 2018 UKACC 12th International Conference on Control Published Use case taken from this source ABB Link
    1 Web Page Accelerating shale production through digital technology integration Published Use case taken from this source Flutura Business Solutions Pvt. Ltd., TechnipFMC Link
    1 company's technical journal Published Hitachi, Ltd., Link
    1 IT company XiaoIce In operation Microsoft Asia
    1 Publication B. Kuhlenkötter, X. Zhang, C. Krewet, Quality Control in Automated Manufacturing Processes – Combined Features for Image Processing Acta Polytechnica Vol. 46 No. 5/2006. Published Use case taken from this reference Czech Technical University Link
    1 Conference Fan Dai, Arne Wahrburg, Björn Matthias, Hao Ding: Robot Assembly Skills Based on Compliant Motion Proceedings of 47th International Symposium on Robotics (ISR 2016), At Munich, Germany Published Cited to support the detailed description ABB Link
    1 Conference Causality-based Thermal Prediction for Data Center. 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA). Turin, Italy. 4-7 Sept. 2018. Published Use case taken from this reference ABB Link
    1 Web Page Upstream Sensor Data + Big Data Analytics = Game Changer in Oil n Gas industry Published Use case take from this case study Flutura Business Solutions Pvt. Ltd. Link
    1 Web link Leveraging Cerebra’s AI to enhance quality – from Quality Inspection to Quality Assurance Published as case study Use case take from this case study Flutura Business Solutions Pvt. Ltd. Link
    1 Conference Cheim, Luiz V., Machine Learning Tools in Support of Transformer Diagnostics, Cigre General, Session Paris 2018, paper reference A2-206 Presented in August 2018 Use case taken from this reference ABB Link
    1 Franco P, Johnston R, McCormick (2019) Demand Responsive transport: generation of activity patterns from mobile phone network data to support the operation of flexible mobility services. - Special issue of Transportation Research Part A (TRA) on developments in Mobility as a Service (MaaS) and intelligent mobility (forthcoming) Link
    1 ?Journal Published online Huawei Technologies Co., Ltd. Link
    1 Press release Fujitsu Link
    2 Conference
    Proceedings
    Sakti Saurav, Pankaj Malhotra, Vishnu TV, Narendhar Gugulothu, Lovekesh Vig, Puneet Agarwal, Gautam Shroff, Online anomaly detection with concept drift adaptation using recurrent neural networks, CoDS-COMAD '18, Proceedings of the ACM India Joint International Conference on Data Science and Management of Data, Goa, India — January 11 - 13, 2018 
    2 Working Group on Research and Innovation of the Plattfom Industrie 4.0 and Alliance Industrie du Futur: Plattform Industrie 4.0 & Alliance Industrie du Futur : Common List of Scenarios. 2018. Link
    2 patent ZHANG MAOLEI;WEI LIXIA;CHEN XIAOMING;LIU XIN;LIU HONGMEI;LI JIN.“Multi-strategy and multi-object self-adaptation traffic control method”. CN201310548921.2013 QINGDAO HISENSE TRANS TECH CO Link
    2 Patent A medical symptom knowledge base classification system construction algorithm and device based on lexical cluster similarity In application Link
    2 Paper Hierarchical Attention Networks for Document Classification Carnegie Mellon University, Microsoft Research, Redmond Link
    2 Working Group on Research and Innovation of the Plattfom Industrie 4.0 and Alliance Industrie du Futur: Plattform Industrie 4.0 & Alliance Industrie du Futur : Common List of Scenarios. 2018. Link
    2 Working Group on Research and Innovation of the Plattfom Industrie 4.0 and Alliance Industrie du Futur: Plattform Industrie 4.0 & Alliance Industrie du Futur : Common List of Scenarios. 2018. Link
    2 Web Page Fundamentals of meter provers and proving methods Published Fundamental definition of Meter Provers Flow Management Devices Link
    2 Conference Te Tang, Hsien-Chun Lin, Masayoshi Tomizuka, A learning-based framework for robot peg-hole-insertion, Proceedings of the ASME 2015 Dynamic Systems and Control Conference, October 28-30, 2015, Columbus, Ohio, USA Published Cited to support the detailed description University of California Link
    2 Web Page Cerebra creating game changing impact on upstream outcomes Published Use case take from this case study Flutura Business Solutions Pvt. Ltd. Link
    2 Franco P, Johnston R, McCormick E (2018) Role of Intelligent Transport Systems applications in the uptake of mobility on demand services, United Nation “Transport and Communications Bulletin for Asia and the Pacific, 2018, No. 88 - Intelligent Transport Systems”. Link
    2 News article Louise Radnofsky. The Robots Are Coming (to Judge Gymnastics). The international gymnastics federation may add new technology in time for the 2020 Olympics—and pave the way for artificial intelligence in judged sports. The Wall Street Journal, Aug. 23, 2019. Fujitsu Link
    3 Communication Promoters Group of the Industry-Science Research Alliance and German National Academy of Science and Engineering. Recommendations for implementing the strategic initiative INDUSTRIE 4.0, Final report of the Industrie 4.0 Working Group, April 2013. Link
    3 patent WANGMENGJIA;MINWANLI.” Road traffic optimization method and device and electronic equipment”. CN201710081075.2017 ALIBABA GROUP HOLDING LTD; Link
    3 Patent Electronic medical record named entity recognition method and device combining Section feature information In application Link
    3 Paper LIBLINEAR: A library for large inear classification Journal of Machine Learning Research National Taiwan University Link
    3 Communication Promoters Group of the Industry-Science Research Alliance and German National Academy of Science and Engineering. Recommendations for implementing the strategic initiative INDUSTRIE 4.0, Final report of the Industrie 4.0 Working Group,, April 2013 Link
    3 Communication Promoters Group of the Industry-Science Research Alliance and German National Academy of Science and Engineering. Recommendations for implementing the strategic initiative INDUSTRIE 4.0, Final report of the Industrie 4.0 Working Group, April 2013. Link
    3 Publication Fares J. Abu-Dakka, Bojan Nemec, Aljaž Kramberger, Anders Glent Buch, Norbert Krüger and Aleš Ude, Solving peg-in-hole tasks by human demonstration and exception strategies, Industrial Robot: An International Journal 41/6 (2014) 575–584 Published Cited to support the detailed description Jožef Stefan Institute, Dept. of Automatics, Biocybernetics, and Robotics, Slovania, Maersk Mc-Kinney Moller Institute, University of Southern Denmark Link
    3 Franco P, McCormick E, Johnston R (2018) Multimodal activity Modelling for supporting mobility service operations, ITS World Congress Copenhagen, 17-21 September 2018
    4 Bo-hu LI, Bao-cun HOU, Wen-tao YU, Xiao-bing LU, Chun-wei YANG. Applications of artificial intelligence in intelligent manufacturing: a review. Frontiers of Information Technology & Electronic Engineering. 2017
    4 patent HUA XIANSHENG” Assessment method and device of traffic condition”. CN201610645412.2016 ALIBABA GROUP HOLDING LTD; Link
    4 Patent Algorithm and device for recognizing nested medical named entities based on two-layer recurrent neural network In application Link
    4 Christoph Legat and Birgit Vogel-Heuser. A configurable partial-order planning approach for field level operation strategies of PLC-based industry 4.0 automated manufacturing systems. Engineering Applications of Artificial Intelligence 66:128-144, DOI: 10.1016/j.engappai.2017.06.014,02017.
    4 Birgit Vogel-Heuser, Jay Lee, and Paolo Leitao. Agents enabling cyber-physical production systems. at – automatisierungstechnik 63(10). DOI: 10.1515/auto-2014-1153, 2015.
    4 Publication ?Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, Martin Riedmiller, Leveraging Demonstrations for Deep Reinforcement, Learning on Robotics Problems with Sparse Rewards, arXiv:1707.08817v2 [cs.AI] 8 Oct 2018 Published Cited to support the detailed description Deepmind Link
    4 Franco P, McCormick E, Van Leeuwen K, Ryan Johnston, Gregor Engelmann (2017) Multi-Modal Activity-Based Models to support Flexible Demand Mobility Services. ITS World Congress 2017, Montreal 29 October- 2 November 2017. Awarded Best Paper
    5 Lee, Jay, Hung-An Kao, and Shanhu Yang. "Service innovation and smart analytics for industry 4.0 and big data environment." Procedia Cirp 16 (2014): 3-8.
    5 patent HUA XIANSHENG,REN PEIRAN,SHEN CHEN,CHU WENQING,LIU YAO.” Road intersection traffic flow control method and device”. CN201610644132.2016 ALIBABA GROUP HOLDING LTD; Link
    5 Patent Algorithm and device for unsupervised keyword-based medical image report key information extraction In application Link
    5 Birgit Vogel-Heuser, Jay Lee, and Paolo Leitao. Agents enabling cyber-physical production systems. at – automatisierungstechnik 63(10). DOI: 10.1515/auto-2014-1153, 2015.
    5 Weyer, Stephan, et al. "Towards Industry 4.0-Standardization as the crucial challenge for highly modular, multi-vendor production systems." Ifac-Papersonline 48.3 (2015): 579-584.
    5 Publication Mel Vecerik, Oleg Sushkov, David Barker, Thomas Roth¨orl, Todd Hester, Jon Scholz, A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning, arXiv:1810.01531v2 [cs.RO] 8 Oct 2018 Published Cited to support the detailed description Deepmind Link
    5 Proceedings Franco P, McCormick E, Van Leeuwen K (2017) Framework for modelling MaaS using ABM and real-time data from ride-sharing services. 12ve ITS Europe Congress 2017, Strasbourg, 19-22 June 2017.
    6 paper Liang Yu,Jingqiang Yu,Maolei Zhang?Xin Zhang?Yuehu Liu.”Large Scale Traffic Signal Network Optimization-a Paradigm Shift Driven by Big Data”. ICDE2019 Alibaba Cloud Computing Hangzhou?China
    6 Patent Medical record text structure analysis algorithm and device based on pseudo corpus generation In application Link
    6 Jens Otto and Oliver Niggemann. Automatic Parameterization of Automation. Software for Plug-and-Produce. AAAI Workshop on Algorithm Configuration, 2015
    6 Lasi, Heiner, et al. "Industry 4.0." Business & information systems engineering 6.4 (2014): 239-242
    7 M. Papageorgiou, C. Diakaki, V. Dinopoulou, A. Kotsialos, and Y.Wang, “Review of road traffic control strategies,” Proceedings of the IEEE, vol. 91, no. 12, pp. 2043–2067, 2003.
    7 Patent Algorithm and device for improving accuracy of medical record quality assurance system by using doctor behavior log In application Link
    7 Christoph Legat, Christian Seitz, Steffen Lamparter und Stefan Feldmann. Semantics to the Shop Floor: Towards Ontology Modularization and Reuse in the Automation Domain. IFAC Processings, Vol. 47, Issue 3, pp. 3444 – 3449, Doi:10.3182/20140824-6-ZA-1003.02512, 2014.
    7 Monostori, László. "Cyber-physical production systems: Roots, expectations and R&D challenges." Procedia Cirp 17 (2014): 9-13.
    8 paper P. Lowrie, “Scats, sydney co-ordinated adaptive traffic system: A traffic responsive method of controlling urban traffic,” 1990.
    8 Patent Medical record text structure analysis algorithm and device based on context-free grammar parsing technology In application Link
    8 Martin Ringsquandl, Steffen Lamparter, Sebastian Brandt, Thomas Hubauer, and Raffaello Lepratti. Semantic-Guided Feature Selection For Industrial Automation Systems. International Semantic Web Conference. Springer, Cham, 2015.
    9 paper F. Corman, A. D’Ariano, D. Pacciarelli, and M. Pranzo, “Evaluatio of green wave policy in real-time railway traffic management,” Transportation Research Part C: Emerging Technologies, vol. 17, no. 6, pp. 607–616, 2009.
    9 Patent Algorithm and device for structural analysis of medical records combined with visual features In application Link
    10 paper L. Singh, S. Tripathi, and H. Arora, “Time optimization for traffic signal control using genetic algorithm,” International Journal of Recent Trends in Engineering, vol. 2, no. 2, p. 4, 2009.
    10 Patent Method and device for Chinese medical record named entity recognition by using Iterated Dilated CNN with condition random field model based on Chinese character structure In application Link
    11 Patent Method and device for Chinese medical field relationship extraction by using residual convolution attention network model? In application Link
    12 Patent Method and device to detect similar electronic medical records In application

  • Peer-reviewed scientific/technical publications on AI applications (e.g. [1]).
  • Patent documents describing AI solutions (e.g. [2], [3]).
  • Technical reports or presentations by renowned AI experts (e.g. [4])
  • High quality company whitepapers and presentations
  • Publicly accessible sources with sufficient detail

    This list is not exhaustive. Other credible sources may be acceptable as well.

    Examples of credible sources:

    [1] B. Du Boulay. "Artificial Intelligence as an Effective Classroom Assistant". IEEE Intelligent Systems, V 31, p.76-81. 2016.

    [2] S. Hong. "Artificial intelligence audio apparatus and operation method thereof". N US 9,948,764, Available at: https://patents.google.com/patent/US20150120618A1/en. 2018.

    [3] M.R. Sumner, B.J. Newendorp and R.M. Orr. "Structured dictation using intelligent automated assistants". N US 9,865,280, 2018.

    [4] J. Hendler, S. Ellis, K. McGuire, N. Negedley, A. Weinstock, M. Klawonn and D. Burns. "WATSON@RPI, Technical Project Review".
    URL: https://www.slideshare.net/jahendler/watson-summer-review82013final. 2013