ISO/IEC JTC 1 SC 42 Artificial Intelligence - Working Group 4
Use Cases & Applications
   06/20/2019
Home

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
No. 42 ID: Use Case Name: Emotion-sensitive AI Customer Service
Application
Domain
Retail
Deployment
Model
On-premise systems
StatusIn operation
ScopeExtracting sentiment and its intensity from customersí input, and responding with appropriate attitude in order to improve the quality of customersí inquiry.
Objective(s)To design an efficient solution for customersí sentiment and intensity detection, especially in the situation of limited training dataset.
Short
Description
(up to
150 words)
The emotion-sensitive AI customer service of JD.com Int., is supported by AI technology and deep learning method. It is developed for ameliorating accuracy of customer sentiment and intensity. In sentiment classification, it has achieved 74% accuracy and 90% recall score while in intensity detection, it has accomplished 85% accuracy and 85% recall. During the special sale of -618-, it has increased customer satisfaction by 57%.
Complete Description JDís customer service representatives need to handle millions of requests on a daily basis. Regular AI customer service systems, 24/7 online, are capable of offering instant assistance, which alleviates the labor resources to a large extent. However, it is quite challenging, if not impossible, for those systems to interpret emotions from customer input and respond as friendly as human.
Under this background, based on huge data set of customer comments and rich experience of Natural Language Processing, our system can automatically detect sentiments like happy, angry, anxious, etc. Moreover, this system can also detect the intensity of customer sentiment. Furthermore, we adapt Convolutional Neural Networks, a widely used techniques in visual computing, to interpret the semantic meaning of customerís expression. It can improve the systemís performance for sentiment classification and intensity detection. Moreover, with the adoption of transfer learning, the system can also be applied into various types of data. To overcome the difficulty of limited training data, we also use data augmentation method such as reverse translation and data noise to increase the variability of training data.
Up to now, the system has reached 90% recall and 74% accuracy rate for sentiment classification over 7 categories. The overall recall and accuracy for sentiment intensity are also around 85%?it has increased customer satisfaction by 57%.
StakeholdersCustomers targeted for the Customer Service system
Stakeholders'
Assets,Values
Systems'
Threats &
Vulnerabilities
The low degree of humanization, and lack of semantic diversity for response. Reducing the number of human customer service.
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Customer Satisfaction The ratio of customer satisfaction when using this system for requests. The expectation is 100% Increasing its ratio as high as possible
2 Accuracy Among all the predicted customer sentiment classification, the ratio of accurate prediction, current value is 76.4% Increasing to 90%
3 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 90% Increasing to 90%
4 Accuracy Among all the predicted customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
5 Recall Among all the customer sentiment intensity, the ratio of accurate prediction, current overall value is 85% Increasing to 90%
AI Features Task(s)Natural language processing
Method(s)Deep learning, transfer learning, data augmentation
Hardware
Topology
Terms &
Concepts Used
Deep learning: a class of machine learning algorithms use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation.
Transfer learning: we adopt multi-task learning method in this system. Jointly training different annotated data in same domain, this method improves the model performance for classification problems.
Data augmentation: we apply reverse translation to firstly translation Chinese into English and then translate it backward. We also use data noise to improve the data diversity.
Standardization
Opportunities
Requirements
Challenges
& Issues
Challenge: the systemís performance should be as good as the human customer server.
Issues: 1) limited training data; 2) sentiment classification among seven categories.
Societal Concerns Description Improving the corresponding efficiency of customer service, improving customer service experience: Reducing labor costs, and reducing operating costs.
SDGs to
be achieved
Industry, Innovation, and Infrastructure
Data Characteristics
Description For sentiment classification: conversation data from after-sales customer services. Itís annotated by professional annotators into 7 categories of sentiments.
For sentiment intensity: Only including sentiment data with ďangerĒ and ďanxiousĒ; itís annotated into 3 degrees of intensity: ďlow, medium, highĒ.
Source Conversation data from JD.com real-time customer services
Type Text
Volume (size) Around 60,000 sentences for sentiment classification and 20,000 for sentiment intensity.
Velocity Batch Processing
Variety Real-time data from JD.com, including various categories of products.
Variability
(rate of change)
Static
Quality High
Scenario Conditions
No. Scenario
Name
Scenario
Description
Triggering Event Pre-condition Post-Condition
1 Data Augmentation Using reverse translation and noise processing to increase the size and diversity of data. Increase the performance of model training.
2 Model Training Based on the large training data, with deep learning method, to develop model for sentiment classification (7 categories) or sentiment intensity (3 categories).
3 Evaluation Evaluate data performance on open dataset and specific data.
4 Execution Apply the trained model on real-time AI customer service. The trained model has been evaluated as deployable
Scenario Name Training
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement
1 Complete data augmentation Design model for training AI algorithm engineers Using CNN for sentiment classification and intensity.
2 Complete model designing Transfer learning AI algorithm engineers Multi-task learning with different data in same domain.
Specification of training data
Scenario Name Evaluation
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement
1 Complete model training Evaluation on open dataset AI algorithm engineers Evaluate different modelsí performance on open dataset
2 Complete model training Evaluation on own dataset AI algorithm engineers Evaluate different modelsí performance on own dataset
Input of Evaluation Independent testing data
Output of Evaluation
Scenario Name Execution
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement
1 Finish model training Application AI engineers Application
2 Given customerís input Data processing AI algorithm engineers Data processing
3 Finish data processing Model prediction AI algorithm engineers Model prediction
4 Completion of Step3 Making response AI algorithm engineers Making response
Input of Execution
Output of Execution
Scenario Name Retraining
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement
1 Certain period of time has passed since the last training/retrainig Improve architecture of model AI algorithm engineers
2 Certain period of time has passed since the last training/retrainig Collecting new data AI algorithm engineers
3 Completing Step1&Step2 Model retraining AI algorithm engineers
Specification of retraining data
References
No. Type Reference Status Impact of
use case
Originator
Organization
Link
1 IT company XiaoIce In operation Microsoft Asia

  • Peer-reviewed scientific/technical publications on AI applications (e.g. [1]).
  • Patent documents describing AI solutions (e.g. [2], [3]).
  • Technical reports or presentations by renowned AI experts (e.g. [4])
  • High quality company whitepapers and presentations
  • Publicly accessible sources with sufficient detail

    This list is not exhaustive. Other credible sources may be acceptable as well.

    Examples of credible sources:

    [1] B. Du Boulay. "Artificial Intelligence as an Effective Classroom Assistant". IEEE Intelligent Systems, V 31, p.76-81. 2016.

    [2] S. Hong. "Artificial intelligence audio apparatus and operation method thereof". N US 9,948,764, Available at: https://patents.google.com/patent/US20150120618A1/en. 2018.

    [3] M.R. Sumner, B.J. Newendorp and R.M. Orr. "Structured dictation using intelligent automated assistants". N US 9,865,280, 2018.

    [4] J. Hendler, S. Ellis, K. McGuire, N. Negedley, A. Weinstock, M. Klawonn and D. Burns. "WATSON@RPI, Technical Project Review".
    URL: https://www.slideshare.net/jahendler/watson-summer-review82013final. 2013