ISO/IEC JTC 1 SC 42 Artificial Intelligence - Working Group 4
Use Cases & Applications 04/24/2024
Editor's comments and enhancements are shown in green. [✓ Reviewed]
The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:
[1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
[2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
[3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
[4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
[5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
[6] Functional aspects, trustworthiness, and societal concerns
[7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.
Extracting sentiment and its intensity from customers’ input, and responding with appropriate attitude in order to improve the quality of customers’ inquiry.
To design an efficient solution for customers’ sentiment and intensity detection, especially in the situation of limited training dataset.
Short Description (up to 150 words)
The emotion-sensitive AI customer service of JD.com Int., is supported by AI technology and deep learning method. It is developed for ameliorating accuracy of customer sentiment and intensity. In sentiment classification, it has achieved 74% accuracy and 90% recall score while in intensity detection, it has accomplished 85% accuracy and 85% recall. During the special sale of -618-, it has increased customer satisfaction by 57%.
Complete Description
JD’s customer service representatives need to handle millions of requests on a daily basis. Regular AI customer service systems, 24/7 online, are capable of offering instant assistance, which alleviates the labor resources to a large extent. However, it is quite challenging, if not impossible, for those systems to interpret emotions from customer input and respond as friendly as human. Under this background, based on huge data set of customer comments and rich experience of Natural Language Processing, our system can automatically detect sentiments like happy, angry, anxious, etc. Moreover, this system can also detect the intensity of customer sentiment. Furthermore, we adapt Convolutional Neural Networks, a widely used techniques in visual computing, to interpret the semantic meaning of customer’s expression. It can improve the system’s performance for sentiment classification and intensity detection. Moreover, with the adoption of transfer learning, the system can also be applied into various types of data. To overcome the difficulty of limited training data, we also use data augmentation method such as reverse translation and data noise to increase the variability of training data. Up to now, the system has reached 90% recall and 74% accuracy rate for sentiment classification over 7 categories. The overall recall and accuracy for sentiment intensity are also around 85%?it has increased customer satisfaction by 57%.
Deep learning: a class of machine learning algorithms use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Transfer learning: we adopt multi-task learning method in this system. Jointly training different annotated data in same domain, this method improves the model performance for classification problems. Data augmentation: we apply reverse translation to firstly translation Chinese into English and then translate it backward. We also use data noise to improve the data diversity.
Standardization Opportunities Requirements
The system can be promoted to as many customer cervices companies as possible once provide with enough training data for the specific Application scenario
Challenges & Issues
Challenge: the system’s performance should be as good as the human customer server. Issues: 1) limited training data; 2) sentiment classification among seven categories.
For sentiment classification: conversation data from after-sales customer services. It’s annotated by professional annotators into 7 categories of sentiments. For sentiment intensity: Only including sentiment data with “anger” and “anxious”; it’s annotated into 3 degrees of intensity: “low, medium, high”.
Using reverse translation and noise processing to increase the size and diversity of data.
Increase the performance of model training.
2
Model Training
Based on the large training data, with deep learning method, to develop model for sentiment classification (7 categories) or sentiment intensity (3 categories).
3
Evaluation
Evaluate data performance on open dataset and specific data.
4
Execution
Apply the trained model on real-time AI customer service.
The trained model has been evaluated as deployable
Certain period of time has passed since the last training/retrainig
Improve architecture of model
AI algorithm engineers
2
Certain period of time has passed since the last training/retrainig
Collecting new data
AI algorithm engineers
3
Completing Step1&Step2
Model retraining
AI algorithm engineers
Specification of retraining data
References
No.
Type
Reference
Status
Impact of use case
Originator Organization
Link
1
IT company
XiaoIce
In operation
Microsoft Asia
Peer-reviewed scientific/technical publications on AI applications (e.g. [1]).
Patent documents describing AI solutions (e.g. [2], [3]).
Technical reports or presentations by renowned AI experts (e.g. [4])
High quality company whitepapers and presentations
Publicly accessible sources with sufficient detail
This list is not exhaustive. Other credible sources may be acceptable as well.
Examples of credible sources:
[1] B. Du Boulay. "Artificial Intelligence as an Effective Classroom Assistant". IEEE Intelligent Systems, V 31, p.76-81. 2016.
[2] S. Hong. "Artificial intelligence audio apparatus and operation method thereof". N US 9,948,764, Available at: https://patents.google.com/patent/US20150120618A1/en. 2018.
[3] M.R. Sumner, B.J. Newendorp and R.M. Orr. "Structured dictation using intelligent automated assistants". N US 9,865,280, 2018.
[4] J. Hendler, S. Ellis, K. McGuire, N. Negedley, A. Weinstock, M. Klawonn and D. Burns. "WATSON@RPI, Technical Project Review".
URL: https://www.slideshare.net/jahendler/watson-summer-review82013final. 2013