ISO/IEC JTC 1 SC 42 Artificial Intelligence - Working Group 4
Use Cases & Applications
   03/28/2024

Editor's comments and enhancements are shown in green. [ Reviewed]

The quality of use case submissions will be evaluated for inclusion in the Working Group's Technical Report based on the application area, relevant AI technologies, credible reference sources (see References section), and the following characteristics:

  • [1] Data Focus & Learning: Use cases for AI system which utilizes Machine Learning, and those that use a fixed a priori knowledge base.
  • [2] Level of Autonomy: Use cases demonstrating several degrees (dependent, autonomous, human/critic in the loop, etc.) of AI system autonomy.
  • [3] Verifiability & Transparency: Use cases demonstrating several types and levels of verifiability and transparency, including approaches for explainable AI, accountability, etc.
  • [4] Impact: Use cases demonstrating the impact of AI systems to society, environment, etc.
  • [5] Architecture: Use cases demonstrating several architectural paradigms for AI systems (e.g., cloud, distributed AI, crowdsourcing, swarm intelligence, etc.)
  • [6] Functional aspects, trustworthiness, and societal concerns
  • [7] AI life cycle components include acquire/process/apply.
These characteristics are identified in red in the use case.

No. 50 ID: Use Case Name: AI solution to quality control of Electronic Medical Record (EMR) in real time
Application
Domain
Healthcare
Deployment
Model
Cloud services
StatusIn operation
ScopeDetecting defects in EMR by inspecting unstructured data based on Natural Language Processing(NLP) ability
Objective(s)To insure the completeness, consistency, punctuality and medical-compliance of EMR written by physicians
Short
Description
(up to
150 words)
This AI solution in ET Medical Brain Medical service support system was developed that could simultaneously detect mistakes while physicians wrote EMR (Electronic Medical Record).

Using NLP (Natural Language Processing) ability, it can process a large amount of unstructured text and judge the accuracy according to recognized medical reference.

It achieved 80% coverage of all the EMR quality control requirements issued by Chinese government, and human labour of EMR QC (Quality Control) was reduced 60%, which translated into cost savings, and enhanced physician education.

Complete Description This AI solution in ET Medical Brain Medical service support system was developed that could simultaneously detect mistakes while physicians wrote EMR (Electronic Medical Record).

Using NLP (Natural Language Processing) ability, it can process a large amount of unstructured text and judge the accuracy according to recognized medical reference.

It achieved 80% coverage of all the EMR quality control requirements issued by Chinese government, and human labour of EMR QC (Quality Control) was reduced 60%, which translated into cost savings, and enhanced physician education.

Medical records are the records of the occurrence, development and prognosis of patients' diseases, as well as the medical activities such as examination, diagnosis and treatment.

A high-quality medical record has great value at medical and legal level.

When medical records are converted from handwritten to electronic input, delayed, uncompleted writing and copying are endangering the quality of medical records.

Once the medical record data does not meet the requirements, it will greatly affect the health of patients, the development of medicine and the judgment of responsibility in medical accidents.

Nowadays, hospital has a Medical Records Department to control medical records quality manually. However, as the number of medical records increases, the inspection requirements become more complex, and the medical professional knowledge requirements are improved, so the medical records quality inspection becomes harder.

The intelligent electronic medical record quality control system is based on NLP. When a doctor writes medical records, it can analyze unstructured medical record text, and control the quality based on government requirements, ensure the integrity, consistency, timeliness and compliance of medical records.

ET (Evolutionary Technology) Medical Brain Medical service support system has learning ability to learn more medical knowledge including clinical pathway, drug compatibility taboo etc. it can learn the habits and rules of doctor’s manual review to inspects records profoundly.

The current system has covered 189 medical records quality inspection requirements, saved 60% review time for medical record department, which greatly saved the cost of the hospital, reduced the inspection time and repeated work, and will help doctors put more energy into the education and training.

StakeholdersDoctor, Hospital, Patient
Stakeholders'
Assets, Values
Systems'
Threats &
Vulnerabilities
New privacy threats, new security threats
Performance
Indicators (KPIs)
Seq. No. Name Description Reference to mentioned
use case objectives
1 Coverage Ratio of EMR QC requirements done in the solution/all issued EMR QC requirements in China. Ideal target is 100%. Improve accuracy
AI Features Task(s)Natural language processing
Method(s)SimHash
HardwareECS
TopologyCloud Service
Terms &
Concepts Used
Jaccard index
Standardization
Opportunities
Requirements
Challenges
& Issues
Challenges: Achieve all EMR QC requirements in different disease areas

Issues: 1) Lack of medical reference data 2) Lack of medical knowledge graph

Societal Concerns Description Achieved 80% coverage of all the EMR quality control requirements issued by Chinese government, and human labour of EMR QC (Quality Control) was reduced 60%, which translated into cost savings, and enhanced physician education.
SDGs to
be achieved
Good health and well-being for people
Data Characteristics
Description EMR text data
Source EMR system
Type Text data from EMR system vendor
Volume (size)
Velocity Real time
Variety Multiple datasets
Variability
(rate of change)
Static
Quality High (depending on EMR system)
Scenario Conditions
No. Scenario
Name
Scenario
Description
Triggering Event Pre-condition Post-Condition
1 Training Train a model (deep neural network) with training samples
2 Evaluation Evaluate whether the trained model can be deployed
3 Execution Detect defects (regions including defects) using the trained model The trained model has been evaluated as deployable
4 Retraining Retrain a model with training samples
Trainng Scenario Name:  
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement
1 Training Train a model (deep neural network) with training samples Sample raw dataset is ready
2 Evaluation Evaluate whether the trained model can be deployed Completion of training/retraining
3 Execution Detect defects (regions including defects) using the trained model Completion of deployment in EMR system The trained model has been evaluated as deployable
4 Retraining Retrain a model with training samples Certain period of time has passed since the last training/retrainig
Specification of training data
Scenario Name Evaluation
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement
1 Completion of training/retraining Preparation AI solution provider Transform sample raw data from EMR system to server on cloud
2 Completion of Step 1 Detection AI solution provider Given the image data from Step 1, detect defects (regions including defects) using the deep neural network trained in the scenario of training
3 Completion of Step 2 Evaluation Manufacturer Compare the result of Step 2 with that of human inspection
Input of Evaluation
Output of Evaluation
Scenario Name Execution
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement






Input of Execution
Output of Execution
Scenario Name Retraining
Step No. Event Name of
Process/Activity
Primary
Actor
Description of
Process/Activity
Requirement






Specification of retraining data
References
No. Type Reference Status Impact of
use case
Originator
Organization
Link
1 Link
2 Patent A medical symptom knowledge base classification system construction algorithm and device based on lexical cluster similarity In application Link
3 Patent Electronic medical record named entity recognition method and device combining Section feature information In application Link
4 Patent Algorithm and device for recognizing nested medical named entities based on two-layer recurrent neural network In application Link
5 Patent Algorithm and device for unsupervised keyword-based medical image report key information extraction In application Link
6 Patent Medical record text structure analysis algorithm and device based on pseudo corpus generation In application Link
7 Patent Algorithm and device for improving accuracy of medical record quality assurance system by using doctor behavior log In application Link
8 Patent Medical record text structure analysis algorithm and device based on context-free grammar parsing technology In application Link
9 Patent Algorithm and device for structural analysis of medical records combined with visual features In application Link
10 Patent Method and device for Chinese medical record named entity recognition by using Iterated Dilated CNN with condition random field model based on Chinese character structure In application Link
11 Patent Method and device for Chinese medical field relationship extraction by using residual convolution attention network model? In application Link
12 Patent Method and device to detect similar electronic medical records In application

  • Peer-reviewed scientific/technical publications on AI applications (e.g. [1]).
  • Patent documents describing AI solutions (e.g. [2], [3]).
  • Technical reports or presentations by renowned AI experts (e.g. [4])
  • High quality company whitepapers and presentations
  • Publicly accessible sources with sufficient detail

    This list is not exhaustive. Other credible sources may be acceptable as well.

    Examples of credible sources:

    [1] B. Du Boulay. "Artificial Intelligence as an Effective Classroom Assistant". IEEE Intelligent Systems, V 31, p.76-81. 2016.

    [2] S. Hong. "Artificial intelligence audio apparatus and operation method thereof". N US 9,948,764, Available at: https://patents.google.com/patent/US20150120618A1/en. 2018.

    [3] M.R. Sumner, B.J. Newendorp and R.M. Orr. "Structured dictation using intelligent automated assistants". N US 9,865,280, 2018.

    [4] J. Hendler, S. Ellis, K. McGuire, N. Negedley, A. Weinstock, M. Klawonn and D. Burns. "WATSON@RPI, Technical Project Review".
    URL: https://www.slideshare.net/jahendler/watson-summer-review82013final. 2013