Continuing from our previous installment, we will focus on data preparation, which often proves to be the most critical and time-consuming phase of the project.
What is Data Preparation?
Data preparation is the process of cleaning, structuring, and optimizing raw data before it is fed into an AI model. A key goal here is to refine our observations to the specific amount and type of data that best suits our system’s processing capabilities.
It is important to remember that including unrelated or “noisy” data increases the number of unnecessary parameters. This leads to higher computational costs and uses more resources than necessary, often without improving the model’s accuracy.
What kind of optimization is required?
Optimization means being precise with your data. Here are the standards for different data types:
- Images: Use only the resolution necessary for the AI to identify the target features. High-resolution images are often unnecessary and slow down training.
- Video: Crop the frames to the area of interest and use an optimal resolution to reduce computational load.
- Personal Data: Use only the fields relevant to your research. Always apply pseudonymization techniques to protect patient identities.
- Facial Data: If your model requires facial recognition, use only consented datasets and strictly limit the data to what is required for the task.
Note: Collecting more data than necessary doesn’t just waste your budget, it creates unnecessary legal risks and privacy liabilities.
How do you handle the Data Cleanup process?
Data cleanup is the rigorous process of auditing your dataset for errors, outliers, bias, and repetitive data. In the medical field, this step is vital because even a small amount of “noisy” data can lead an AI to a dangerously incorrect diagnosis.
The cleanup process typically focuses on four key areas:
- Error & Outlier Detection: We identify data points that fall outside the realm of medical possibility; such as a heart rate of 900 BPM or a height of 5 centimeters, usually caused by sensor malfunctions or human entry errors.
- De-duplication: We remove repetitive records. If the same patient’s data is entered multiple times, the AI may “overfit,” meaning it memorizes specific cases rather than learning how to generalize for new patients.
- Normalization & Scaling: Medical data often comes from different hospitals using different units (e.g., Celsius vs. Fahrenheit). We must normalize these into a single standard so the AI can compare them accurately.
- Handling Missing Values: Because medical records are often incomplete, we must decide whether to discard a record or use imputation (filling in the gaps using statistical averages) to prevent significant data loss.
While the cleaning process can make a lot of data “invalid,” a strong preparation strategy early on helps minimize this loss. By the end of this phase, your data should be a clean, balanced, and reliable foundation for the next step: Labeling.
What is the Data Labeling Process?
Data labeling means indexing your information so the system can understand the exact relationship between each dataset. It is about creating a “map” that allows the AI to refer back to the source accurately and learn the “Ground Truth.”
To do this effectively, you must focus on two main areas: Structural Organization and Expert Verification.
1. Structural Organization (Tracing your Data)
You must maintain strict organization so the AI doesn’t get “lost.” For example:
- Image Slicing: If you cut a single large medical image into 100 smaller pieces, every piece should retain the original filename followed by its positional order. This ensures the system knows exactly where each piece belongs in the “big picture.”
- Video Extraction: When using video as a source, every frame must be traceable. The filenames should include the source video name and a timestamp. This allows the AI to understand the sequence, knowing which frame came first and which followed.
2. Expert Verification (The Medical "Answer Key")
In the medical field, labeling isn’t just about naming files; it’s about accuracy.
Subject Matter Experts: Unlike labeling common objects, medical data requires doctors or radiologists to provide the labels. If a “Ground Truth” label is wrong, the AI will learn a false diagnosis.
Consistency: Because doctors sometimes disagree, many projects use a “Consensus” model where multiple experts label the same data to ensure the AI learns from a verified medical agreement.
By combining strict file indexing with expert medical knowledge, you create a high-quality dataset that is ready for the training phase.
As we discussed in the previous part of this series, implementing ML in medicine requires a foundation of high-quality data. By optimizing resolutions, aggressively cleaning outliers, and maintaining expert-verified labels with perfect traceability, you move from “messy data” to “clinical-grade data.”
Preparing data is the most demanding part of the journey, but in the medical field, it is the most vital. Accurate data doesn’t just make a better model. It ensures better outcomes for the patients who will eventually rely on it.