In our previous article, we emphasized the importance of establishing clear and interpretable data standards for all organizational data assets. Once established, these standards facilitate the direct generation of data in the agreed-upon format. However, there are many situations where you have limited control over how data is created. Some reasons for this include:
- Historical data: Created long before your data standards were established.
- Partner’s data (e.g., CRO): Created using their own guidelines, which may not align with yours.
- Public data: Highly customized datasets with varying formats.
- Internal process variability: Manual data generation can lead to ignored guidelines, errors, or typos.
In such cases, data must be transformed to match the required guidelines and templates. This process is typically called curation or harmonization.
We will primarily focus on the harmonization of experimental attributes (e.g., experimental design, patient phenotypes, and sample descriptions) rather than measurement data (e.g., lab device measurements or omics data), as the latter can often be handled using computational approaches at scale.
Scenarios for harmonizing experimental descriptions may include:
- Convert free text: Map to values from a selected vocabulary or ontology. Mapping 'heart attack' to 'myocardial infarction' using a medical ontology.
- Convert numeric values: Change from one unit scale to another. Converting patient ages from years to days for a pediatric study.
- Split complex descriptions: Break down into several attributes. Separating 'blood pressure reading' into 'systolic' and 'diastolic' values.
- Extract attributes: Derive from larger text segments. Identifying and extracting 'smoking status' from patient medical histories.
- Handle missing values: Address and rectify gaps in data. Imputing missing values in a dataset using the mean or median of existing data.
- Check for anomalies: Identify and correct inconsistencies. Flagging and reviewing outliers in patient weight measurements.
Traditionally, until the recent advancements in LLM capabilities, curation processes typically fell into one of two categories: manual or automated.
Traditional Curation Process
The most straightforward curation approach involves a dedicated expert (or a team of experts) well-versed in the guidelines and the experimental data. With proper training and domain knowledge, these professionals can make informed decisions and interpretations of the original experimental descriptions to align with the required guidelines. Human curators excel in cases requiring natural language understanding, such as dealing with synonyms or typos, and are invaluable for transforming large unstructured text segments into formatted sample attribute tables.
If you have a large volume of data that already follows an established format, you can create a straightforward set of rules for automatic transformation. For example, if you receive data from a partner or CRO that follows their own guidelines (which may differ from yours or were deemed adequate when the data was produced), you need to understand their guidelines and the actual values used. Initially, this requires a manual curation process to identify and set the rules, but once established, these rules can be executed automatically. For instance:
Rule 1: Change attribute name "Organism" to "Species".
Rule 2: In "Species" attribute, convert value "Human" to "H. sapiens".
Rule 3: …
Such rules become more sophisticated when certain vocabularies / ontologies (and corresponding value IDs) or massive formatting transformations (e.g., convert age in years to age in days) are involved. But eventually, such a file could be reused as many times as you need so hopefully only a few amendments will be required.
If your data is highly customized and historically generated by different individuals and teams, the number of potential rules increases significantly. You need to familiarize yourself with all the datasets to understand their contents and identify corner cases. Manual curation is appropriate here, but the main limitation is the curators' capacity to handle large amounts of data. Typically, curation is not a full-time job but an additional task someone must complete.
The table below summarizes the traditional approaches most suitable for typical curation cases.
Table 1: Traditional approaches to typical curation cases.
| Scenario | Manual Curation | Automated Curation |
|---|---|---|
| Convert Free Text | ||
| Convert Free Text | Better When: Understanding context and handling ambiguities is crucial. Expertise in context and handling ambiguities. Large volume of custom data. | Better When: Few known changes are needed for large datasets. Fast, consistent, scalable. |
| Convert Numeric Values | ||
| Convert Numeric Values | Better When: High accuracy and error detection are critical. Cons: Prone to human error. | Better When: Large volumes of data require fast, consistent processing. Cons: struggles with outliers. |
| Split Complex Descriptions | ||
| Split Complex Descriptions | Better When: Contextual understanding and quality control are needed. Contextual understanding, quality control. Cons: Inconsistent interpretations. | Better When: Efficiency and scalability are required. Efficient, consistent, scalable. Cons: Limited flexibility, depends on training data. |
| Working with unstructured texts | ||
| Working with unstructured texts | Better when: Nuanced understanding and ambiguity resolution are important. Human curators traditionally excel in this task. | Traditional machine learning approaches demonstrate poor quality. |
| Handle Missing Values | ||
| Handle Missing Values | Better When: Values are complex, variable and only able to be determined on a case by case basis. Cons: Slow, subject to human error, not scalable | Better When: Need for efficient, consistent handling of large data volumes. |
| Check for Anomalies | ||
| Check for Anomalies | Better When: Expert judgment and contextual understanding are required. Cons: Inconsistent handling at large scale. | Better When: Rapid detection and consistent handling of anomalies in large datasets. |
Applying LLMs for Curation
The recent advancements in large language models (LLMs) technology can now replicate the manual, highly customized decision-making process of curation at scale. LLMs are a type of artificial intelligence that can generate human-like text based on the input they receive. They are trained on vast amounts of data, allowing them to understand and respond to a wide range of prompts.
LLMs generate text in response to a prompt. While LLMs are known for occasionally generating factually incorrect information (known as hallucinations), the specificity of the reply largely depends on the strength and focus of the original prompt. Another important aspect of prompts is the context size or number of words added. Despite the evolution of LLMs enabling larger context sizes, they still perform better when the task is relatively short and focused.
While an in-depth introduction to LLMs (e.g., GPT, Claude, Llama, Gemini, Mixtrall) is beyond the scope of this blogpost, we will outline the most important DOs and DON'Ts of using LLMs for curation purposes.
DON'T:
- Try to harmonize the entire dataset in one run. Posting the original sample metadata table as a piece of prompt will result in the LLM being overloaded with context, formatting, and other information not directly related to curation generation (like re-generating all the original Sample IDs intact back to the user, messing with the proper number of rows and columns, or making slight changes in numeric values).
- Feed all your ontologies to the LLM or assume it perfectly knows all public ontologies. Large custom ontologies may contain thousands of terms (including synonyms and descriptions); adding them directly to the prompt and asking the LLM to map all the terms in one prompt can cause incorrect results.
- Neglect human oversight. Even the best prompts that seem to generate reproducible results can output some incorrect interpretations in several cases. The output of an LLM must be reviewed by a person in order to either confirm the correctness of the mapping or to spot potential errors.
DO:
- Provide context. When generating a prompt for a data harmonization task, make sure you have included the main attributes, their value types, and suggestions for field interpretation.
- Harmonize one piece at a time. Instead of feeding in the entire table (including all the harmonization rules and ontology content you'd like to use), harmonize attribute by attribute.
- Prepare your data. Instead of feeding in the entire table, create a summary representation of each attribute you want to be mapped or created.
- Use appropriate tools. Imagine you need to convert the Age variable from years to days. You can make the calculations in your mind or use the old-fashioned pen-and-paper approach. The fastest way to do that, however, is to use a calculator. If you need to do that 100 times, you'd use spreadsheet formulas or some code to do it for you. These are your external tools. The same goes for LLMs; instead of asking them to generate a synonym from an ontology or make numeric transformations, connect them to external APIs or mini-programs they can call to solve different tasks. All you need is to provide instructions in the original prompt.
- For example, you can use a tool like OpenRefine to clean and standardize your data before feeding it to the LLM, or connect the LLM to a specific ontology API to ensure accurate term mapping.
- Keep your unstructured free text focused. Say you already have some sample annotation table, but you know there are more attributes mentioned and described in a related text document (e.g., experimental protocol, lab report, descriptions, or scientific article). In that case, you need to provide clear instructions of the attributes you want to find and make sure you have prepared the right section of the document. For example, if you want to extract data from scientific articles, you most likely need only the Materials and Methods section and/or Results section; keeping your context relatively low decreases the chances of the LLM misinterpreting results.
As you can see, LLMs can complete most of the tasks traditionally attributed to manual curation. However, LLMs cannot execute a curation of relatively big data tables in one run. Just like humans, they excel when the big task is split into smaller steps, focusing on one step at a time.
Conclusion
LLMs have the potential to revolutionize the curation process by automating tasks that were previously manual and time-consuming. By following the DOs and DON'Ts outlined in this post, you can effectively leverage LLMs to augment your curation workflows and achieve results at scale.
Remember to provide clear context, prepare your data, and use appropriate tools to ensure the best possible outcomes. And most importantly, always include human oversight to review and validate the LLM's output.
By incorporating LLMs into your curation process, you can harness the power of artificial intelligence to streamline your workflows, reduce manual effort, and ultimately achieve better results faster. Working with a data management system with data curation and governance solutions built into it can also ensure that your data is kept consistent and accurate, further empowering your LLMs.
Learn more about how Genestack can help to supercharge your LLMs for any research task at www.genestack.com/ai