Using NLP to Explore Unconventional Targets
Introduction and Data Issues
Introduction
Extracting information from file content and file pathways presents a challenging task. While Natural Language Processing (NLP) techniques can be beneficial in dealing with these tasks, file pathways are not the conventional target for NLP. Consequently, the data must undergo cleaning and processing steps tailored for NLP applications.
This information extraction process contributes to the precision of search and retrieval functions, facilitates information extraction, and helps to discover similar content related to the target or source.
Problems with the Data
The data presents inconsistent and complex patterns, making the use of Regular Expressions difficult. File paths lack a standardized format or structure, adding to the complexity. Additionally, there are numerous instances of uncommon and domain-specific terminology, posing recognition difficulties for most NLP models. This project aims to address these challenges and find effective ways to detect and extract important information from complex structures.
Data Processing
Getting the Data Prepared for NLP
The first stage of this project involved preparing the data for use in NLP packages. The package I used for this project is called spaCy, a conventional NLP package. Since my data was not in the conventional format it needed to be processed first beforehand. Traditional NLP operates on sentences and bodies of text, so I modified the file paths to resemble sentences more closely. I removed any characters in the file paths that natural language would typically have such as the "/" between file sections. I also added periods in between these sections along with a space so the package would treat them as separate units. These adjustments significantly improved spaCy's ability to recognize and extract information from this data.
Extracting Information
File Path Extraction
After the file paths were processed, the important information could be extracted, after it was broken down into tokens by spaCy.
These tokens allowed for search functions to be created to allow for specific token search across all files. I created a search function to return all file pathways that included a specific token along with a searchable list of all the tokens from the paths.
Term Frequency-Inverse Document Frequency (TF-IDF)
The important information was identified using Term Frequency-Inverse Document Frequency (TF-IDF). TF-IDF scoring is a statistic that represents term-importance in a document relative to a collection of documents or corpus. The formula for this scoring is shown below:
The TF portion represents the number of times term t appears in document d over the total number of terms in document d. The IDF portion represents the total number of documents in the corpus D divided by the number of documents containing the original term t. These two portions are then solved and multiplied together to get the TF-IDF score for each term.
This scoring was then used to give a rank to all the terms within the file paths. Each file path was equal to one document d and the entire directory would be the corpus D. Once the top terms were identified, they were used for Named Entity Recognition (NER) and Non-Negative Matrix Factorization (NMF) later on.
Named Entity Recognition (NER)
NER was used through the spaCy package to help identify and classify information into named entities such as people, dates, and so on. Specific tokens of interest were labeled with general NER along with custom NER for domain-specific terminology and uncommon terms from the TF-IDF scoring.
Some custom NER labels used in the file paths:
- Directory Folder
- Instrument Folders
- Data Folders
- Data, Reports, and other Analysis Files
- Image, Temporary, and other File Types
Search functions were created to search for specific entities and return all files with the target entity.
File Content Extraction
The files provided had limited content information due to encryption or other permissions that prevented the data from being viewed. I created test files to use in their place. The file content was processed and tokenized using similar methods as the file pathways.
For the processing step, the file content had new line spacing and commas if in csv format that needed to be changed to periods to be more sentence-like. The content was created into a searchable list, and search functions to find a specific token were created. These functions would return the content and identify where the content was from based on the target token.
Term Frequency-Inverse Document Frequency (TF-IDF)
TF-IDF scoring was used on the file content just like the file pathways. The content of each file was one document d, and the entire collection of file content would be the corpus D. Some stop words were added to ensure only important words were taken and not numerical values like results values (limited file content caused result values to be important thus created a need for stop words).
Named Entity Recognition (NER)
NER was used for the file content the same way as the file paths. Specific tokens were labeled, and custom NER was created for domain-specific and uncommon terminology that the base spaCy package did not recognize.
Some custom NER labels used in the file content:
- Test Names
- Test Type
- Instrument (Machine) Used
- Data Values and Measurements
- Tech Name
Non-Negative Matrix Factorization (NMF)
File Pathway Matrix
NMF was used for topic modeling as a way to identify the topics within the data and later use that for different grouping methods. The initial NMF matrix was created by making a matrix of file paths as rows by important words as columns. The important words used were the top scoring ones from the previous TF-IDF scoring metric. Limiting the columns to only the important words helps keep the dimensions reasonable. The resulting matrix for the file paths was a matrix with dimensions of 2843 x 18. This original matrix will be referred to as matrix V.
The matrix V then gets decomposed into two separate matrices of W and H. When W and H are multiplied back together, they equal the original matrix (with some rounding). With this W matrix we can gather information about the topic of each file path and later calculate the similarity to other file paths within the matrix.
The general form of NMF with an example is shown below, where V is the original matrix that gets decomposed into W and H:
Below is a more accurate form of how NMF is used this in this project, where V is a matrix of documents (file paths or file content) by important terms (from TF-IDF scoring). To find the topic matrix we need to decompose it to find W, which can be used for document similarity and clustering.
File Content Matrix
The same process was done for the file content for NMF. The original matrix V was comprised of file content as rows by important words as columns. Again, using the most important words as the columns based on previous TF-IDF scoring. The resulting matrix was a 5x5 matrix (very small due to limited data and using mostly test files). Matrix V gets decomposed into W and H, and we then use the W (file content topic matrix) for further topic and similarity analysis.
Document Similarity
Documents within the corpus (file paths or file content) were evaluated for similarity using Cosine Similarity to compare the vectors of their respective W matrices. Cosine Similarity calculates the cosine angle between vectors, providing insight into how similar or dissimilar each document is in terms of topics. A Cosine Similarity of 1 indicates perfect similarity, with the vectors pointing in the same direction. A Cosine Similarity of 0 indicates the vectors are perpendicular to each other, indicating no similarity. This scoring was used to compare similarity between all documents in file content and file paths.
Clustering Documents
K-means clustering was used on the vectors within the W matrix to organize the file paths and file content into similar clusters. The vectors in the W matrix were then assigned to the cluster whose K center was closest, determined by Euclidean distance. The algorithm updates the clusters based on the mean of vectors assigned to each cluster until it converges. The end result is the file paths and file content grouped into numbered clusters based on the original K value.
Retrieval Augmented Generation (RAG)
ChatGPT-3.5 Turbo API and RAG
OpenAI's ChatGPT API was also tested to see how it would perform taking information from this project and being able to answer questions and help extract information. The model was fed information from the project such as important terms, file paths, and file content into the first prompt of the chat model to provide information for RAG, where chat models can learn from information it has yet to be trained on. The ‘system’ content was also changed so the model knows that it is working in a specific setting.
It proved very effective at answering basic questions and extracting the relevant information from file content and file paths based on the RAG information fed to it in the first prompt. While this is an effective method, it might not be wise to feed real or sensitive data into a public chat model instead of a custom or bespoke model. API calls can also become expensive, so you need to be sure that they will provide significant business value to offset the costs.
Conclusions and Future Work
Applications and Conclusions
This project could help standardize data within its domain, establishing uniform file naming conventions that could be easily processed by NLP and other extraction methods. These conventions may draw inspiration from the Named Entity Recognition (NER) utilized in the project or adopt a similar approach.
Furthermore, the project has the capability to enhance the management and organization of large directories of files and their content. Documents could be organized based on document similarity or by clustering into similar groups. The similarity scoring mechanism could also be leveraged to recommend related documents from external sources, contributing to the development of a comprehensive knowledge base in the domain.
These extraction methods represent an important first step in automating results analysis by identifying targets containing important information for further review and later release.
Future Work
Future work could involve making all the token and entity search functions user-friendly (as using it through code is not intuitive), with the development of a UI.
Additionally, I would like to continue this work by training a custom model on domain-specific words for custom word vectorization. This approach would allow for searching for similar words based on their vector positions in vector space, allowing users to input words and find similar words based on the closest or most similar vectors.
Gathering and annotating real-world data to be able to train a custom NER model for more precise identification and extraction would also prove a useful addition to this project.
About Author
