Looking for words in a haystack

The assignment was clear: The need to search for dozens of word combinations in thousands of documents and emails. And all with a limited budget.

Initially we went with fashion and tried out ChatGPT – after all, what else would you think of with all the AI buzzwords flying around the place? However, we soon found out that this solution was like catching sardines with shark nets, so we chose a more conservative and, most importantly, more efficient way.

“In these kinds of projects, you have to predict what type of output will be most useful for the client. Sometimes what you have to look for is not the trendiest technology, but a tried-and-tested solution. In this case, we have a table that’s like a hub with outputs.”

František Kopřiva, Project Engagement Manager,Revolt BI

We received over 12,000 documents and 2,000 email strings, with the most recent email at the top and all the previous ones listed below. Our task was to find the keywords in such a way as to reduce as much as possible the number of documents that had to be read manually. For these purposes, we chose, perhaps surprisingly, a straightforward solution – upload all the text to the database and enter a query for it using SQL, specifically BigQuery, over Google Cloud Storage. 

Our path was not without complications. Some PDF files weren’t machine-readable because they were scans (image only). These first had to be converted into plain text using optical character recognition. Documents over a thousand pages in length also had to be divided into smaller parts. 

For each word string searched, there were a number of variations, e.g. for a three-word phrase, each word of which had two additional synonyms, giving twenty-seven possible combinations. In addition, we weren’t able to estimate in advance how many occurrences we would find, i.e. whether we would be able to reduce sufficiently the number of documents to be read. So, we prepared two levels for each word string – a general level, where we searched for the occurrence of at least one of the synonyms, and a detailed level, where we searched for an exact match. 

At the general level, in some cases we detected a “misleading footprint”, where none of the synonyms appeared in the documents at all. Conversely, in other cases there were too many matches (typically hundreds). This is where the detailed level, when there were fewer matches and the context could be better estimated, came in. 

Based on this solution, the final output was a simple table in which the number of matches found and a list of specific documents to be examined were given for each variant of the word chain. Thanks to the conversion of all documents into plain text, it was possible to find the exact position of the word in a specific document with a classic full-text search. 

Our solution saved the client hundreds of hours of labour without a certain result.