Unlike some taxonomies that have a more “academic” (read: stodgy) approach to what is considered a synonym, we put real-world equivalents in our thesaurus because we want it to work for real-world users. Many users of our health care information sites are pressed for time and are looking for an answer to a specific question. They shouldn’t have to think very hard about how to structure a query so that a search engine can understand it. It’s our job to be knowledgeable about both their language and their lingo. At Silverchair, we believe the searcher is never wrong (our version of “the customer is always right”).

Bob Wachter, with whom we’re privileged to work on two sites sponsored by the Agency for Healthcare Research and Quality (PSNet and WebM&M), recently wrote a humorous blog post about the way his hospital colleagues at UCSF (and other hospitals) commonly turn the nouns of their everyday work life into verbs as a shorthand way of communicating. For example, a resident might report that she “heparinized” her patient, or that a patient ready for discharge has been “housed and spoused,” meaning it had been determined that the patient had somewhere to go and someone to care for him. In addition, he reports the creation of new terms based on healthcare IT functionality, based, for example, on the way buttons are named in an EHR (“I done-ed it”).

That post is a fun reminder of the many ways medical lingo—and language—evolve, and the importance of attentive, systematic approaches to managing and supporting the information needs of those who invent the common parlance of their specialty in the course of doing their work (we hope, while using the sites we develop for them).

As we at Silverchair and Semedica see more and more interest in automated tagging solutions (such as our Tagmaster system), we are more frequently encountering questions about how to evaluate their results. Here are a few ideas on the subject:

Evaluation: Humans Required!

It is hard to get around the fact that you will need human editors (or professional indexers) and your human technology team (who will use the tags to create interesting new features) to verify that an automated system is working correctly and that the tagging is accurate and useful. 

Recently, someone asked our CEO Thane Kerner if we had an automated system to verify the accuracy of our automated tagging. Thane replied (rather cheekily, I must say): “If we had an automated review system that could measure tagging accuracy more precisely than the current tagging system, we wouldn’t use it to verify tags, we’d use it to tag the content to begin with!” The lesson: Once you’ve deployed your best automated system to do the tagging, humans are the next logical reviewers. 

Here are four factors your humans should consider in their review:

View inside Semedica's Tagmaster, showing tags automatically inserted at the paragraph level

1.  Expert/Editorial Accuracy Confidence

One key target for evaluation is to assess how much confidence your key stakeholders (journal boards, editors, etc.) express in the output of the system. But confidence is not a linear equation. I posit the following values:

• Impeccable tag placement: +1
• Debatable tag placement: −1
• Debatable tag omission: −1
• Obvious tag omission: −10
• Obvious irrelevant tag placement: −50

The first thing you’ll notice is the weight of positive to negative. In high-stakes fields (including science and medicine), humans are naturally biased to more heavily favor negative experiences.  (Of course, this has aided us well in survival: “Don’t eat that type of berry again, it made you sick last time!”) What that means in terms of confidence is that stakeholders will need a disproportionate amount of positive reassurance to get over negative outcomes. And the impact of a particularly egregious negative outcome (resulting from a particularly poorly placed tag) can be devastating to your stakeholder’s impression of a tagging system. (This is why Silverchair’s system defaults to using conservative methods with very little “guessing” to avoid obvious irrelevant tag placement.) 

2.  Usefulness!

The next key target for evaluation for both editorial and technical stakeholders to assess is usefulness of the tagging applied. Tags should be highly relevant in a domain-specific context and they should drive better discoverability and linking. Primary care, genetics, surgery, and emergency care all take very different approaches to the same topics, and their tagging should reflect their uses. 

The tagging system you are evaluating may have added tagged concepts that are tangential or irrelevant to the use model of the content, and such tags would not be capable of driving innovative site features (in many cases, tangential tagging actually inhibits the ability for new systems to work effectively). For example, it is a nice-to-have if your tagging system can recognize place names and person names, but if it misses or miscategorizes important topics like clinical trial names it doesn’t matter how many people or places it can tag. (Clinical trial acronyms can be particularly tricky to tag―see our post about them.)

3.  Granularity

Does the system still work with “documents” or can it identify topics down to the section/paragraph/figure/table/equation level? At Silverchair we work with many dense medical chapters that may cover more than 200 distinct topics, so we see it as a necessity for our tagging system to break those documents down into smaller parts in order to deliver precise packets of highly relevant information to our users.

4.  Control and Ongoing Improvement

Any system selected is not going to be extremely accurate “out-of-the-box.” (I write that as a realist, not as a pessimist!) So during evaluation you must ask, “How easy is it to make impactful positive changes to the system?” This can take a variety of methods—some systems suggest manually selecting training documents for each topic or category (which can get onerous when you have 20,000 topics), some systems allow your software developers to go in and tinker with the code (you have data classification expert software developers, right?!?), and some systems allow you to load and use a taxonomy or thesaurus to aid in topic identification and tagging (assumes a taxonomy/thesaurus exists or can be created for your domain).

At Silverchair, we work primarily in medicine, which is a taxonomy-rich domain with an ever-growing list of topics. For that reason, we’ve chosen the last method as our control and improvement strategy. Our editors update our Cortex medical taxonomy and its related thesaurus every day to keep pace with the topics being written about and searched for. 

Summary

If you choose a system that 1) is accurate enough to instill confidence in your editorial team, 2) is useful enough to drive meaningful new features and improvements, 3) classifies your data at a granular level, and 4) is flexible enough to allow explicit control and ongoing improvements―you’ve made a wise purchase!

Location of Jupiter in Palm Beach County, Florida (image via Wikipedia)

The names of clinical trials are often long and unwieldy, as they try to describe the focus and mission of the trial in their name—for example, a clinical trial studying drug treatment of high cholesterol is “Arterial Biology for the Investigation of the Treatment Effects of Reducing Cholesterol 6–HDL and LDL Treatment Strategies.” Because of these long names, trials are more commonly known by their acronyms—in this case, “ARBITER 6–HALTS” trial—and no doubt their full names are being crafted to result in a catchy or apropos—or hopeful—acronym. For example, the acronym for the trial studying the effect of the drug Vytorin on cholesterol levels is “IMPROVE-IT.” (See this blogpost for some humorous trial names and acronyms.)

One of my pet peeves is the incorrect use of the word “acronym” to mean any abbreviation for a term. Actually an abbreviation is also an acronym only when the abbreviation spells a word or is a combination of letters that people can pronounce as a word. So yes—abbreviations of clinical trials are acronyms, and ah, there’s the rub for commonly used full-text nonsemantic search engines. A full-text search engine treats them like any other word.

So yikes—a PubMed search for “JUPITER” (the acronym for the trial “Justification for the Use of Statins in Prevention: an Intervention Trial Evaluating Rosuvastatin”) delivers the first two results correctly, but the third result appears because the name of the institution that issued the paper is in Jupiter, Florida! OK so yes—the PubMed search box tries to help you by suggesting “Jupiter trial” (98 results) … but it also suggests “Jupiter study” (257 results). People—the JUPITER trial and the JUPITER study are exactly the same thing to any searcher wanting to know about JUPITER. The number of results should be the same for both searches. And nobody searching PubMed for JUPITER wants to know more about Jupiter, Florida. Trust me.

We can do better. At Silverchair, our Cortex taxonomy contains a list of clinical trials and the accompanying thesaurus includes their acronyms, so when our tagging and retrieval systems encounter those concepts, we’re able to separate them from their normal English language counterparts and tag them correctly.  Yet another benefit of an automated tagging system supported by a robust and up-to-date medical thesaurus. It understands medical information and the health care professionals who depend on it so that we can give them results, not guesses.

The web servers are designed to execute application logic, store session tracking data, handle user interaction input, and synthesize, parse, and display data from a variety of sources—they are logic processing engines that handle data storage only when necessary. On the other hand, the SAN has one purpose—to store a large amount of data and enable a super-efficient data delivery channel that rapidly responds to content requests from the web servers.

The more I thought about it, the more I realized it was a fitting metaphor for how humans work. We are fantastic logic processing engines. We parse, synthesize, analyze, and use data input from a variety of sources to perform creative problem solving. And most importantly to this metaphor, we only store data internally when absolutely necessary. In the present day, the comprehensiveness and ubiquity of the Internet have allowed us to store an unprecedented amount of collective memory in external sources and access it from wherever we may be.

To be clear, human use of external memory did not arrive with the Internet—it has been around since the beginning of civilization. We are used to storing memory in external sources and freeing up our internal resources. Papyrus eliminated the need to memorize long epic poems. Abaci eliminated the need to memorize multiplication tables. (NB: Don’t try telling that to a 2nd grade teacher.) In modern medicine, drug handbooks store dosage and safety information that is too complex for doctors to memorize in toto. Phone numbers stored in our mobile phones eliminate the need to memorize the phone numbers of friends. We even store memories in our friends and family—I recently asked my wife, “What was the name of that hotel we liked in Chicago?” She knew, and voila, I had accessed my external memory successfully.

Alas, my comparison of human activity to Silverchair’s web farm breaks down at a key point. In many cases, accessing our external memory is not fast and efficient. Currently the external memory sources of humans are not deployed as efficiently as a SAN. Internet content sources can be hard to access, store content in highly variable forms, require a special vocabulary or technique to query, and return data in a way that does not suit our purpose.

This is the fundamental problem that Silverchair’s Semedica division addresses with semantic enrichment of data sources. We’re organizing a specific external memory category (in our case, online medical and health care information) in a way that allows it to be accessed more quickly and to return data in the right form for efficient use by clinicians and researchers. The less data that health care workers need to store internally, the more of their “processing time” can be used toward envisioning creative solutions for preventing and curing diseases. That is something that the Internet cannot do. (Yet.)

Image via Wikipedia

The NIH has rolled out their new RePORT (Research Portfolio Online Reporting Tool) web site for information on funding, grants, and NIH research. As someone who works on government grants and contracts, I’m happy with this new level of transparency and clarity as to what topics (and who!) is being funded. It is a big upgrade from the incumbent system, which was hard to navigate and understand.

The most useful area of the site to me is the categorical spending section. It really gives you an idea of NIH’s funding priorities—it offers over 200 categories of funding.

However, it still has ample room for improvement. Currently it is an alphabetical list that contains items that are hard to compare. Here are some example categories that are not equivalent in scope:

• Allergic Rhinitis (Hay Fever)
• American Indians / Alaska Natives
• Burden of Illness
• Cancer
• Cardiovascular
• Clinical Trials
• Conditions Affecting Unborn Children
• Gene Therapy
• Gene Therapy Clinical Trials
• Genetic Testing
• Genetics

Some are very specific (hay fever), some are broad (cancer), some are ambiguous (cardiovascular), some take a completely different approach than the dominant disease/condition approach (American Indians/Alaska Natives), and some seem to be repetitive.

With a bit of work, this information could be turned from its current flat list expression into a multilevel taxonomy that allows users to slice it up in the ways that appeal to them (conditions or target populations, for example). Silverchair does this for the Agency for Healthcare Research and Quality on their PSNet patient safety clearinghouse. A small amount of classification work can go a long way in creating valuable new features—NIH has proven that with their RePORT upgrade, but I’d like to see them go farther.

I’d be happy to help out with the NIH site, but I’m not sure what category that would be funded under…