Lit review methods at the abstract stage

With the volume of medical literature that's published, even a relatively narrow search may get thousands of hits in PubMed.  A lot of people have published on the topic of rating full text articles - see this, for example - but when you're doing a large-scale lit review, it's often necessary to do some paring down at the abstract stage. 

What I usually do is work through my pile of abstracts (easiest for me to do this on hard copy) and make clear notes on each one about whether or not it fits my search criteria. This is usually not that hard.

For example, say I'm doing a lit review of cost-effectiveness of treatments for multiple sclerosis. Up front, I have a list of reasons for excluding articles altogether:

• Not an economic analysis
• Review/opinion article with no meta-analysis
• Fewer than 30 subjects per study arm
• Trial not conducted in a relevant country (often I'm looking specifically for US studies)
• Study concerns a treatment that is no longer in use in the US
• etc.

Once I have narrowed it down to just the abstracts for articles that might contain something relevant, I usually pull the full texts of all of them. Even if a particular article itself isn't useful, I still need to go through each bibliography to identify articles that I may have missed in my searches of lit databases.

Mistakes Were Made

Database analyses are the hardest part of our work, in my opinion.  I've mostly managed analyses of large government databases so far - like Medicare claims data and data from the Hospital Cost and Utilization Project (HCUP). The difficulty is really that it's hard to know when you get the wrong answer.  If you get something that looks right, you have to pretty much trust it once you've checked all the SAS code and any formulas used to get results into Excel or wherever the results are being displayed. 

It's different from clinical trial data analysis because in a clinical trial, you always have a source document to go back to.  The source data in a large government database are often so numerous that it's really hard to go back and verify that the analysis is correct.  I guess it's similar to modeling in some respects, but models can often be recreated in another language to verify the results.  Databases could conceivably be analyzed in multiple programs (like SAS and STATA, for example), but that would probably add 10-25% to the timeline and budget.  In my small company, at least, we don't have the luxury.

At least the error I caught today was detected before we had already published the results!

Evaluating a Job Offer

I've been at my current job for 3 years now, and while I'm generally satisfied, I have occasionally looked around for other opportunities. Earlier this year, I received a very generous offer from a competitor. Of course, as everyone knows, job satisfaction is about more than money. Here's my little system I use to help me in the decision making process. It mainly consists of a cost/benefit matrix. Here's how part of my matrix looked:

Pros of staying at current job	Cons of staying at current job
Comfortable/known quantity Nice office Decent commute (bike-able) Get closure on long-standing projects More time in one place on my resume ...	Less money Less support for school/conferences/professional development Same title for the foreseeable future Stagnation? ...
Pros of going to competitor	Cons of going to competitor
More money Better support for conferences/prof dev/classes Better title now and more opportunity for advancement later New challenges and more opportunity for learning ...	Longer commute/not bike-able Longer hours Fewer publication opportunities ...

As you can see, this matrix is different for every situation, but it helps to brainstorm and list everything, no matter how minor. This system allows me to really think through the decisional balance process!