With the volume of medical literature that's published, even a relatively narrow search may get thousands of hits in PubMed. A lot of people have published on the topic of rating full text articles - see
this, for example - but when you're doing a large-scale lit review, it's often necessary to do some paring down at the abstract stage.
What I usually do is work through my pile of abstracts (easiest for me to do this on hard copy) and make clear notes on each one about whether or not it fits my search criteria. This is usually not that hard.
For example, say I'm doing a lit review of cost-effectiveness of treatments for multiple sclerosis. Up front, I have a list of reasons for excluding articles altogether:
- Not an economic analysis
- Review/opinion article with no meta-analysis
- Fewer than 30 subjects per study arm
- Trial not conducted in a relevant country (often I'm looking specifically for US studies)
- Study concerns a treatment that is no longer in use in the US
- etc.
Once I have narrowed it down to just the abstracts for articles that might contain something relevant, I usually pull the full texts of all of them. Even if a particular article itself isn't useful, I still need to go through each bibliography to identify articles that I may have missed in my searches of lit databases.
