01
Three datasets, one table
The data is patient reviews from public datasets (Drugs.com and DrugLib.com among them), each with a drug, a condition, the review text and a rating from 1 to 10. Merged, that's 606,077 rows. Cleaning removed 239,262 of them, leaving 366,815 reviews of 5,711 drugs. Ramzi Masri cleaned the DrugLib portion.
I called a review “effective” if it rated the drug 7 or higher, which splits the data 60/40 (220,975 effective, 145,840 not). The ratings aren't spread evenly: they bunch at the ends, at 1 and at 10, condition after condition.
02
From reviews to features
Each review was cleaned and lemmatised, then scored for sentiment with both VADER and TextBlob (all 366,815 of them, in batches of 10,000). On top of that: the 500 most frequent words as TF-IDF features, the review's length, the year and month, the drug and condition, how many reviews each drug and condition has, and each drug's and each condition's average rating. That's 520 features in all.
The data was split 80/20, with 293,452 reviews to train on and 73,363 held out. I trained three classifiers (logistic regression, random forest, XGBoost) for effective or not, and three regressors for the exact rating, plus an LDA topic model to see what people actually write about.
03
What the models said
XGBoost won both tasks. Predicting the exact rating is much harder than effective-or-not: an R² of 0.43 means the text explains under half of the variation in ratings, which makes sense when the same words show up in five-star and one-star reviews.
The topic model found ten themes, and they're very human: side effects, sleep and anxiety; periods, cramps and bleeding for birth control; and the first days and weeks of taking something new.
| Effective or not | Accuracy | F1 | ROC-AUC |
|---|---|---|---|
| Logistic regression | 73.5% | 0.79 | 0.80 |
| Random forest | 76.1% | 0.82 | 0.83 |
| XGBoost | 78.6% | 0.83 | 0.86 |
For the exact rating, XGBoost reached an RMSE of 2.63 and R² of 0.43, against 2.77 and 0.37 for linear regression. All on the 73,363 held-out reviews, and all optimistic, for the reason in the next chapter.
04
Re-reading it: the model was handed part of the answer
Two of those 520 features are each drug's and each condition's average rating. They were computed over all 366,815 reviews before the train/test split. So every review's own rating, held-out reviews included, went into the average the model used to predict that same rating. That's target leakage.
How much it matters depends on how many reviews a drug has. For a popular drug, one rating barely moves the average. For a drug with a handful of reviews, the average is close to the answer, and for a drug with one review, it is the answer. The app's own comparison page opens on two drugs with one review each, rated 1.0 and 10.0.
So the 78.6% and 0.86 were an upper bound, not a result. The fix is simple: compute those averages from the training reviews only, then run everything again. So I did.
For a drug with one review, its average rating is the answer.
05
Running it again without the leak
I re-ran the repository's own pipeline end to end: the same 366,815 reviews, the same 80/20 split, the same models and settings. One thing changed. Each drug's and each condition's average rating now comes from the 293,452 training reviews only, and a drug the training set has never seen gets the overall training average. First I checked that the unchanged pipeline reproduces the original notebook, and it does, to within a twentieth of a point (78.63% against 78.58%).
Overall, the leak was worth about one point. XGBoost goes from 78.6% to 77.6% accuracy and from 0.86 to 0.85 ROC-AUC; random forest from 76.1% to 75.4%; logistic regression from 73.5% to 72.9%. That's because most held-out reviews are of popular drugs, where one review barely moves an average. Break the results down by how many reviews a drug has, and the picture changes. There are 911 drugs with a single review, and on them accuracy falls from 90.5% to 70.0%.
The averages are still worth having once they're honest. Dropped entirely, XGBoost reaches 76.4%; computed from training reviews only, 77.6%. A drug's track record is real signal. It just has to come from reviews the model isn't being tested on.
Overall, the leak was worth a point. For a drug with one review, it was worth twenty.
| Drug's reviews in the data | Held-out reviews | With the leak | Leak-free |
|---|---|---|---|
| 1 | 190 | 90.5% | 70.0% |
| 2–5 | 802 | 87.2% | 72.8% |
| 6–50 | 9,585 | 80.3% | 77.4% |
| 51–500 | 30,782 | 78.0% | 77.4% |
| 501+ | 32,004 | 78.5% | 77.9% |
XGBoost accuracy on the 73,363 held-out reviews. For the exact rating, R² goes from 0.43 to 0.41. Run on current library versions, with two adjustments to match the original environment: the one missing condition is kept as its own category, and absent TF-IDF entries are zeros, as they were under pandas 2.
06
What the cleaning missed
Writing this up meant looking at every figure again, and they show three more things I'd fix:
- Merging datasets merged their labelling conventions. One source writes “For Birth Control” and another “Birth Control”, so the same condition is counted twice, which helps explain how there are 7,798 “conditions”.
- Text artefacts survived into the features: “_x000d_” is an Excel line-break code, and “quot” and “amp” are HTML entities. They're common enough to sit in both word clouds, so they very likely made the TF-IDF vocabulary too.
- The app compares averages with no minimum. A drug with one review gets the same big headline numbers as one with ten thousand.