AI Prompts for Machine Learning Code

Ever wonder why an AI can write a model that trains to 99% accuracy and still be completely broken? If you've done any machine learning, you've probably felt this suspicion. The code runs, the metrics look incredible, and something in your gut says it's too good. Usually it is — and usually the cause is data leakage the AI introduced without either of you noticing. This case study walks through exactly that, and shows the AI prompts for machine learning code that prevent it.

The Problem a Data Scientist Faced

Priya, a data scientist at a healthcare analytics company, needed a model to predict patient readmission risk. She had a clean dataset, a deadline, and a plan to move fast with AI help. She asked the AI to build a full pipeline: preprocessing, feature scaling, train/test split, and a gradient-boosted model.

The first version trained beautifully. Validation accuracy came back at 96%. For a readmission model, that's suspiciously high — good readmission models struggle to break 75%. Priya's experience told her something was wrong, but the code looked correct at a glance. Finding the flaw took real digging.

The Wrong Approach

Here's the prompt that caused the trouble:

Build a full ML pipeline in Python for predicting patient
readmission. Include preprocessing, scaling, train/test
split, and a gradient boosting model with accuracy metrics.

Reasonable-looking. But the generated code scaled the features before splitting into train and test sets. That's data leakage: the scaler learned the mean and variance from the entire dataset, including the test data, so the model got a peek at information it shouldn't have. The inflated accuracy was an illusion that would collapse the moment the model saw truly new patients.

⚠️ Common mistake: Letting AI generate a pipeline where preprocessing happens before the train/test split. Scaling, imputation, and encoding must be fit on training data only, then applied to test data. Fit-then-split leaks information and inflates every metric you care about. This is the single most common flaw in AI-generated ML pipelines.

Priya spent half a day confused by the great numbers before spotting the ordering problem — and only then because she knew what a realistic score looked like.

The Correct Prompt

She rewrote the request to make correctness the explicit priority:

Build a scikit-learn pipeline for patient readmission
prediction. Critical requirements:
- Split into train/test FIRST, before any preprocessing
- Fit the scaler and any imputers on TRAINING data only,
  then transform test data
- Use a Pipeline object so preprocessing is inside
  cross-validation and can't leak
- Report accuracy, precision, recall, and ROC-AUC
  (accuracy alone is misleading on imbalanced data)
- Add a comment explaining how this prevents data leakage
Data: [paste column names and dtypes]

What this does: It orders the steps correctly, forces preprocessing inside a

Pipeline

The rewrite produced a model reporting a realistic 73% ROC-AUC. Lower number, but a real one — a model that would actually work on new patients instead of one that memorized the test set.

⚡ Pro tip: Always ask for a scikit-learn

Pipeline

Results and What Changed

The honest version of the model was less impressive on paper and far more valuable in practice. When the company later validated it on a fresh cohort of patients, the ROC-AUC held near 71% — barely down from the reported 73%, which is exactly what you want to see. The first model, had it shipped, would have cratered on real patients and destroyed trust in the whole project.

Beyond the single fix, Priya's team adopted a rule: every AI-generated ML pipeline gets a leakage review before the metrics are trusted. And they standardized the "split first, fit on train only, use a Pipeline" language in their prompts so the leakage was prevented rather than caught.

⚡ Pro tip: When a model's metrics look too good, suspect leakage before you celebrate. Common sources beyond scaling: target leakage (a feature that's a proxy for the label), temporal leakage (using future data to predict the past), and duplicate rows spanning train and test. Ask the AI to audit for all four explicitly.

How to Apply This to Your Situation

The specific bug was scaling order, but the lesson generalizes to any ML work with AI.

A computer vision engineer should prompt for the train/test split before computing normalization statistics, and before any augmentation that could bleed test images into training.

An NLP practitioner should watch for vocabulary leakage — building a tokenizer or TF-IDF vectorizer on the full corpus instead of training data only. Same disease, different symptom.

A time-series analyst faces the trickiest version: standard random splits leak the future into the past. Prompt explicitly for time-based splits where the test set always comes chronologically after the training set.

Here's a reusable ML correctness prompt:

Before writing the pipeline, list the top 3 data leakage
risks for this problem type. Then write code that prevents
each one, with a comment marking where each safeguard is.
Split before preprocessing; fit transformers on train only.
Report metrics appropriate for [balanced/imbalanced] data.

What this does: The "list the risks first" step turns the model into a leakage auditor before it writes a line, surfacing the specific traps for your problem type instead of assuming the defaults are safe.

⚡ Pro tip: For imbalanced datasets, explicitly tell the AI the class ratio and ask for appropriate handling — stratified splits, class weights, or resampling — plus metrics beyond accuracy. A 95%-accurate model on 95%-negative data has learned nothing, and only precision, recall, and AUC reveal that.

Next Steps

Take your next ML task and, before generating any pipeline, ask the AI to list the leakage risks for that problem. Read them, then require the safeguards in your prompt. That one habit prevents the most expensive class of ML bugs — the ones that look like success.

The data scientists who trust their AI-generated pipelines aren't lucky; they've encoded correctness into reusable prompts. PromptABCD is where those live — save your leakage-audit template, your Pipeline-first template, and your metrics-for-imbalanced-data template, and pull them up on every project. Priya keeps hers tagged by problem type, and says the leakage-audit prompt has caught issues on models she was sure were clean. A realistic 73% you can trust beats a fantasy 96% that falls apart in production, every time.

The deeper habit worth building is skepticism proportional to surprise. When a model's metrics land in the range you expected, a normal review is fine. When they're dramatically better than the problem should allow, treat that as an alarm, not a triumph. Great ML practitioners have an internal sense of what's achievable for a given problem — readmission prediction tops out around 75%, credit default around the low 80s, and so on — and they use that sense to catch leakage the moment the numbers violate it. You can borrow this by asking the AI directly: "what's a realistic performance ceiling for this problem, and does my result exceed it suspiciously?" A model that reports 96% on a problem that caps at 75% is telling you something is broken, if you're listening.

⚡ Pro tip: Ask the AI to hold out a truly untouched final test set that's used exactly once, at the very end. Any dataset you look at repeatedly during development slowly leaks into your decisions, even without a code bug. A single-use holdout is the closest thing to an honest estimate of real-world performance you'll get. The temptation to peek at that holdout is real, especially near a deadline when a slightly better number would feel reassuring. Resist it. The entire value of a single-use test set evaporates the moment you use it to make a decision and then keep tuning. Treat it like a sealed envelope you open exactly once, and the number inside will actually mean something when you report it to stakeholders who are betting real decisions on your model.

Stakeholders remember the models that failed loudly after launch far longer than they remember a modest number reported honestly up front. Protecting the integrity of that final test set is really about protecting your own credibility, because the alternative — a great number that collapses in production — is the fastest way to lose a team's trust in machine learning altogether. The good news is that all of this discipline can live in your saved prompts rather than your willpower. When the leakage audit, the realistic-ceiling check, and the single-use holdout are baked into the templates you reuse, correctness stops depending on remembering to be careful and starts happening by default, which is exactly where you want it on a Friday deadline. Correctness by default is the only kind that survives a deadline intact, and saved templates are how you get it.