I usually skip around an AI paper on the first pass. Abstract, main result, method, then back to the parts I didn’t understand. Reading straight through is easier once I know what the authors are trying to establish.
I look at the comparison before getting attached to the claim. Did both methods get the same data and compute budget? Did the new system take more attempts? Was the baseline tuned? An improvement may be real and still be irrelevant to the constraints of something I’m building.
The examples deserve a separate look. A few impressive outputs are useful for understanding the idea, but I want to see the failures and the distribution of results. An average can hide exactly the behaviour I care about.
My notes need to answer three questions: what changed, what stayed the same, and what would I need to reproduce it? That keeps me from confusing an interesting architecture with evidence that I should adopt it.
Sometimes I can turn the idea into a small experiment. Sometimes the data, weights or compute aren’t available, and the honest outcome is a note about an unresolved claim. I enjoy reading the research either way. It doesn’t all have to become a product feature.