One of the most common currencies for scientists is publications. There are, of course, many other measures like patents, grants or other funding, Investigational New Drug (IND), (text)books, code repository forks/downloads, and many more. In general though, every measure of productivity breaks down into some condensed summarization of a ton of work, crafted for consumption by others.
Across almost all fields, not just science, people use resumes or curriculum vitae (CV, which is just an extra long resume basically) in order to quickly and clearly communicate career productivity. Of course, resumes and CVs are imperfect, as they will never fully capture the nuance of someone’s life and experience, but that also highlights how important written communication is. There’s a whole rant here about scientific productivity being measured most commonly by written communication, while never really getting trained well in written communication as STEM majors, which is in part what launched my whole aspirational goal this month of writing 50k words. (Aside, to be clear, 50k words is not happening, as it’s 7PM on November 30 here and I have at the moment just over 15k words, so I don’t think it’s possible to crank out another 35k words in five hours. I’ll probably write tomorrow a conclusions or summary post that’ll continue from this one as a reflective retrospective of how I think this inaugural #NovemberWritingChallenge went. Spoiler: I’m of course disappointed that I didn’t even come close to 50k, but also I’ve learned a lot about what holds me back from writing.)
I’ve written previously about the COMMUNICATION aspect of publications and other avenues for dissemination of scientific work, but since the last chapter on MESS I’ve been thinking about it from the standpoint of repeatability and reproducibility in scientific literature. First, to clarify, repeatability typically refers to the same person repeating the same experiment with the same system and getting the same result, while reproducibility refers to a different person attempting the same experiment with the same or similar system and getting the same conclusion. Anything published should really be repeated, because if you can’t get the same result you’re claiming in your publication, then you probably shouldn’t publish it. But reproducibility can be harder. Borrowing from statistical descriptions of missingness, in my mind, there’s irreproducibility “at random” and irreproducibility “not at random”. The former is where biology is just hard and there’s some hidden aspect of the experimental system that is unknown, and this is where the scientist is not really at fault for irreproducibility. Irreproducibility “not at random” is where the scientist just did a terrible job of describing the methods, the system, or the analysis. I’m assuming laziness here and not straight malicious lack of detail, although there are of course examples of malicious intent by manipulated data or straight fake analyses.
Irreproducible “not at random” is at least in part bad methods, speaking to the specific section of a scientific manuscript. Methods sections are the second easiest place for me to start writing a paper, aside from the results or figures, because I’m just describing what I did. Usually my methods are pretty generic and widely used in the field so they don’t need much detail, but sometimes there’s specific twists that I’ve added. It’s not unlike cooking and having recipes. Most people have some idea of what goes into a chocolate chip cookie recipe, but some people might have a specific twist based on their personal taste preferences, like using brown butter instead of regular, or based on necessary accommodations, like adjustments to account for baking at high elevations. So the equivalent is that maybe the scientist of the irreproducible work is just not realizing that their method works great for them because their experiment is being done in Denver, but a scientist at sea-level in Boston needs a different recipe, i.e. protocol or method.
Not to get back into the whole artificial intelligence (AI) debate, but maybe AI would be helpful for reproducibility of analyses. I’d be shocked if a lot of the papers coming out now aren’t using analyses that were written, at least in part, by AI like ChatGPT, Claude, etc. If people are already relying on AI to write their data analyses (and therefore guide their conclusions), then it’s not a huge leap to use the same AI to take things one step further and capture the whole “chat” and publish that as a supplemental method. At the bare minimum, people should be capturing the code and publishing those scripts or notebooks alongside their papers for repeatability, but I know a lot of people put terrible code out there that can’t be rerun by anybody else due to hardcoded paths or missing dependencies, and many more people just never even make their figure-producing code available.
Maybe AI could go one step further though? If we capture the protocols, alongside the data processing, and the result-producing post-processing analyses, that should be the most ideal reproducibility scenario. I don’t know exactly what that might look like in practice, but that’s something that would massively help me in implementing other people’s methods in my own lab. Upload someone else’s paper, and have the AI generate a shopping list based on the methods so that I can get all the supplies I need, and then spit out the protocols based again on the methods and a bit of the results, maybe. There’s probably also something like that out there, if only for cooking.
Anyway, all of this reproducibility ramble got me away from the initial thought, which was productivity as measured by writing. In management-speak, what goes into the resume or CV are the OKRs (Objectives and Key Results) and what helps get you to the OKR is the KPIs (Key Performance Indicators). Setting goals, or even “resolutions” as we’re now coming up on the end of 2025, is usually at the OKR level, but without some KPI to help get you there, you’re probably never going to make your goal or resolution. When you set out to run a marathon, you don’t usually just walk up to the start line and then bang out 26 miles. Usually you decide that you want to run the marathon (OKR) and then break it down in a training plan with gradually increasing mileage each week (KPI). As another example, this whole writing challenge this month to write 50k words (OKR) came with some clear daily mini-goals like shooting for 2k words/day (KPI).
One place I think people struggle, both with methods for papers and just in general productivity measurement, is figuring out whether the KPI is really what the audience needs to know, or if the audience really just cares about the OKR. For the methods section, you really need to be specific and detailed, but for reproducibility, it’s enough to be shooting for the OKR – in fact, it’s probably even better for the scientific community and furthering human knowledge if we can reproduce the idea or conclusion by orthogonal means, rather than directly reproducing the exact experimental conditions which might be correlated but not causative to the conclusion being drawn by the original scientist.
Similarly with personal productivity, it’s easy to punch out a bunch of KPIs and make progress in ticking off to-do list boxes, but if you’re not keeping the OKR in mind, you may be doing a bunch of busy work without making meaningful progress on the real goal. “Not everything that can be counted counts, and not everything that counts can be counted”, as William Bruce Cameron is quoted as saying. A few weeks ago, I was talking about this with some other women at a professional event, where we were discussing the challenges of accommodating alternative paths. Specifically, the conversation turned to the subject of childcare stipends for conferences, and a couple young mothers emphatically supported the idea. To be clear, I love the concept and stipends to help people afford traveling to professional events and career advancement opportunities. That said, a couple other women, myself included, cautioned that bringing children to the conference may prevent you from getting the full value of the conference, which isn’t usually in the formal programming but rather the informal networking that happens after the official programming ends and tends to happen in the evenings. For me this is a personal observation, as when I’ve tried to bring my child to conferences, it just ends up with me doing a pretty terrible job both professionally and personally, and nobody got my full attention because I couldn’t engage fully in either setting. While, yes, I hit the KPI of “attend conference, give talk” and “conduct bedtime routine”, I didn’t really make progress on the OKRs of “advance career” or “be a present, available parent”. That said, I also recognize that I’m lucky to have the option of leaving my child with my partner when I go to conferences now, so that while I miss spending time with my family when I travel to events, it’s a choice that not everyone has the luxury of making.
I certainly wish there was a better system, both speaking specifically of conference networking and also more broadly of productivity measurements, but until scientific society at large changes, I think we’re stuck with the aforementioned productivity metrics and figuring out tools to manage it or otherwise cope with it.