Which journals offer the best peer-reviewer experience?

Mar 29, 2020 /

I’ve heard people say that scientists should be paid to review papers. I think this is a terrible idea. Journals need to attract field experts that care most about advancement of knowledge, not charlatans out to make an easy fifty bucks. But I do think the journals should make a significant effort to make the process intellectually rewarding and hassle-free. There are two things that distinguish journals in this regard:

Transparency of reviews and editor decisions. If I spend four hours reviewing a paper, I want to compare my conclusions and recommendations with those of the other reviewers. I also want to see the associate editor’s recommendations, and the editor’s letter. This way I can learn. The best journals actually require that reviewers read each others' reports and then provide a “cross review.”
Ease and simplicity. Have the administrators ensured that pages are numbered, word-limits respected etc? Has the editor made sure the subject is relevant and writing quality acceptable? Can I just write my review into a simple text box, or is there an annoying form to fill out?

Here are my ratings (journals I’ve reviewed for recently only):

Science (10 / 10) The only journal I’ve reviewed for that has cross-review. The process is clean, simple, and rewarding.

Hydrology and Earth System Sciences (9 / 10) Fully transparent. All reviews are published immediately online for the world to see (also great for authors, who get to see reviewer reports as soon as they land!). One little niggle is that the system requires you to write your review into a text box that is then compiled to pdf. Special characters often get converted to gibberish and you almost always have to go back and fix.

Journal of Water Resources Planning and Management (8 / 10) Nice and simple. As soon as you submit your report you get to look at the other reviewer comments. All of the editors I’ve reviewed for are friendly and flexible. One slightly annoying thing is that it’s really hard to find the editorial portal through the journal website.

Science of the Total Environment (8 / 10) As above. Nice and simple. Friendly editors.

Environmental Research Letters (7 / 10) Clean and simple process. Reviewer reports made available after editor’s decision letter goes out.

Nature Climate Change (6 / 10) Standard process; reviewers are copied into the editor’s decision letter. But there isn’t a formal user portal where past reviews are stored.

Environmental Modelling and Software (6 / 10) I had trouble accessing the EVISE portal to submit my reports. Get to see other reviewer comments only after editors letter is sent out.

Advances in Water Resources (5 / 10) Same shitty system and process as Environmental Modelling and Software. I’m scoring AWR one less than EMS because an editor was unresponsive when I requested some detail on why a particular decision was made.

Journal of Hydrology (5 / 10) Good that I can check other reviewer comments as they are posted. But there’s an incredibly annoying template to fill out in the “comments to editor box”. It has something like 15 questions to be answered with “Low”, “Medium”, or “High”. Dear editor: just read my report.

Water Resources Research (3 / 10) If the paper is rejected or accepted then you don’t get to see the other reviews or the editor’s letter (just an email indicating the decision). I’ll continue to accept review requests for WRR as long as the quality of papers remains high. But I feel they should do more to make the process more rewarding for reviewers.

Stochastic Environmental Research and Risk Assessment (1 / 10) Reviewed once for this journal. The paper was complete garbage, but the editor published it anyway on the back of a one sentence review from a friend of the submitting author. Will not review here again.