The role of scientific conferences in R&D

In this post I'm dealing with a very important question from the perspective of a person managing or financing R&D, how does one know how well is R&D performing? If your thought was that you'll measure it by economic success of a product that uses the results of R&D then you are on a wrong track. Namely, the product can be success or a failure because of a number of reasons, of which R&D is only one. So, another way has to be used, and actually this question is very hard. In this post I'll try to point you to a possible solution along with some of its negative sides. Before continuing, just to reiterate that this post is from the perspective of a person managing or financing R&D.

The best possible solution would be that you absolutely trust all your researchers and that they produce only the best results. But this is idealistic case, namely there are no perfect researchers, and even the best ones could produce mediocre results if they are under sufficiently high pressure. So, some form of quality assurance is necessary.

The next best solution would be for you to check what every researcher did and evaluate it by yourself, after all, whom do you trust more than yourself? But this approach also has problems, and not the small ones:

1. When good researchers does something, the only way to track him would be to do the same things he does, and that means doing his job. 
2. Even if you would know so much to be able to analyze how someone does his or her job, that wouldn't scale.
3. Finally, people tend to hate micromanagement, and this would be micromanagement.

So, this approach also wouldn't work. Another approach would be to assign for each researcher another person that would check his work. But this has almost the same problems as if you are doing everything by yourself. Especially problematic could be potential collusion between researchers, i.e. one praises other's work knowing that his own work will be reviewed, too. So, reviewers might have incentive to praise each other's work.

Thus, it is necessary to have review, but the point of the review is to be independent, done by an expert that knows the topic being reviewed and trying to be as objective as possible. You can pay independent researchers for doing review, but that's not done. What's done instead is sending papers to scientific conferences and journals where they are reviewed before being published. The review process is such that the authors don't know who reviewed their paper (blind review) or even reviewers don't know who's paper they are reviewing (double blind review). Before being published in a journal or on a conference, papers have to pass review process and authors are notified about the decision along with receiving reviewers' comments.

So, there is a way you can receive feedback about the work done by your researchers by sending them to conferences or requiring them to publish in journals. But there are additional benefits as well:

1. Even if your researches have the best intention of producing top class results, it is good to have a feedback. In the reviews there could be suggestions on how to improve the work.
2. By participating on conferences your researchers build their professional network from people doing the same or similar things and that might be very helpful on the long run.
3. You should not forget marketing aspects of scientific publications. Namely, this makes you and your people known as an organization that does research and supports their researchers which might attract new researchers and employees.

Many companies having serious R&D do publish on scientific conferences and in journals and they put on their Web pages lists of published works, here are some:

• Hitachi R&D publications
• Google Research Publications
• Microsoft Research Publications
• Facebook Research Publications
• IBM Research Publications (Haifa Lab)
• HP Enterprise Research Publications
• Sony Research Publications (Paris Laboratory)

There are many others, and I might add more to the list later.

One very important thing before I continue. People tend to think that I say that publications are mean and a goal and thus are opposing to the idea of publishing on a scientific conferences. But that's not true. Publications are only a side-product of a work who's goal is to produce something new that could be used to improve company's products!

But, nothing is perfect and so this approach has some issues you have to be aware of:

1. There are a huge number of conferences in the world many of which are at best average. You should strive to go to the best ones because there you'll receive the best feedback and also meet people that are more likely to be researching things that interest you. Which conferences are those depends on the specific research area and you have to search for them, but as a general rule of thumb the lower acceptance ratio, the better conference.
2. As I've said, the papers are only a side-product of the actual work done. But, if too great emphasize is put on conference/journal publication, then researchers start to optimize that criteria instead of doing a good work.
3. You should be careful what you publish in the papers. The moment its published, effectively it's a public knowledge. This is very good from the society perspective, but it might not be so good from the perspective of a company.
4. Publication on the conference is not so cheap. You have to pay conference fee, travel and accommodation expenses, and maybe few more things. This builds up very quickly.
5. Publication in a journal might cost nothing, but it can take time, up to 18 months. The review process for conferences is several months at most.

But in any case, I think that companies should publish as much as possible on a good conferences or in good journals as it has more benefits than drawbacks.

Anonymous paper reviews and threat of a legal action

I just stumbled on a news story in which scientist claims that his career was severely damaged by anonymous comments on some of his works published on PubPeer. This is very interesting story to follow for several reasons.

For a start, PubPeer is a site for a post publication review. I strongly support such a practice because I believe that everything has to be scrutinized and tested, and it helps authors who can get the best possible feedback, but also helps society in general, too because there is ever increasing problem with scientific ethic. As a side note, I was, and I'm still a big proponent of doing review process in public. That, in my opinion, significantly increases transparency. Anyway, PubPeer fulfils my wishes, but unfortunately for me, it is only concerned with papers from medicine, chemistry and related fields, not from computer science.

In this particular case, the problem is that the author was offered a job on the University of Mississippi, with quite a large annual salary, and for that purpose he quitted his current job. University then revoked the offer and so he lost both the new job, and his current job. Now, he claims that the reason for this are some anonymous negative comments on PubPeer and threatens with a lawsuit asking for identities of those who made those negative claims.

While, as I said, it is very good to have such a site, it doesn't mean that everything should be allowed, more specifically:

1. Any claims made have to be justified. Unfortunately, anonymity also allows people to make damaging or unjustified claims by being certain that there will be no repercussions.
2. Unfortunately, negative claim even if not justified casts doubts, so that might be a problem.
3. In this particular case it is also unknown why the author didn't respond to presented claims about problems in his paper. PubPeer claims they invite first and last author to comment on comments.
4. Finally, no one should take lightly claims about some paper being invalid, not good, etc. In this particular case, I hope that University of Mississippi verified negative claims and that they didn't take lightly what some anonymous commenters said.

In any case, we'll see what will happen with this particular case.