Mendeley: How NOT to run a beta preview program

One of the (many) influences Google has had on the software industry is the concept of the beta release as product. In some ways, this is a good idea, as it creates a community of early-adopters who can act as a massive beta testing community, giving the company feedback on real world use and making for an even more stable general release. The early adopters benefit from access to early technology, the public benefits from better software, and the company benefits from advance publicity and testing.

It can backfire if not done correctly, however, and Mendeley is providing a good object lesson in that. In theory, Mendeley is a killer app for people in academics. It is a cross-platform (including web), cloud-synced database for papers that handles citations and automatic import from all manner of online journals.

Unfortunately, the beta releases have been so bad that most of the word-of-mouth on Mendeley has been poor. Do a quick Google search on them and you’ll see a lot of complaining. In my experience, the software has tremendous potential but is so poorly implemented that it is currently unusable. Import of any paper with an accented letter in an author name, for example, fails. In my field, it seems half the people have umlauts in their name. Page numbers aren’t imported correctly, either, requiring the user to manually enter them. If you import a PDF for a paper already imported through other avenues, the software is happy to create duplicate entries. And so on…

After the frustration of importing their citation database from other software, only to find Mendeley too buggy to be usable, it’s likely many of the early users will not bother return for more punishment. So, what Mendeley is actually accomplishing with their beta program is the alienation of exactly the kind of people they are supposed to be winning over: technologically-minded members of their target audience. These are the people their collegues will to turn to when they are looking for citation software. Mendeley won’t be their answer.

Unfortunately for Mendeley, they may eventually have a great product, but when that final bug is fixed and they drop “beta” from the name, it may be a tree falling in a forest with nobody to hear.

39 thoughts on “Mendeley: How NOT to run a beta preview program

  1. Victor

    Hi Jonathan,

    Victor here, one of the founders of Mendeley. First of all, apologies for the trouble you encountered, and thank you for your feedback! I appreciate your point about releasing versions too early and scaring off early adopters. On the other hand, as a start-up, we have to rely on continuous user feedback for improvement, as it’s impossible for us to test the myriad of hardware configurations across all platforms we support (Win, Mac, Linux).

    Let me specifically address the issues you pointed out:

    – Author name/page number import from PDFs: This is one of the trickiest issues to get right, because most PDFs don’t contain clean metadata (as photos or MP3 files tend to do). We have to look at the full-text and try to “guess” the correct data; we also try to match the extracted data against external databases such as CrossRef, PubMed, arXiv, or Google Scholar. We are continuously trying to improve the quality of this process – just today, we released a new version (beta 0.9.6) which has improved extraction algorithms and also makes use of embedded XMP metadata if available. If it’s not too much to ask, would you mind e-mailing a couple of PDFs where extraction failed badly to support@mendeley.com, so we can figure out where the problems lie and add them to our test set?

    – Duplicates: Indeed, we have some work left to do here. Some duplicate detection mechanisms are in place, e.g. if you import a PDF twice, the records will get merged. Also, if you import a metadata file (EndNote XML, BibTex, RIS) first and the PDF second, the PDF should automatically get attached to the first record, though it appears this isn’t working well enough yet (we chose to err on the side of too many false negatives instead of false positives). Just today we were discussing additional duplicate detection mechanisms, and your feedback of course adds to the sense of urgency in this matter.

    In all fairness, I would like to point out that not everybody has had a bad experience with Mendeley – in fact, quite the opposite. You can find a number of very positive reviews and tweets here: http://www.mendeley.com/review.

    Thanks again and best wishes from London,
    Victor

    1. Jonathan Post author

      Thanks very much for writing, Victor. Actually, the problem with accented characters and page numbers occurs no matter the method of import (including directly from online journals), so I think it may be something problematic with the client itself. I’ve verified that the correct data is there in the sources, so it’s not an issue of missing or incorrect metadata.

      I’d also like to make clear that I have no problem with Mendeley not correctly parsing out things like titles when metadata is missing. I understand that that is a difficult problem and I’m actually pretty impressed that the PDF extraction works as well as it does.

      I agree not everybody has had a poor experience, but I hope you will take my criticism as constructive. I think in some areas the beta versions are too rough to really be considered beta and you do yourself a disservice there. I only bothered to complain because it would be such a shame if Mendeley were to fail; as I said, it has the potential to be a killer application. Nobody else seems to be interested in doing a cross-platform citation manager with actual desktop clients and online storage, and yet I think that’s important. So, I sincerely hope you succeed.

      I have submitted examples of URLs which don’t work to support already, and am happy to send PDFs, as well.

      Before I end this, permit me to throw out yet another piece of unsolicited advice: The rest of the market in citation software is pretty horrible. I don’t know why, but nobody can seem to figure this space out. Nothing does everything (search, import, storage, citations) even adequately, let alone well. However, it’s important for one program to do it all, since nobody wants to keep importing/exporting their reference data between a family of programs. That’s too much work, not to mention a versioning nightmare. In the end, most reference software ends up causing more work than it saves. If you get Mendeley working up to its potential, it will kill this year or next. There’s no need to rush. I’m willing to wait, and I’m sure I’m not alone. But I’m not willing to spend too many more iterations of trying Mendeley only to find out its done something crazy, like chopping off the last name of half the German authors in my database.

      1. Bill

        Zotero? Although it runs inside Firefox, it’s a fully functional local application. Looks fully Unicode compliant and localized in many languages.

      2. Victor

        Much appreciated, Jonathan – as much as it hurts to hear when things are not working, we are very grateful for constructive feedback. Also, thank you for taking the time to submit examples of URLs that fail.

        We’ll keep your comment about not wanting to try too many iterations in mind – perhaps slower release cycles (> our current 6-8 weeks) would be a solution.

    2. Frank

      Hi Victor,
      I just have come across Mendeley a couple of days ago. When I first used it I thought: PERFECT!
      Then I noticed the extremely poor (or rather absent) XMP support.
      You say metadata in PDFs are bad? Well, ask yourself: why? If I import thousands of files into Mendeley, spend hours editing the erroneously generated metadata and then Mendeley does not even bother to implement any functionality to write back that metadata as XMP into the files, how do you think the “bad metadata” in the files is ever supposed to change???
      Honestly your response strikes me as incredibly ignorant. You blame PDF providers for not adding decent metadata and Mendeley itself does not even bother writing metadata to the files AT ALL!

      Have a look at some decent image management software with full metadata support. You could learn a lesson or two.
      Sorry about the ranting, but I find your response just incredible!

      Your software has fantastic potential and it is very fairly and accurately reviewed here. You are simply not willing to learn the lesson.

      Frank

  2. Bill

    The above comment is symptomatic of Mendeley’s problems. In their desperate bid for relevancy in a crowded marketplace, they’ve consistently put spin ahead of product. Absurd promises aside, their few actual functioning features have been unceremoniously lifted from existing commercial and (worse yet) free and open-source competitors. I can’t imagine entrusting serious scholarship and research to a marketing machine that trumpets “reviews” and “endorsements” that are written by paid Mendeley employees.

  3. Peter Schmidt

    Thank you Jonathan and Bill for calling out “The Emporer has no clothes.” We in the academic community do not begrudge that Mendeley is not perfect. However what we object to is the hype before they can deliver useful functionalities. The hype sucks a lot of academics into trying Mendeley only to find after investing a significant amount of time that Mendeley is not ready for scholarly work.
    What is more worrisome is the “what is free now will continue to be free” model. Presumably Mendeley will charge for “premium” services in the future. What if they cannot sell their “premium” services? Mendeley has between 20 to 30 employees and has recently moved to new premises. One can assume that rent in London is not cheap. In the event they cannot generate revenue, how long can they survive on “what is free will continue to be free” model. As an academic, I am apprenhensive about entrusting my data to Mendeley.

    1. Mr. Gunn

      Thanks for your comments, Bill and Peter. I just wanted to clarify a few things briefly:

      Mendeley doesn’t pay for reviews and endorsements. There was a cases where someone subsequently joined Mendeley after an endorsement, but I’m hoping you won’t hold that against them.

      For premium services, They’ve discussed useful things like more storage space, more detailed access to statistics, elevated API access, and things such as that. Everything that is free will continue to remain free.

      It’s unfortunate that you didn’t find it adequate for your use, but it’s improving rapidly and I hope you’ll check back again. As the academic community liaison for Mendeley, I tend to avoid promising things that can’t be delivered, but do forgive me if I get excited about the promise and potential.

    2. David Rinker

      Peter is absolutely correct. I downloaded Mendeley in the hopes that it could deliver what it promised it could. It cannot, and I wasted a lot of time in coming to that conclusion. I’ll give it another look in a year, but it’s lack luster performance at this point does not inspire much in the way of hope.

  4. Pingback: How to REALLY not run a public beta: Dropbox | Jonathan Birge

    1. paloma

      AMEN Jonathan, we´ve been trying the same with the same results of ending with our heads toasted.. any new solution?

      1. Jonathan Post author

        I haven’t found anything that does everything. On the Mac, Papers is pretty good. What platform are you using?

  5. Juan

    I’m not sure what you guys seem to be complaining about. First of all Mendeley is free (whether they will charge for it later on is a different story, but as of the moment is free). being a linux user I’m very appreciative of a software that works well under linux and integrates with openoffice. Yes, the program has crashed on me and previous versions were pretty buggy (0.9.5 and below) but the people at Mendeley are trying to be innovative and provide a simple but powerful user interface versus the archaic and never-ending frustration with Endnote. In my own opinion Mendeley has achieved far more in its short development period than endnote has in 10+ years. By the way, I have imported many references with umlauts and all kinds of other characters without a hitch, perhaps it’s a problem with your computer. Let me guess, windoze 😉

    1. Jonathan Post author

      Thanks for writing, Juan. My main point was to talk about how doing a public beta can cut both ways. In the case of Mendeley, I think it’s hurting them more than the exposure is helping them. You’re the first person to come to their defense (who’s not from Mendeley) and yet your experience doesn’t seem to be that great, either.

      Also, they ARE going to be charging for this. Mendeley isn’t making Open Source software here.

      Finally, I’m using linux, too. My import problems were from imports from OSA journals, which I guess use an encoding that Mendeley doesn’t like (despite other programs doing just fine with it).

      1. Mr. Gunn

        Jonathan –

        Your main point about the early beta is well made. We did get a lot of reasonable complaints about earlier versions, but we’re finding that many of our early adopters who weren’t able to use Mendeley before are trying it again and liking it, so whether it’s hurting us or helping us in the long run will have to be answered over time.

        Just to finally clarify questions about the business model, it’s a freemium model that we’re going for. In other words, everything that’s currently free will remain free, all of it, and we say so publicly: http://www.mendeley.com/faq/#is-mendeley-free

        However, we plan to add some features such as more storage space (like Dropbox) or enhanced professional features. They’ll be priced very reasonably and I think they’ll represent enough value that people will pay for them.

        We appreciate your comments, of course, and please stay in touch.

  6. Pingback: Blind.Scientist » Blog Archive » I thought I was alone, but then I found another light

  7. Steve Piccolo

    I’ve been frustrated with the bugs in Mendeley but so far have found it superior to alternatives that I have tried, including Zotero. If someone wants to give me a better solution, I’m happy to try it. In the meantime, I’ll wait hopefully for future revisions of Mendeley.

  8. David Rinker

    Ok, I need to amend my above comment. I have *finally* got Mendeley working and, while it’s annoyingly imperfect (still cannot get it to import a 2000 pdf library as it always hangs up before it finishes),what it does do it seems to do pretty well and it *is* the best game for us PC folks.

    1. Jonathan Post author

      I’m glad to hear you got it working, David. Last I tried, it wasn’t correctly importing for me (on the Mac and Linux). Perhaps I should check to see if there’s been a recent update.

  9. UOLabRat

    The thing about Mendeley that really steams me is the glacial development cycle. I’ve been fiddling with it on and off for about 18 months now and they still haven’t implemented printing from the application… simple printing! One of the more active threads in the application’s forum is the plea to open up the source and get it developing in a timely manner. The developers seem to be intent on maintaining a closed source vertical model similar to what apple does. The only problem is that they don’t take the time to do any of the polish. They started out with an application that did a bunch of stuff poorly-to half way decent, rather than starting out with an application that does a few things with well implemented basic (*cough* printing *cough*)functionality then expanding.

    1. Jonathan Post author

      Amen. I kept checking back with Mendeley for awhile, but I’ve since given up. Our whole group has switched to using Zotero, in case you haven’t looked at them recently.

      1. UOLabRat

        Yeah, I’m actually looking for a close analogue of Papers (that I have on my mac) for windows. I’m really looking forward to this paperpile thing (google it, first entry) that is in beta. Looks quite promising.

  10. Mr. Gunn

    Look for a new Mendeley release in about a week. It will have some new features to make it easier to publish reading lists and collaboratively annotate papers.

    Printing is on the agenda, but instead of just adding a button that does the same thing you could do via right clicking and selecting “Open Externally”, we’re going to make the feature really useful. You’ll be able to print annotation summaries, for example, so you can get a printout of notes you’ve written on papers in a collection.

    It takes a little extra time to really do something right, but you’re welcome to get started on your own Mendeley client if you would like. Here’s an example in Python: http://github.com/Mendeley/mendeley-oapi-example

    1. UOLabRat

      Thanks very much for making my point…
      1. You would rather hold back basic functionality, until you can get around to making a really useful version. I’m all for enhancing basic functionality, but I insist that the basic functionality be there first. Modify and enhance rather than hold back for some promised but as set ethereal better function. (Did I mention that people have been asking for you guys to let them print natively for about a year now?)

      2. This open externally option is apparently hit or miss… I am currently in the camp for whom this feature isn’t functional… despite the 4 pdf viewers that I have available. Again, rolling out half polished and unstable features that aren’t reliable.

      3. How is it that you have time to crank out an API for mendeley, and can’t crank out bug fix releases in a timely manner? It actually borders on arrogance, which is also how I would characterize your last statement. How is it that when the flaws in a product are plainly pointed out rather than fixing the flaws, you take the stance that I should get lost and make my own?

  11. Mr. Gunn

    UOLabRat – Mendeley has a release cycle of about 4 weeks. That’s pretty rapid by most standards. Like you, I’m disappointed when the bugfix or feature I wanted doesn’t make it into the new release. We could be more clear about the timeline and milestones, however, and we’re working on that.

    I certainly didn’t mean to imply that you should get lost by suggesting you could work on your own client. Rather, I think you should join us in helping to make Mendeley better. The point of the API is to empower people who are dying to add something to just go and do it, without waiting for the in-house developers.

  12. Frank

    Hi Victor,
    I just have come across Mendeley a couple of days ago. When I first used it I thought: PERFECT!
    Then I noticed the extremely poor (or rather absent) XMP support.
    You say metadata in PDFs are bad? Well, ask yourself: why? If I import thousands of files into Mendeley, spend hours editing the erroneously generated metadata and then Mendeley does not even bother to implement any functionality to write back that metadata as XMP into the files, how do you think the “bad metadata” in the files is ever supposed to change???
    Honestly your response strikes me as incredibly ignorant. You blame PDF providers for not adding decent metadata and Mendeley itself does not even bother writing metadata to the files AT ALL!

    Have a look at some decent image management software with full metadata support. You could learn a lesson or two.
    Sorry about the ranting, but I find your response just incredible!

    Your software has fantastic potential and it is very fairly and accurately reviewed here. You are simply not willing to learn the lesson.

    Frank

  13. MD

    It is unbelievable to me how many nasty critics there are on this page. Must be a symptom of belonging to the academic world — a strong sense of entitlement and wish to belittle others.

    Mendeley is fantastic for my use. The citation system is flawed, and it does come down to faulty importing of information, BUT of the thousands of articles that I read only some become cited. I am more than happy at this point to use the traditional programs to cite articles that are relevant, until the feature is more useful in Mendeley. The point is, Mendeley clearly helps me to organize and identify important papers easily and that is valuable.

    Don’t like it? Take a hike. Don’t diss the company. Don’t hate on Victor. If you think you are such hot stuff and can fix the problem, join their team and do it. It’s far too easy for you to take the easy route and complain from the cheap seats. My bet is, the research you are doing probably isn’t important anyways.

    1. UOLabRat

      Its kind of funny how the person branding everyone else a “nasty critic” could, based on the none-to-suttle venom of their comments, themselves be judged a “nasty critic.” It is incredibly easy to point the finger at others from ones own ivory tower, while missing the inherent hypocrisy of their snide comments. May I suggest that you work a little harder to filter your inner monologue before committing it to the world stage?

  14. Jens

    Hej,

    first of all, I’m using mendeley since some month and made huge commercial for you guys in our department and gained at least ten users.
    BUT
    a)
    half of them jumped of again when they tried to use Mendeley in publications: Duplicates is one thing but are easily found later nd not really a huge prob. However not to be able to finely tune the format of the citation and the bibliography is a killing argument.
    I know there is this external tool for the citation style format
    but this is by far not ready yet and all my collegues exported to endnote (free from our university) and stayed again with it and i have to agree all the nice fancy things are great but if the basic tasks of a software doesn’t work…
    b)
    the searching function is great and rescues some of this point but still. i have about 5000 references in many different fields and would like to file them detailed. in jabref is the option to create subfolders. A libary build with this structure stays in order even if the numbers of articles increase. i have up to now over 30 collections and i need subfolders to keep track of what belongs where. i saw that this is on the track but it was on the track since over 2 years. any comment when this will come?

    over all i think i give you guys a chance and keep paying my fee for the larger database. thumbs up for a nice piece of software.

  15. Mr. Gunn

    Hi Jens, and thanks for your comments. Please understand that we aren’t writing the citation styles, those styles were written by members of the academic community such as yourself. Likewise, we’re not writing the editor for those styles. That’s also a open source effort – code repository here: http://bitbucket.org/csledit/csl-wysiwyg-editor/

    That said, I recognize the importance of being able to modify and customize these styles, so we’re doing what we can to help move the CSL editor project along. If you really need a style modified, you should send a message in to support, leave feedback at http://feedback.mendeley.com/, or ask nicely over at the zotero forums if someone can fix one up for you.

  16. Jeff

    I’m amazed by the negative comments directed at Mendeley. I realize Victor and Mr Gunn are representing/marketing Mendeley but they seem to be taking feedback not just providing it.

    For certain features I cannot find a better product. They are offering it for free and I’m grateful that they’ve chosen this approach because I can’t find anything better (Zotero is close, I used it until I found Mendeley).

    Developing software is not an easy or inexpensive task and its disappointing when a beta user community would apply terms like arrogant and ignorant or respond in a hostile manor when invited contribute to a solution. I can see why the term entitlement was used.

    Given the level of cynicism in this forum I should probably point out I’m in no way affiliated with Mendeley or to my knowledge anyone in these posts.

    1. Jonathan Post author

      Thanks for writing, Jeff. You have a point, and I should’ve been more diplomatic in my criticism. Civility costs nothing, and always helps. However, Mendeley isn’t a charity. They will eventually hope to make money from this, either from users directly or from advertising or from selling its user data or graph to others. In fact, Mendeley is nothing without its users, who are essentially organizing the scientific literature and giving that work to Mendeley for no charge. So, we expect a working software solution in return. You are right that we should be civil, but Mendeley isn’t doing this because they love us. Finally, if one had to place a value to Mendeley on my and Jens posts, or yours, which do you think would be of more material use to them?

      1. Stereotactic

        Exactly and unless they tone up the forums and the web support, it is useless to expect any paid users in the future 🙂

  17. Stereotactic

    I am pretty late to the party but a lot has changed in the past 1 year.

    1) Zotero has come up with a stand alone client; although they have something for Linux but dodos refuse to come up with either a debian/rpm format (or other packaging).

    2) Despite repeated requests, they refuse to give something on a ppa which would have made it terribly easy to stay updated.

    3) Mendeley comes a little close but heck, they don’t have forums!!! How bad can it get?

    4) The criticisms are very valid; support on Twitter is messy and quite pathetic.

    5) Program works; (I am using the latest “beta”) tends to crash (am on Linux Mint 9) but works kind of.

    6) Their highlighting is a boon and can be exported but beyond this basic functionality nothing else. I can’t change colors or apply different lines. No way I could submit a “feedback” or request.

    7) Their Chromium extension works but their java script is BADLY coded; it doesn’t close on it’s own but rather assumes a new window and stays like that.

    8) Nothing is perfect but software coders need to engage with end users and respond proactively.

    9) Crash logs don’t get numbered 🙁 No way to know what happened and the follow up thereafter (on Mendeley).

    10) Overall Zotero sucks for sticking on to Firefox alone, slow response to users and bad overall user experience. Mendeley for lack of access/forums and no support email. I hate filling up the damned web forms (unless they feel that their users are demented idiots). Heck if the free users get such a deal why would I pay them in the first case? Are you listening Mr. Gunn?

  18. Mr. Gunn

    Of course I’m listening! 🙂

    I’ll address the issues you’ve raised specifically.

    3) Mendeley does have forums. We use Uservoice. They’re not the style of forum you’ve come to expect, but they work essentially the same. You can access the feedback forum at: http://feedback.mendeley.com

    4) If you have a specific complaint about how we provide support on twitter, I’m happy to hear it. I’ve helped hundreds of people on Twitter and satisfied nearly all of them. See: http://amplicate.com/love/mendeley Zotero doesn’t provide twitter support: https://twitter.com/#!/stakats/statuses/27383206118428672 so you can’t blame them.

    6) The annotation features are constantly being improved, but we think the current features cover the majority of use cases. If you have a specific use case, we’d love hear it at http://feedback.mendeley.com

    7) Yes, you’re right. We should have an improved version by the end of this month. How’s that for proactive?

    9) Surely the crash log has a “created by” date?

    10) If you’d rather send in email, just mail support@mendeley.com No need to fill in web forms, although forms do help prompt people to give complete information, which helps us help you faster.

    I would encourage anyone to email me at william.gunn at mendeley or find me on Twitter as mrgunn if you have further questions, or just leave another comment here.

  19. Stereotactic

    This is brilliant. I have never seen such support EVER. I shall mail you shortly Mr. Gunn. I am sold on Mendeley.

    Software is in transition and fact that the support is willing to consider the shortcomings AND listening AND more importantly acting on it, helps considerably.

Comments are closed.