Tuesday, 31 October 2006

When I did not invent the scissors

At the end of 1992 I started writing my second processing program. The first one was named SuperBalac and, after using it on a Mac Plus for a couple of months, I had realized that neither the hardware nor the software were the right choice for the task. My ideas for the new program were clear: the command "Reload the FID" would substitute the several (slow) routines performing inverse processing, 2D spectroscopy would be central (1D would represent a special case, just like a square a special rectangle), and there would be a pair of scissors. I had always seen truncated plots (I don't mean only NMR spectra) in books and articles (who hasn't?) and I liked the idea. I added a specialized tool into my new program. Picking it with the mouse, the user could hide one or two intermediate empty regions of a spectrum. The action was obviously reversible. At any time the user could have returned to the full spectral width.
When the program was ready, with my own surprise, those scissors were a surprise for everybody. Not only people were not used to find a pair of scissors inside an NMR software, the object in itself generated emotional reaction (love or disgust).
When I began printing and distributing fragmented spectra the reaction was natural: the receiver remained with the doubt that I had hidden or forgotten a peak.
The point here is not to like or dislike the scissors (it's a personal choice), but how persistent the CW mentality was and still is. It was difficult to realize that I had not destroyed the spectrum in itself, I had only created a special form of zoom. Not everybody could understand that the spectrum was still intact into the computer. Somebody believed that it was necessary to reprocess the FID (wrong!).
Up to this point, nothing new. I myself am extremely dumb in learning software written by others (that's why I prefer writing it by myself!). The paradox is that people easily accept a zoom tool, and realize that its action is reversible. What's different with a cutter tool? With the zoom you hide something on the left and something on the right, while with the cutter you hide something in the middle. Nothing else. So accept or refuse both of them. For coherence, if you ban the cutter, not only you have to ban the zoom, you shall also print all 1H spectra from 15 to -5 ppm.
In my experience, the more I have used the scissors, the more I couldn't do without them. The same can be said of my colleagues, even those who initially were critical about them. Without the scissors we feel the same sensation of when we have forgotten our lenses at home.
After 14 years the diffusion of a cutter tool is still restricted. While the program that introduced them, SwaN-MR, could cut an axis up to three fragments, in 2005 iNMR introduced the unlimited cutter, but no other did. 13 years to copy and improve my idea? Improved, definitely. Copied not really, because iNMR is another program of mine. I have titled a previous post "the stubborn chemist". I can add: the stubborn programmer. In our field we don't need the protection given by the copyright. Every NMR programmer is the founder of his own philosophy. We don't even understand each other's baby, how can we copy them?

Live NMR

Before starting an acquisition, you set the spectral width. That is an hard limit. What you truncate at this stage will be difficult (let's say impossible) to recover afterwards. Before printing you can further reduce the spectral width. I judge this as good practice because the zoomed-in spectrum is more readable. How can we define the limit? If the spectrum is not saved on disc, or if it is saved but the disc becomes unreadable, that limit will be another hard limit. If you keep the spectrum on your computer, it will be a soft limit. Tomorrow you can enlarge the spectral width again.
It's not my intention to bore you with trivial considerations. I still need to make people understand that the printed copy is not all that remains of the spectrum, it's just a copy, a translation. People with the CW mentality still don't understand which is the true spectrum. The true spectrum is the FID!
There is still the tendency to consider the processing as a phase in the generation of the spectrum. Like the clothes you are dressing are important but are not a part of you, processing is vital but it's not forever. The spectrum (namely the FID) can survive without processing, can go to bed naked and can dress new clothes every day, and still being the same spectrum.
Back in the old days when processing a 2D spectrum could take a whole hour, it was a necessity to send the spectrum to bed with its clothes on. Today it takes more time to change your real clothes than to reprocess an NMR spectrum. There is a page that shows how even a large FID can be read from disc, weighted, transformed, phase- and baseline- corrected, and displayed in a fraction of second. The whole process can be repeated more times per second. You can download the program from the same site and check it out. And you don't even need the last generation of computers!
While yesteryear we had to process our spectra, all we have to do now is to choose the appropriate processing. At the end of the day, instead of storing the processed spectrum, we store the processing parameters into a wardrobe. Next time we open that spectrum, the computer will automatically recover the parameters from the wardrobe and dress the FID, in a fraction of a second. But you can also choose a new dress. What's more important: you are never losing anything, but only if you have realized that the FID is the true spectrum.
If you try to store the transformed spectrum ("for not to lose my work") you are actually throwing away the most precious part, the original, and keeping the copy!
Beware that not all softwares are equal. Those in widest use, like VNMR and TopSpin, by default store the FID, the parameters and the transformed spectrum. iNMR and SwaN-MR store the FID and the parameters but not the transformed spectrum. Mestre-C stores the parameters and the processed spectrum. It's your responsibility to store the original FID. The additional weakness of Mestre-C is that you can't keep the parameters separated from the spectrum. Let's say you want to reprocess the FID with a different weighting function. You recover your original FID from your back-up storage and you can do what you want. Unfortunately you have to perform everything again, including phase correction, baseline correction, defining the integral regions, the display limits, etc... even if don't want to change them. It ain't to be this way and, fortunately, Mestre-C is an exception.
I need to digress now. At the end of 2005 I decided to adopt Mestre-C as my NMR software. I tested it during that year and started using it from 2006. (The idea was to keep the whole 2005 archive in SwaN-MR format and the whole 2006 archive in Mestre-C format). I knew that the rationale behind Mestre-C is the outrageous "undo" logic and I was suspicious. I discovered a command called "Reload" or something like that. After selecting it, the processed spectrum disppeared and the FID was displayed. I said to myself: "There must be a private storage of the FID into the Mestre-C document". After two months in which I had, with confidence, thrown away all my Bruker FIDs, I discoverd that the Reload had ceased to work. The reason is that it only worked as long as the FID file was not moved or removed (practically: only during the first session of usage). This is to say that even an expert and suspicious spectroscopist like me can make mistakes. The desperating thing is not the software itself, but the discovery the I was the first user (after 20000 earlier adopters) to complain! (Do you realize, now, why I am so boring with trivial considerations? I need to re-educate the whole world!).
To close the digression, the rationale behind Mestre-C is that you can reconstruct the FID with an inverse FT and the imaginary part with an Hilbert Transform. It's like generating the negative from the photograph: unnecessary, expensive, time-consuming, avoidable, risky, painful.
Last but not least: I hate GLP, but the original data must be preserved in its integrity.
You have seen a facet of what I call the old mentality, the "recorded music" mentality. In my "live" NMR, processing is to be repeated every time (by the computer), and I am the one who writes the score. If someday I want to add or remove an instrument, it will not be overdubbed, everything will remain authentic and live. Certainly the "recorded music" mentality is the daughter of the CW mentality ("if NMR existed before pulsed NMR, NMR can exists without the FID") that implicitly renounces to one of the advantages of FT-NMR, namely processing (you can process the same experiment in many ways, and it costs nothing). The pure CW mentality, still existing today, ignores even the possibility to inspect a spectrum on the screen (only printed paper matters). Maybe they don't even observe the printed spectrum, what matter is to store it into a file cabinet...

Monday, 30 October 2006

CW mentality

In the sixties NMR was a young and successful technique, just like today. The common observe frequency was 60 MHz, and proton the common nucleus; FFT was one of the great achievements of the decade but was not yet implemented by routine instruments. To increase the readability of the spectra it was customary to plot them always from 10 to 0 ppm. You can understand that, if all spectra were taken at the same magnetic field and plotted on the same paper with the same limits, our colleagues of the previous generation soon developed an inner mechanism for pattern recognition of NMR spectra. Remember that they were already trained by the use of IR and UV spectroscopy.
Reading an NMR spectrum didn't take so much time; the value of two integrals or the position of a singlet could be the only information extracted, and it was enough, because that information came to enrich the findings of IR and elemental analysis.
In those days the only way to interpret the spectrum was by analogy with known compounds. Today CW-NMR (continuous wave) is definitely dead. We have FT-NMR (Fourier transform).

Many things have changed:

  1. Higher magnetic fields simplify the spectra.
  2. Bidimensional spectra are the routine.
  3. It is often possible to extract all the parameters (or a good portion of them). With the word "parameters" I mean here chemical shifts and Js.
  4. If we want to use analogy, we have databases of spectra and we have prediction programs.
  5. There is no need to resort to analogy and to pattern-recognition (although they are not dangerous). Many spectra can and should be assigned and interpreted with direct methods. Direct methods often don't require any special expertize. I include in the concept:

    • a gradient cosy which can be acquired in a few minutes, using defalut parameters, and processed in a couple of seconds and a couple of mouse clicks;
    • water exchange;
    • correlating by the values of J: if only two multiplets show a J = 11 Hz they are obviously mutually coupled;
    • observation of a roof-effect or of a characteristic broadening.
      More powerful and useful methods exist, but this does not mean we can't save our time with older and simpler ones.

  6. If we continue to print spectra from 10 to 0 ppm we can't see the Js no more, because now the magnetic field is 10 times higher, the spectra width (measured in Hz) is 10 times wider and Js appear narrower (10 times, if you measure them in mm of paper).
  7. We don't rely on IR and elemental analysis. NMR has taken the place of the former and MS the place of the latter.
  8. We can compare CW-NMR to painting, and FT-NMR to music. We can further divide FT-NMR in two eras. The ancient era, still predominant, is equivalent to recorded music. Modern day FT-NMR, still extraneous to the mentality of the majority, is like live music. Technologically we are well into this new era. The average mentality is still in the old era. The mind of many chemists is, unfortunately, still in the CW era.

NMR software is what created the "live music" era.

Casual Encounters

In a previous post (Why is NMR software so important?) I explained the true reason why a chemist should adopt an NMR software. It's impossible, however, that the chemist can feel the needs explained there. What happens in reality is one of the following three cases:

1) The manager of a NMR facility discovers that the software on the spectrometer is obsolete or that people are just spending too much time in front of the machine. He calls the spectrometer manufacturer and receives a quotation for a software upgrade or for additional licenses. The price reported on the quotation is not compatible with the budget. The manager discovers a freeware alternative on the web and enforces all NMR users to intall that software. It's the beginning of a civil war...

2) The chemists needs a special program. For example he needs to measure a specific coupling constant to discerne the stereochemistry of a compound. The spectrum is overcrowded or the coupling pattern is not not simple. He discovers a program on the web that can simulate a second order coupling pattern and fit it to an experimental spectrum. Subsequently he discover that the program is actually a standard processing software with an added quantum-chemistry treatment. It's the beginning of a flirt...

3) Case 3 is what really happens in 99% of cases. A student wants to insert a plot of the spectrum in a thesis. Just because he doesn't write the thesis on the spectrometer (why??) he looks for an alternative software to generate the picture.

The funny thing is that it is often impossible to insert a spectrum into MS Office. Part of the problem derives from the fact that Word is not Quark Xpress (student never get this fact!), part derives from the quantity of lines (thousands) contained into a spectrum (compared to the few lines that make a chemical formula), part from the Microsoft vice to make their products less and less usable at each update. Acorn has dedicated a long page on the subject.

If you are using Mac OS X, that information is of little use. I have recovered some explanations and tips from my own notes.

The explanation is quite long and boring. If you are only interested into the solution read here.
Press Command + Shift + Control + 4 (four keys simultaneously). You will notice a cross-hair cursor. Select the region of interest. Paste into Office. Done. You can copy the picture from any application, not just iNMR, and paste the picture anywhere, not just Office. This trick is as old as the Macintosh, not specific to OS 10.
Now the explanation. Today's Mac OS, compared to the Classic Mac OS, has introduced many interesting new technologies, none of which was strictly necessary. This is why people were so slow in migrating from 9 to 10. Should I tell the two really outstanding differences between the two OS, I would mention Spotlight and Quartz. The latter name indicates the new graphic engine, which substitutes the old QuickDraw. As the name implies, Quartz is slower, yet more accurate, powerful, flexible. It's not the perfection, yet solves many problems of the past in an elegant way. While Apple introduced Quartz as a substitute for QuickDraw, they chose not to remove QuickDraw from the operative system. In this way programmers were nor enforced to rewrite their applications completely. Starting with the recent release of Tiger, however, the old QuickDraw is officially "deprecated". iNMR does not use QuickDraw. It is a Quartz-only application. When you paste a picture into iNMR, however, it is always accepted whether it is a Quartz or a QuickDraw picture.
I cannot explain why Microsoft Office only accepts old QuickDraw pictures and not newer Quartz pictures. Certainly this is not what I expect from an X-native application. This unfortunate incident enforces developers to choose between compatibility with OS X and compatibility with Office. I certainly prefer the former.

THE ULTIMATE TRICK
Last year I was faced with this problem: a colleague of mine had written an article using Word for Windows and he needed to insert a picture for a series of spectra. The latter were only available in SwaN-MR format, and we had a Mac with OS 8.6. The solution I found is worth trying in other cases (with iNMR, for example). I switched the resolution of the monitor to the maximum, enlarged the window size to the maximum, put everything in black & white, then grabbed a screen-shot with the well-known combination Cmd-Shift-4. The result was a very large bitmap. I saved it in PNG format, yet the format is irrelevant. You can use JPEG or BMP. We reduced the reproduction size from 3 to 4 times into Word. (Remember that normally the resolution required for printing is 4 times higher than for visualizing on the monitor). We saved our day. Remember: Word remains a word-processor; you can't use it in place of Quark XPress, even if millions of people insist in doing so. If you try, you are going to pay for its limitations.

MEMORIES OF THE PAST
The best Word version remains 5.1. Its heir, Word 98, still had its virtues. I remember having used thousands of times the command "Insert/Image from File" and then choosing an EPS file (with advanced Mac Preview) created via the Print dialog. It was a convoluted and lengthy operation, I know. The printed results were oustanding. The main problem was that the same Word file, if moved to a Windows-PC, had empty ares instead of pictures. This was the last version of Word I have used.

The Stubborn Chemist

Chemists, like anybody else, don't like to learn a new software. There is more than this, though.
There are other specific resons. What follows is an e-mail from one of the few chemists who like NMR software.

Our labs have mostly Macintosh computers, but our NMR department uses three Windows machines on which everybody except me processes their spectra, print them out and scan them into a report if it is needed, i.e. if the chemical shift reports are not sufficient in the experimental part. I don't know even one PhD student who knows, or let's say uses MestreC, which they could download for free on their (unfortunetely Windows) laptops. Although, there are a lot of universities where students don't even process their spectra themselves but just take their printout and don't even know that these are saved let alone can be processed without being an expert.So what to do?
Also one point is that if one buys a spectrometer, they (have) to buy the software as well which runs on Windows (maybe I am wrong here but for Bruker for instance I am not aware of a XWinNMR solution for Mac, perhaps better...).

The following e-mail goes even further. It's more colorful and I leave it in its original language.

Il problema non e' il costo del software, ma l'approccio che hanno i chimici e soprattutto i chimici organici con la tecnica NMR. Faccio anche il consulente e talvolta mi trovo a spiegare ai chimici della aziende e delle universita' italiane che gli spettri vanno guardati a video e non sulla carta. Non c'e' nulla da fare: il chimico STAMPA gli spettri, magari fa 10 copie a
ingrandimento successivo, ma li vuole su CARTA. A questo punto l'unica cosa che interessa al chimico (che al 95% e' di sintesi) e' il risultato. Il chimico si disinteressa della bellezza di un processing fatto bene o di una reference deconvolution che gli permetterebbe di vedere "quella J che non si vede a causa della disomogeneita' di campo". Il chimico STAMPA, spesso non salva neanche lo spettro! Semplicemente non gli interessa. Il prodotto e' venuto: questo basta. A questo aggiunga che - ad esempio - il JOC raramente richiede un HSQC o un HMBC se l'assegnamento di un 13C e' chiaro sulla base del "buon senso chimico" o "basandosi su composti simili" e dunque, come si dice "why bother"? E lo spettro su carta si fa "allo strumento" non dopo, sul proprio PC.

Detto questo, bisogna considerare anche altri fattori. Un dipartimento che compra uno strumento bruker, varian o jeol, compra di solito anche il software. Come sa il sofware moderno consente oggigiorno anche veloci sessioni in remoto. Io stesso utilizzo comodamente da casa il software sul mio strumento che d'altra parte HO GIA' PAGATO quando ho acquistato lo strumento. Perche' quindi dovrei acquistare altro software, seppure economico? I miei fondi del 60% sono di circa 2000 euro l'anno con i quali devo pagare i prodotti, il telefono, le missioni e il tempo macchina...quindi...

Is anybody willing to translate it? My time is short: I must add my own thoughts.

Sunday, 29 October 2006

Why is NMR software so important?

If you have no NMR spectrum to inspect, NMR software is not for you. That's the only exception I know. In all other cases: can you explain what your computer is for? For transforming you into a clerk? I suppose you are a scientist, that you use your computer to examine your data, to operate on them, to speculate about them...
The best thing about your computer is the sense of relaxation it can inspire: you are at your desk, seated, you can take your time... you are not distracted, your time is not limited, the ideal situation to study, to make discoveries... is it the same at the spectrometer?
NMR software is also good for your spectrometer. Let's say you have borrowed an hour of processing time from a colleague of yours. The spectrometer is acquiring a spectrum of his. The computer crashes, acquisition is interrupted... not the ideal situation. It doesn't happen anymore, but is it so necessary to run the risk? Aren't there any alternatives?
The great thing about NMR (compared to other analytic techniques) is that it is often possible to explain everything you can see into a spectrum, to assign all peaks, to translate all the spectroscopic information into a chemical information. Sometimes it happens that a spectrum is interpreted erroneously. It happened to everybody. It happened to me countless times. The chemist must do something to avoid an incident like this. My first rule is to try to explain everything of each spectrum. If, for example, a chemical shift looks unusual, probably the compound is not the one expected. If a single peak has not been assigned, it may belong to an impurity, but even this fact must be rationalized. When I can explain everything, I am confident about my conclusion. So the intermediate goal is to assign all peaks and explain it all.
The common practice to achieve this goal is to print the spectrum, maybe 10 different expansions of it, and study the spectrum on paper. This practice is wrong for many reasons:
1) You can print properly a spectrum only after you have already interpreted it.
2) If you annotate the spectrum on the screen, rather than on paper, it will be more readable for other people. The annotations will automatically appear on all future prints.
3) A good software lets you interact with your spectrum, paper does not.
4) Although most of the programs I have seen can't show a spectrum on screen with the same quality they can print, I know at least one notable exception. The pictures shown at www.inmr.net/facts.html are enough an evidence.
I understand that many of you don't like your current software. The purpose of this blog, indeed, is not convince you to use NMR software, but to show how the software currently available is not up to the task. There is work in the area. TopSpin is a great improvement over Win-NMR. iNMR is a great improvement over SwaN-MR. MestreNova, the forthcoming substitute of MestreC, still under development, claims to be a revolution. All these program appeared too late (or haven't appeared yet). Most of the software in circulation is painful. Because of the natural resistance to changes, many chemists will stick for years to their outdated, ugly, crappy programs. Eventually they'll try to convince you that theirs is the correct way to process NMR spectra...

manifesto

What I really wanted to do was to READ a review on NMR software. I have been waiting for more than a decade. During this prolonged period the things I wanted to read have increased up to the point that not only I am now able to write the review by myself, I can fill pages and pages and pages. A review is not enough: a blog better suits my needs. I will explain why NMR software is the most useful of all softwares, why nobody really cares about it, how you can use it, how you can get desperate with it, how you can write your own and why, and related issues.
I dare to say I am a spectroscopist, although there is no spectrometer in my work place and I have never owned one. Certainly I am not a chemist. I have worked with chemists for two decades. They treated me just like one of them. They used to talk about the Mitsunobu reaction and other things that never entered my mind. I don't feel like being a chemist. I am not a programmer. Therefore I am a spectroscopist.
You may have realized that I have not the proper know-how to write about NMR software. Isn't it good news? I can't enter into boring details. My reviews will be readable and enjoyable even by people who don't know what NMR is about! They can be offensive, because NMR software is offensive. It shouldn't be so, but I often got desperate with it, and other people got too. It was also my fault, I have written not just one, but two successful NMR programs... I'll talk about them too.
You can add your own reviews from this moment on...