Tuesday, 18 November 2008

Metadata

It's not easy to obtain the permission to show a nice spectrum. For this article I have downloaded a collection of spectra of a standard compound from the Biological Magnetic Resonance Data Bank. They are not very nice but can serve my purpose. I have also processed the spectra to highlight the relevant information.
Now I'll show you what is visible after you close the processing application. All the snapshot are taken while working with the operative system. I have uploaded very large pictures, so you don't lose too much (you'll only lose the animation effects). The blog shows them at a reduced size, but if you click a picture, the full-size original appears.
As a starting point, I have selected a file, without opening it; the command "Get Info" shows this panel:
This is only moderately useful, because selecting a file is already a time-consuming operation (you have to navigate through the folder hierarchy). The good news is that the computer can find the spectra for you, if you specify the same properties. This is the job of the command "Find". For example, we can ask the list of all the files relating to substances of general formula C5H7NO3:
Yes, the computer understand a chemical formula! Such a search is not restricted to NMR documents. We can however add more conditions to restrict the search, how many conditions we like. For example:
Every time I type a single character the list of results is updated (live). There is also the option of displaying an animated list. It's a cinematic effect that's very characteristic. If you have ever seen iTunes or the iPhone you know what I mean. On my blog I am limiting myself to displaying static pictures, however. Here is how my spectra look like:
And this is only the appetizer! [continues...]

Monday, 17 November 2008

What's New

For more than two years I have been describing application software ("the programs"), and not the operative systems. Sounds obvious? It isn't. I had noticed, in the past, the attempt of a couple of programs (namely the ACD suite and the "NMRnotebook") to substitute the operative system. What they do is to put all your NMR data into a monolithic archive. From that moment on you can't search your files into the normal directory tree: the application program manages everything, substituting for the system. Can they do it better?
I have also seen the opposite phenomenon, that is the appearance of new operative systems that do every kind of things, up to the point to make you feeling that application programs are no more necessary. My preference goes to the latter trend, for a practical reason. When you are in troubles with an operative system, you can find a neighbor or relative that knows it well and that can help you. If you are at work, you can call the IT department. If your troubles arise from the application software, instead, you don't know whom you can turn to. That's why I prefer doing more things than possible with the operative system, if I am allowed.
While I will be describing a specific OS, I am writing for everybody: it's the idea that counts, not the implementation.
Now, in my life I have studied only two OS. The first one was called "IBM-DOS" and the second one "MS-DOS". It happened in an era when the computers included heavy and bulky printed manuals. I have been commuting for half of my life, so I had plenty of opportunities to read paper manuals (over and over again). I am not using those OS any more and I have forgotten what I had learned. I have come in touch with the modern systems, but still feel like a stranger (I haven't found the new manuals yet!). Last week I have begun working with Mac OS 10.5.5. It resembles the old version 7 that I was using in the early 90s, so it looks familiar to me. I have tried to read the installation manual, which shows a few things you can do but doesn't tell how to do them. It only explains how to install the system. I am learning by practice.
The rationale is that customers don't know what to do during the endless installation process, so they are given a manual to spend their time with. Unfortunately, you can read as slowly as you possibly can, but the installation will always be slower than you. To give you an idea, the whole manual, including the unreadable license agreement, weights 94 g.
What has this to do with NMR? Well, this OS seems to be almost ready to handle NMR data out of the box. I want to show you the technologies that I have explored so far, and they are enough to write posts for a week. Mainly I will describe two commands: "Get File Info" and "Find...", that are so ancient I believe they have always been there, with the same keyboard shortcuts (maybe under a different menu). What's new? Now you can install custom plug-ins for those commands. The operative system will instantly become NMR-savvy. Writing a plug-in is not terribly complicated. The Apple site gives you all the tools, libraries, instructions, examples and templates you need. You can write a plug-in in an afternoon. If you have more time, with the same tools you can even write a complete NMR program. No need to look around for the FFT algorithm, it's already a component of the OS. Of course, it has been put there to manipulate sounds, not to process NMR spectra. If you can't write a line of code and want everything ready and free... then you are lucky! What I will be showing during this week can be done with freely available plug-ins. You don't need to know how to write them.
Let me clarify my intent. It's not that I am supporting Apple. I think it's a greedy corporation, no better than Microsoft or IBM. Their products are over-priced in the U.S. and sold at an outrageous price elsewhere. I would rather encourage you to stick to your old hardware and software. (Beware that programs written for System 7 are not compatible with 10.5). Nonetheless, it's nice to be curious about the new technologies. What I have discovered last week sounded new to me, probably is well known to you, surely will become normal practice in the near future. Enough said, for today [continues...].

Sunday, 26 October 2008

ProSpect

A rare event: I have found an NMR program that's been a pleasure to inspect. It comes with an HTML manual (with pictures) and sample data. These two ingredients remove the usual obstacle for learning (you rarely guess right how to arrive to a frequency domain spectrum). In this case, instead, I could easily open one of the included sample spectra, because they come with their own processing scripts, and experiment with the buttons (relying on the help of the yellow tips). It's clear that they spent their time and money in making this program. It's well done.
The program is called ProSpect ND and comes from Utrecth. It's completely free. In theory, it's cross-platform. I have tested the Windows version.
Why isn't it more famous? Well, to start with the name, it is focused on multi-dimensional spectroscopy and favors serious processing and flexibility over simplicity of use. You realize this fact from the very beginning. To open a spectrum (Bruker or Varian, no other format is recognized) you must first convert it into the ProSpect format with a script; it's not trivial.
The program also includes all the routines to process and analyze a 1D spectrum. The graphical layout is, well, arguable. For example, the peak-picking labels tend to overlap with each other.
Speaking of 1D, LAOCOON is also included. Forgive my personal reminiscence: the manual doesn't refer to the first version, as everybody else has always been doing, but to the the last version by Cassidei and Sciacovelli, who happen to be the ones who teached me how to use a spectrometer (and some theory too, in the case of Sciacovelli).
My first impression: they have written a program the way they like it. If you need to print your 1D spectra, I bet you'll have diverging opinions. If you like to spend much time in 2D processing to obtain the optimal results, and the other free alternatives don't satisfy you, chances are that ProSpect ND will.
This product confirms the rule: it takes time and money to make NMR software. If you want it free, you must find a sponsor with large shoulders. There's an important difference that puts ProSpect apart from more famous free programs like the original Mestre-C, NMRPipe, SpinWorks, NPK. Prospect is open source. In the case of NPK I may be wrong. Anyway, you can directly download all the source code of ProSpectND without asking any permission, without registering, without logging in. Guess the language? Phyton? Noo! Java? Noo! C++? Noo! OK, here's an hint: the first date on many source files says 1997.
C? Yes, I like it!

Friday, 24 October 2008

Students

They say that students buy Macs. Is it true? If it is so, and you are a student, be sure to check this promotion out. It's a full-fledged, up-to-date application, offered at a price that's certainly affordable for any student. It's possible to test the product before buying it, of course.
...and you can even leave your comment here below!

Friday, 10 October 2008

End of the World

Il mercato ormai di questo ritmo sta scontando la fine del mondo, altro che recessione! Se per caso avete perso del denaro in questi giorni, potete consolarvi in fondo era solo carta straccia. Presto torneremo al baratto, torneremo ad essere agricoltori, due galline, una capretta per il latte, ci potremo sedere intorno ad un focolare e stringerci con i nostri familiari e le persone più care, il pianeta non sarà più inquinato, sarà un luogo più vivibile,niente stress, niente lavoro forzato, il capo che rompe le palle, niente più calcio e calciatori fighette, niente più auto, Briatore e cocainomani del billionaire, niente più grande fratello e isola dei coglioni...che spettacolo forse stiamo andando verso un mondo migliore e allora forza S&P 500 ancora 900 punti verso lo zero! Hahahaha!
taken from Ferro Azzurro
Will all the NMR programs resist to this crisis?

Thursday, 2 October 2008

Whitening

In 2007 Carlos proposed to me a joint project to invent a new algorithm for 2D automatic phase correction. There were two reasons why I replied negatively. First thing first, I am doing research with no grants and no salary. Although there was the certainty of making some money in case the algorithm had been invented, I was doubtful that I could invent this thing any time soon. The second reason was that I have fun correcting the phase manually and I didn't want to spoil my fun. Instead of admitting that any normal person would benefit from an automatic procedure, I am still tempted to teach manual phase correction to everybody, guessing that they can have the same fun that I have. Carlos accepted my answers, at least the first reason. The story seemed to end there, but it was not forever.
This summer I wrote to Carlos that I really couldn't find anything interesting to do during my holidays and that it was better to do some work. He said he had an idea to keep me busy. Guess what? Inventing the automatic 2D phase correction. This time I agreed. I proposed a three-steps roadmap. The first step consisted into trying to correct the phase of a single isolated peak, adjusting simultaneously the 4 parameters involved. The second step was a semi-automatic method in which the user was required to select any number of peaks, even of different sign, and the computer had to do the rest. The preliminary method invented at the first stage would have been the main ingredient. The third stage was the completely automated method.
I liked this roadmap because the first stage looked feasible (I hate wasting time on hopeless research). The other stages were challenging, but, in my heart, I was hoping to find an escape route before arriving there. I was glad enough to have something to do for the first week.
Carlos asked me if I wanted any 2D spectrum to test my ideas. He had already mentally created a list of different cases with different characteristics. An important case was the edited HSQC, containing both positive and negative peaks (and more noise; and no diagonal). It looked more difficult then the other cases. I said: "I have never acquired this kind of spectrum; send me a specimen". When the file arrived I was already working on my project, but not making any progress because my code was infested by bugs (I was out of shape). I suspended my activity to inspect the edited HSQC. Processing it was a relaxing activity, which I did on autopilot. When I saw the 2D plot in front of me, I said to myself: "What have you done? You have corrected the phase without paying the least attention to which peaks are positive and which are negative. Actually you haven't even paid the least attention to where the peaks are located, but the phase is perfectly corrected! What does it mean?"
There was an explanation. Manual phase correction is ALWAYS so easy, at least the coarse correction is easy. I know that if I am moving the slider into the good direction, peaks become narrower and take less space into the map. Because my background is usually white, I know that if the white regions are growing, that is a sign that I am going into the right direction. I stop when the white regions start shrinking again. I immediately realized that my mental process could be easily translated into a computer algorithm. I also noted that the correction along the X axis is independent from the correction along the Y axis. The four-dimensional problem could be factorized into two bi-dimensional problems, easier and faster. I suspended the original project and began to write the new algorithm. In 24 hours I verified that it worked and worked very well. I also debugged the old code (belonging to the original project) and discovered that it was quite unstable: it only worked under ideal conditions.
I was forced to abort the first project, but very happy of having invented something really useful and really new (in a single stage). The weakness of the aborted project was that it had to observe a limited number of points at each time. The whitening method, instead, looks at one million points simultaneously. It's a statistic measure and it's reliable just because it condenses one million of observations.
Today the whitening method is a component of 3 commercial products. It has been already presented at a meeting and (in the form of poster) at a conference. We have written an article and submitted it to Magnetic Resonance in Chemistry. You may be wondering what the hell we could have written into that article, if six words are enough to state the method: "maximize the number of white pixels". Wait and see. I am convinced that the article will be accepted just because the principle is clear (have you noticed the pun?). If they have published (in many cases of 1-D automatic correction) the articles that describe algorithms that nobody has never seen in practice, why shouldn't they publish ours? Our products can be freely downloaded and you too can test the algorithm!
The natural question is: if this method is so obvious and simple, why nobody else had invented it before? The answer can be found inside Carlos' blog. While I have been performing REAL 2D phase correction for years, there are people that even ignore that such a thing is possible. What they are doing is the correction of selected rows and columns (traces) or their sum. Actually it's a mono-dimensional correction, but they believe it's a 2D correction. A pertaining example can be found on Glenn's Bruker-oriented blog. I have had the privilege of performing true 2-D manual correction and it helped me.
Finally, I must confess: I am having a lot of fun with automatic phase correction and after I have invented it I don't want to go back!

Wednesday, 1 October 2008

Commuting

The canonical sequence is:
Rance -> FT -> shuffle -> FT -> shuffle -> FT
This is the result:
The following sequence also arrives there:
FT -> Rance -> shuffle -> FT -> shuffle -> FT
which proves the commutability between FT and Rance. When I try this:
FT -> shuffle -> FT -> Rance -> shuffle -> FT
the result is: