Il mercato ormai di questo ritmo sta scontando la fine del mondo, altro che recessione! Se per caso avete perso del denaro in questi giorni, potete consolarvi in fondo era solo carta straccia. Presto torneremo al baratto, torneremo ad essere agricoltori, due galline, una capretta per il latte, ci potremo sedere intorno ad un focolare e stringerci con i nostri familiari e le persone più care, il pianeta non sarà più inquinato, sarà un luogo più vivibile,niente stress, niente lavoro forzato, il capo che rompe le palle, niente più calcio e calciatori fighette, niente più auto, Briatore e cocainomani del billionaire, niente più grande fratello e isola dei coglioni...che spettacolo forse stiamo andando verso un mondo migliore e allora forza S&P 500 ancora 900 punti verso lo zero! Hahahaha!
taken from Ferro Azzurro
Will all the NMR programs resist to this crisis?
Friday, 10 October 2008
Thursday, 2 October 2008
Whitening
In 2007 Carlos proposed to me a joint project to invent a new algorithm for 2D automatic phase correction. There were two reasons why I replied negatively. First thing first, I am doing research with no grants and no salary. Although there was the certainty of making some money in case the algorithm had been invented, I was doubtful that I could invent this thing any time soon. The second reason was that I have fun correcting the phase manually and I didn't want to spoil my fun. Instead of admitting that any normal person would benefit from an automatic procedure, I am still tempted to teach manual phase correction to everybody, guessing that they can have the same fun that I have. Carlos accepted my answers, at least the first reason. The story seemed to end there, but it was not forever.
This summer I wrote to Carlos that I really couldn't find anything interesting to do during my holidays and that it was better to do some work. He said he had an idea to keep me busy. Guess what? Inventing the automatic 2D phase correction. This time I agreed. I proposed a three-steps roadmap. The first step consisted into trying to correct the phase of a single isolated peak, adjusting simultaneously the 4 parameters involved. The second step was a semi-automatic method in which the user was required to select any number of peaks, even of different sign, and the computer had to do the rest. The preliminary method invented at the first stage would have been the main ingredient. The third stage was the completely automated method.
I liked this roadmap because the first stage looked feasible (I hate wasting time on hopeless research). The other stages were challenging, but, in my heart, I was hoping to find an escape route before arriving there. I was glad enough to have something to do for the first week.
Carlos asked me if I wanted any 2D spectrum to test my ideas. He had already mentally created a list of different cases with different characteristics. An important case was the edited HSQC, containing both positive and negative peaks (and more noise; and no diagonal). It looked more difficult then the other cases. I said: "I have never acquired this kind of spectrum; send me a specimen". When the file arrived I was already working on my project, but not making any progress because my code was infested by bugs (I was out of shape). I suspended my activity to inspect the edited HSQC. Processing it was a relaxing activity, which I did on autopilot. When I saw the 2D plot in front of me, I said to myself: "What have you done? You have corrected the phase without paying the least attention to which peaks are positive and which are negative. Actually you haven't even paid the least attention to where the peaks are located, but the phase is perfectly corrected! What does it mean?"
There was an explanation. Manual phase correction is ALWAYS so easy, at least the coarse correction is easy. I know that if I am moving the slider into the good direction, peaks become narrower and take less space into the map. Because my background is usually white, I know that if the white regions are growing, that is a sign that I am going into the right direction. I stop when the white regions start shrinking again. I immediately realized that my mental process could be easily translated into a computer algorithm. I also noted that the correction along the X axis is independent from the correction along the Y axis. The four-dimensional problem could be factorized into two bi-dimensional problems, easier and faster. I suspended the original project and began to write the new algorithm. In 24 hours I verified that it worked and worked very well. I also debugged the old code (belonging to the original project) and discovered that it was quite unstable: it only worked under ideal conditions.
I was forced to abort the first project, but very happy of having invented something really useful and really new (in a single stage). The weakness of the aborted project was that it had to observe a limited number of points at each time. The whitening method, instead, looks at one million points simultaneously. It's a statistic measure and it's reliable just because it condenses one million of observations.
Today the whitening method is a component of 3 commercial products. It has been already presented at a meeting and (in the form of poster) at a conference. We have written an article and submitted it to Magnetic Resonance in Chemistry. You may be wondering what the hell we could have written into that article, if six words are enough to state the method: "maximize the number of white pixels". Wait and see. I am convinced that the article will be accepted just because the principle is clear (have you noticed the pun?). If they have published (in many cases of 1-D automatic correction) the articles that describe algorithms that nobody has never seen in practice, why shouldn't they publish ours? Our products can be freely downloaded and you too can test the algorithm!
The natural question is: if this method is so obvious and simple, why nobody else had invented it before? The answer can be found inside Carlos' blog. While I have been performing REAL 2D phase correction for years, there are people that even ignore that such a thing is possible. What they are doing is the correction of selected rows and columns (traces) or their sum. Actually it's a mono-dimensional correction, but they believe it's a 2D correction. A pertaining example can be found on Glenn's Bruker-oriented blog. I have had the privilege of performing true 2-D manual correction and it helped me.
Finally, I must confess: I am having a lot of fun with automatic phase correction and after I have invented it I don't want to go back!
This summer I wrote to Carlos that I really couldn't find anything interesting to do during my holidays and that it was better to do some work. He said he had an idea to keep me busy. Guess what? Inventing the automatic 2D phase correction. This time I agreed. I proposed a three-steps roadmap. The first step consisted into trying to correct the phase of a single isolated peak, adjusting simultaneously the 4 parameters involved. The second step was a semi-automatic method in which the user was required to select any number of peaks, even of different sign, and the computer had to do the rest. The preliminary method invented at the first stage would have been the main ingredient. The third stage was the completely automated method.
I liked this roadmap because the first stage looked feasible (I hate wasting time on hopeless research). The other stages were challenging, but, in my heart, I was hoping to find an escape route before arriving there. I was glad enough to have something to do for the first week.
Carlos asked me if I wanted any 2D spectrum to test my ideas. He had already mentally created a list of different cases with different characteristics. An important case was the edited HSQC, containing both positive and negative peaks (and more noise; and no diagonal). It looked more difficult then the other cases. I said: "I have never acquired this kind of spectrum; send me a specimen". When the file arrived I was already working on my project, but not making any progress because my code was infested by bugs (I was out of shape). I suspended my activity to inspect the edited HSQC. Processing it was a relaxing activity, which I did on autopilot. When I saw the 2D plot in front of me, I said to myself: "What have you done? You have corrected the phase without paying the least attention to which peaks are positive and which are negative. Actually you haven't even paid the least attention to where the peaks are located, but the phase is perfectly corrected! What does it mean?"
There was an explanation. Manual phase correction is ALWAYS so easy, at least the coarse correction is easy. I know that if I am moving the slider into the good direction, peaks become narrower and take less space into the map. Because my background is usually white, I know that if the white regions are growing, that is a sign that I am going into the right direction. I stop when the white regions start shrinking again. I immediately realized that my mental process could be easily translated into a computer algorithm. I also noted that the correction along the X axis is independent from the correction along the Y axis. The four-dimensional problem could be factorized into two bi-dimensional problems, easier and faster. I suspended the original project and began to write the new algorithm. In 24 hours I verified that it worked and worked very well. I also debugged the old code (belonging to the original project) and discovered that it was quite unstable: it only worked under ideal conditions.
I was forced to abort the first project, but very happy of having invented something really useful and really new (in a single stage). The weakness of the aborted project was that it had to observe a limited number of points at each time. The whitening method, instead, looks at one million points simultaneously. It's a statistic measure and it's reliable just because it condenses one million of observations.
Today the whitening method is a component of 3 commercial products. It has been already presented at a meeting and (in the form of poster) at a conference. We have written an article and submitted it to Magnetic Resonance in Chemistry. You may be wondering what the hell we could have written into that article, if six words are enough to state the method: "maximize the number of white pixels". Wait and see. I am convinced that the article will be accepted just because the principle is clear (have you noticed the pun?). If they have published (in many cases of 1-D automatic correction) the articles that describe algorithms that nobody has never seen in practice, why shouldn't they publish ours? Our products can be freely downloaded and you too can test the algorithm!
The natural question is: if this method is so obvious and simple, why nobody else had invented it before? The answer can be found inside Carlos' blog. While I have been performing REAL 2D phase correction for years, there are people that even ignore that such a thing is possible. What they are doing is the correction of selected rows and columns (traces) or their sum. Actually it's a mono-dimensional correction, but they believe it's a 2D correction. A pertaining example can be found on Glenn's Bruker-oriented blog. I have had the privilege of performing true 2-D manual correction and it helped me.
Finally, I must confess: I am having a lot of fun with automatic phase correction and after I have invented it I don't want to go back!
Wednesday, 1 October 2008
Commuting
The canonical sequence is:
Rance -> FT -> shuffle -> FT -> shuffle -> FT
This is the result:
The following sequence also arrives there:
FT -> Rance -> shuffle -> FT -> shuffle -> FT
which proves the commutability between FT and Rance. When I try this:
FT -> shuffle -> FT -> Rance -> shuffle -> FT
the result is:
Rance -> FT -> shuffle -> FT -> shuffle -> FT
This is the result:
The following sequence also arrives there:
FT -> Rance -> shuffle -> FT -> shuffle -> FT
which proves the commutability between FT and Rance. When I try this:
FT -> shuffle -> FT -> Rance -> shuffle -> FT
the result is:
Tuesday, 30 September 2008
Ray Nance
I have a challenge for you. It's a puzzle that I can't find the solution to. Before your challenge, I need to finish the story of my challenge (I already told the beginning and the middle part: see previous posts). To tell my story, however, I need to explain my language first. When I asked for help, on September 9, nobody understood what I really needed, because we speak different languages. While I look at the mathematical description of NMR processing, you think at the software command that performs it. This time I will try to be the clearest I can. The world of NMR is complicated because every program stores data in a different way. There is the sequential storage and the storage in blocks, the little endian and the big endian, the interleaved and the non-interleaved. Add to this that the spectrometer is free, when writing the data on disc, to change the sign to a part of the points. When acquiring a multidimensional spectrum, which is made of many FIDs, it is also free to collect those FIDs in any order (this is the case of Varian's, which has two different ways to acquire a 3D spectra, although one was enough, and none of the two ways is vaguely similar to the Bruker way...). NMRPipe adds a further complication, because when it read Varian data it changes the sign (and the users can't know). There are also different FT algorithms that don't yield the same spectrum (the real part is always the same, but the sign of the imaginary part looks like an arbitrary choice).
I don't want to talk about these dull details: they are hackers' business (and, unfortunately, my business too).
I want to talk, instead, about phase-sensitive detection, because this is where my deepest troubles came from.
Phase-sensitive detection brings so many advantages, in terms of sensitivity and resolution, that it's almost a must in NMR spectroscopy. To accomplish it, the spectrometer measures the magnetization twice (in time or in space). When the spectrum arrives on the disc, two numbers are stored where there used to be one. To accomplish phase-sensitive detection in bidimensional spectroscpy, two FIDs are acquired instead of one. To accomplish phase-detection in 3-D spectroscopy, two bidimensional experiments are performed instead of one. In summary, a 3D FID contains 8 values that correspond to an identical combination of t1, t2 and t3 coordinates (a couple of couples of couples). In practice it can become a mess, but the rules are simple enough. Whatever the number of dimensions, each and every value has a counterpart (think like brother and sister). You don't need to think in terms of rows, planes or cubes. Just remember that each sister has a corresponding brother and vice versa. There is no difference between 2D and 3D spectra. There is an extra operation, however, that is absent in 1D spectroscopy. It's called the Ruben-States protocol (you can add more names to the brand, if you like; I call it, more generically: "shuffling"). How does it work?
We have a white horse and a white mare (brother and sister). We know that every mare has a brother and every horse has a sister. This is also true for the mother of the two white horses. She had a black brother, with a black son and a black daughter. Now, let's swap the black horse with the white mare. The members of the new pairs have the same sex. With this combination you can continue the race towards the frequency domain and win. The trick works in almost all kinds of homo-nuclear spectra. In the 2D case, you have to perform two FTs. The shuffling goes in between. In the 3D case, there are 3 FTs and 2 intermediate shuffles: FT, shuffle, FT, shuffle, FT. Maybe you have heard about alternatives called States-TPPI or the likes. Actually they all are exactly the same thing.
Our trick is not enough when gradient-selection is employed in hetero-nuclear spectroscopy. You hear the names "sensitivity enhanced" or "echo-antiecho" or "Rance-Key". It's easier for me to remember Ray Nance instead, although he had nothing to do with today's subject. Whatever the name, this kind of experiments require an ADDITIONAL step, to be performed before anything else, or at least before the shuffling. Here is what you have to do. Let's say that A is the pair of white horses and B the pair of black ones. Calculate:
S = A + B.
D = A - B.
Now rotate D by 90 degrees. This latter operation has no equivalent in the realm of animals. It would consists into a double transformation: the brother would become female and the sister a... negative male, if such an expression has any sense. Finally, replace A with S and B with the rotated D.
The 3 operations involved (addition, subtraction and phase rotation) all commute with the FT. Knowing this fact, I have always moved Ray Nance from his canonical place (before the first FT), to the moment of the shuffle, and fused the two together. In this way I have eliminated a processing step. The condensed work-flow looks simpler to my eyes, but this is a matter of opinion.
Unfortunately, this was the reason why I could not process Varian 3D spectra: I have tried all the possible sign inversions, but no one worked. Every time the number of peaks was the double (or quadruple) than it should have been. The mirror image was mixed with the regular spectrum.
If, instead, I process the raw data as described in literature, that is before the standard workflow, I get the correct spectrum. Now: I still believe that sum, addition and phase rotation commute with FT. I can apply this property to all the 2D spectra and to all the Bruker 3D spectra I have met so far. The property ceases to work in the case of Varian spectra.
Question: do you know why? I don't.
I don't want to talk about these dull details: they are hackers' business (and, unfortunately, my business too).
I want to talk, instead, about phase-sensitive detection, because this is where my deepest troubles came from.
Phase-sensitive detection brings so many advantages, in terms of sensitivity and resolution, that it's almost a must in NMR spectroscopy. To accomplish it, the spectrometer measures the magnetization twice (in time or in space). When the spectrum arrives on the disc, two numbers are stored where there used to be one. To accomplish phase-sensitive detection in bidimensional spectroscpy, two FIDs are acquired instead of one. To accomplish phase-detection in 3-D spectroscopy, two bidimensional experiments are performed instead of one. In summary, a 3D FID contains 8 values that correspond to an identical combination of t1, t2 and t3 coordinates (a couple of couples of couples). In practice it can become a mess, but the rules are simple enough. Whatever the number of dimensions, each and every value has a counterpart (think like brother and sister). You don't need to think in terms of rows, planes or cubes. Just remember that each sister has a corresponding brother and vice versa. There is no difference between 2D and 3D spectra. There is an extra operation, however, that is absent in 1D spectroscopy. It's called the Ruben-States protocol (you can add more names to the brand, if you like; I call it, more generically: "shuffling"). How does it work?
We have a white horse and a white mare (brother and sister). We know that every mare has a brother and every horse has a sister. This is also true for the mother of the two white horses. She had a black brother, with a black son and a black daughter. Now, let's swap the black horse with the white mare. The members of the new pairs have the same sex. With this combination you can continue the race towards the frequency domain and win. The trick works in almost all kinds of homo-nuclear spectra. In the 2D case, you have to perform two FTs. The shuffling goes in between. In the 3D case, there are 3 FTs and 2 intermediate shuffles: FT, shuffle, FT, shuffle, FT. Maybe you have heard about alternatives called States-TPPI or the likes. Actually they all are exactly the same thing.
Our trick is not enough when gradient-selection is employed in hetero-nuclear spectroscopy. You hear the names "sensitivity enhanced" or "echo-antiecho" or "Rance-Key". It's easier for me to remember Ray Nance instead, although he had nothing to do with today's subject. Whatever the name, this kind of experiments require an ADDITIONAL step, to be performed before anything else, or at least before the shuffling. Here is what you have to do. Let's say that A is the pair of white horses and B the pair of black ones. Calculate:
S = A + B.
D = A - B.
Now rotate D by 90 degrees. This latter operation has no equivalent in the realm of animals. It would consists into a double transformation: the brother would become female and the sister a... negative male, if such an expression has any sense. Finally, replace A with S and B with the rotated D.
The 3 operations involved (addition, subtraction and phase rotation) all commute with the FT. Knowing this fact, I have always moved Ray Nance from his canonical place (before the first FT), to the moment of the shuffle, and fused the two together. In this way I have eliminated a processing step. The condensed work-flow looks simpler to my eyes, but this is a matter of opinion.
Unfortunately, this was the reason why I could not process Varian 3D spectra: I have tried all the possible sign inversions, but no one worked. Every time the number of peaks was the double (or quadruple) than it should have been. The mirror image was mixed with the regular spectrum.
If, instead, I process the raw data as described in literature, that is before the standard workflow, I get the correct spectrum. Now: I still believe that sum, addition and phase rotation commute with FT. I can apply this property to all the 2D spectra and to all the Bruker 3D spectra I have met so far. The property ceases to work in the case of Varian spectra.
Question: do you know why? I don't.
Saturday, 13 September 2008
Passion
I want to thank Dan, Rolf and Richard who answered to my request for help. I have carefully studied their replies, including the enclosed documents, and I have learned to process a couple of beautiful 3D Varian spectra. There are other cases that still resist to my attempts, yet the progress is remarkable.
I have decided to share my work with you (and anybody else). Instead of publishing the details (that few would read), I have uploaded the program that processes THOSE 3D spectra (see previous post). It's lightweight and it's free. The access is also unrestricted. What else you want? Simplicity? It's included! Even a kid could transform a 3D spectrum...
If you are interested into visual 3D processing "the way I like it", you can continue reading here. Happy Processing!
I have decided to share my work with you (and anybody else). Instead of publishing the details (that few would read), I have uploaded the program that processes THOSE 3D spectra (see previous post). It's lightweight and it's free. The access is also unrestricted. What else you want? Simplicity? It's included! Even a kid could transform a 3D spectrum...
If you are interested into visual 3D processing "the way I like it", you can continue reading here. Happy Processing!
Tuesday, 9 September 2008
Help me Please
For the first time after almost 2 years, you have the oppurtunity to give me something.
There is an apparently nice collection of 3D spectra on the web:
http://www.biochem.ucl.ac.uk/bsm/nmr/ubq/
but I can't process them because they are Varian spectra. If you are able to process at least one of them with any software, would you please explain me what's happening? Or, in other words, what you see?
There is an apparently nice collection of 3D spectra on the web:
http://www.biochem.ucl.ac.uk/bsm/nmr/ubq/
but I can't process them because they are Varian spectra. If you are able to process at least one of them with any software, would you please explain me what's happening? Or, in other words, what you see?
Sunday, 7 September 2008
North
I have began editing the comparative price list, instead of republishing it periodically. You find the link at the top of the sidebar. Today I have added three new entries, and you'll notice they are all quite expensive; this fact is remarkable, because those programs aren't generally considered commercial. Actually they are the most expensive ones! NMR is extremely specialistic, and for this reason it becomes impossible to keep a freeware alive for a long time.
You go nowhere without money, as SideSpin exemplifies so clearly!
Apart from the boring pricing considerations, you can follow the links to explore the new websites of NMRPipe, CYANA and LCModel. There is a striking contrast between the informative and elegant site of the first one and the extreme simplicity of the last one. According to the title and the URL, it is the personal site of Stephen Provencher and not the site of the product, but there is no information at all about the author. Even his geographical location is a bit of an enigma. The working address, reported on the publications, is Göttingen, but the home address is Oakville, Ontario. I thought that retired people preferred warm places. There are exceptions to all the rules; what can be an explanation, in this case? The fact that Canada is the home of NMR software? The magnetic attraction of the North Pole?
You go nowhere without money, as SideSpin exemplifies so clearly!
Apart from the boring pricing considerations, you can follow the links to explore the new websites of NMRPipe, CYANA and LCModel. There is a striking contrast between the informative and elegant site of the first one and the extreme simplicity of the last one. According to the title and the URL, it is the personal site of Stephen Provencher and not the site of the product, but there is no information at all about the author. Even his geographical location is a bit of an enigma. The working address, reported on the publications, is Göttingen, but the home address is Oakville, Ontario. I thought that retired people preferred warm places. There are exceptions to all the rules; what can be an explanation, in this case? The fact that Canada is the home of NMR software? The magnetic attraction of the North Pole?
Subscribe to:
Posts (Atom)