Fityk - User's Manual

The window of fityk program consists of (from the top): menu bar, toolbar, main plot, auxiliary plot, output window, input field and status bar. The input field allows to type and execute commands in similar way as it is done in CLI version. The output windows shows results. Click the right mouse button on it to see its configuration options.

Most of menu commands correspond to commands described in Chapter 3, Command reference . Some of them are simply translated to text strings and executed - you can see it in output window. I hope that menu commands are self-explaining (especially after reading the rest of this manual) - just try them.

The main plot can display data points, functions, phases and/or the sum of all functions. Use pop-up menu (click right button on the plot) to configure it. Phase is a sum of related functions. The word comes from the crystallographic module, where a phase of investigated powder is represented by a set of peaks.

The auxiliary plot can display a difference of data and the sum of functions, the difference divided by standard deviation of data or another plot. It can be controlled by its pop-up menu. I hope having a look at this menu and a minute of experiments will explain possibilities of the auxiliary plot.

Sometimes it is useful to print effects of our fitting work. Hard copy will contain plots visible on the screen, scaled to fit the page. The only difference is that backgrounds of plots will be white (to not waste toner/ink). So it may be necessary to change color of axis or data to darker one.

Configuration of GUI (everything that can be changed using pop-up menus and size of the window) can be saved using GUI->Save current config. Two different configurations can be saved, what allows easy changing of colors for printing. On Unix platform, these configurations are stored in file in user's home directory. On Windows - they are stored in registry (perhaps in future they will be also stored in file).

Now it is a time to explain quite complicated mouse usage. The usage of the mouse on menu, dialog windows, input field and output window is intuitive, so the only topic described here will be: how to effectively operate mouse on plots.

Let us start with the auxiliary plot. Right button displays pop-up menu, with left button you can select range to be displayed - range on x axis. Clicking with middle button (or with left button and pressed Shift) will zoom it out and display all data.

On the main plot, meaning of the left and right mouse button depends on current mode, that can be changed using toolbar or menu. You will see hints on status bar. In normal mode, left button is used for zooming and right invokes pop-up menu. The same behaviour can be obtained in any mode by pressing Ctrl (or Alt or Shift). Middle button can be used to select a rectangle, that you want to zoom in. If an operation has two steps, like rectangle zooming (first you press button to select first corner, then you move mouse and release button to select second corner of rectangle), you can cancel it by pressing another button when first one is pressed.

Data are stored in files. Unfortunately, there are various formats of files with data. Usually we have text files and one line in file corresponds to one data points. Every line should contain at least two numbers: x and y of point. It can also contain standard deviation of y coordinate. Numbers can be separated by whitespace, commas, colons or semicolons. Some lines can contain comments or extra informations. If these lines have a hash (#) in first column, they are ignored. In other case, they are usually also ignored. There are also other file types, that can be read: .rit and .mca. In future, the way the special file formats are handled will be changed.

Points are loaded from files using command

where xcol, ycol, scol, are unsigned integers. Note that the name of the file is inside of single quotation marks. If the file is in a normal format (columns with numbers) it can be specified which column contains x, y and, optionally, std. dev. of y.

Command

can export data to ASCII file, that has 3 columns: x, y and standard deviation. Only active points are being exported (see next chapter to learn about active and inactive points). + will cause, if specified file exist, appending to the file rather than overwritting.

Some information about current data can be obtained using command:

We often have situation, that only part of data from file is interesting for us. We should be able to exclude selected points from fitting and all computations. It can be done with command d.transform (which will be explained later) or with mouse-click in GUI. The idea of active and inactive points is simple: only the active ones are subject to fitting and peak-finding, inactive ones are neglected in these cases.

When fitting data, we assume that only y coordinate of data is subject to statistical error in measurement. Is is very common assumption. To see how y standard deviation influences fitting (optimization), look at weighted sum of squared residuals formula in the section called “Nonlinear optimization ”. We can also think about weights of points - every point has a weight assigned, that is equal

Standard deviation of points can be read from file together with x and y coordinates. Otherwise, it is set to max(sqrt(y), 1.0), Setting std. dev. as a square root of value is common and has theoretical grounds when y is the number of independent events.

In fityk 0.5.0 new powerful command d.transform was introduced. It's not documented here yet. The best documentation for a while is the extensive comment at the beginning of datatrans.cpp source file. You can see CVS version of the file here: http://cvs.sourceforge.net/viewcvs.py/fityk/fityk/src/datatrans.cpp?only_with_tag=HEAD&view=markup

Historically, there were separete commands for dealing with spline background and with calibration. Now all changes to data points are made using d.transform command, but there is still background mode in GUI, which helps in manual baseline substraction and generates proper d.transform command to apply the change to the data. Have a look at two useful GUI options related to background mode: background type and minimal BG points distance.

Sometimes the instrument introduces errors to the x coordinate of the data points, which can be determined by using a reference sample. Correcting these errors is called here calibration. (See also description of zero-shift.) Historically, there was a special command for calibration in fityk, but it was removed to simplify the program. Now you can change x coordinate of data points using d.transform command, analogically to baseline substraction.

Let call a set of data that usually comes from one file - a dataset. All operations described above concern one dataset, but you can also work with many datasets. Datasets can be grouped in plots. There can be any number of plots, and any number of datasets in each plot. There is always one active plot and one active dataset in each plot. Inactive datasets are drawn, but do not influence any operations. So there are only two reasons to have more than one datasets in one plot - it is possible to compare them visually and to easily switch between them. Let me repeat: all command are related only to the active dataset.

A so-called plot contains not only datasets, but also described below so-called sum. Only datasets and sum in active plot are drawn and all commands except fitting are related to active plot. But for f.xxx commands it doesn't matter if plot is active - all plots are fitted simultanously. This is why one may want to have many plots - to fit simultanously datasets with functions, that can share its parameters.

You can manage plots and datasets using the data pane in GUI or d.activate command. The following syntax:

is used to switch to plot p and dataset d. It is possible to change only active plot or to change only active dataset. To create (and activate) new plot or new dataset replace p or d with asterisk (*).

The sum of functions S - curve that is fitted to data - is itself a function. The value of the whole sum is computed as a sum of functions, like Gaussians or polynomials, and can be given by formula: , where f_i is a function of x, and depends on a vector of parameters a. This vector contains all fitted parameters. Because we often have the situation, that the error in the x coordinate of data points can be modeled with function z(x; a), we introduce this term to sum:

where . Note that the same z(x) is used in all functions.

Now we will have a closer look at f_i functions. Every function f_i has a type chosen from function types available in program. One of these types is the Gaussian. It has the following formula:

a₀ is a height of peak, a₁ - position of center and a₂ - half width at half maximum. Every of these parameters can be:

Functions f_i are called f-functions. See the list of available f-function types in Appendix C, List of functions.

Functions z_i(x;a) are called zero-shifts. This name is not precise (FIXME: what is a better name? calibration function? instrumental function?). The simplest of them has a formula: z = a₀ and can be used for modeling error of shifting zero in instrument. In general case, so-called zero-shifts are parametrized functions of x, like f-functions.

Every f-function, zero-shift and g-parameter has its type and its parameters - the number of its parameters is determined by the type. Two examples:

F-functions, zero-shifts, g-parameters and simple-parameters are elements of a sum. Every sort of element has a symbol: f-function (^), zero-shift (<), g-parameter ($) and simple-parameter (@). Every f-function contained in the sum has a number assigned, that allows referring to this f-function. If there is only one f-function defined, it has number 0. If there are two f-functions, the first has number 0 and the second 1. As you will see further in this chapter, commands refer to f-functions using symbol (^) and number, eg. ^0 or ^15. The same about other sorts of elements: first zero-shift is referred to as <0, fifth simple-parameter - as @4, etc.

Three kinds of parameters were described above. They are used in f-functions, zero-shifts and g-parameters, and are denoted in adequate formulae as a_i. Note, that actually only simple-parameters are fitted. The term parameter used in context of fitting and the term vector of parameters refer to simple-parameters only.

Every simple parameter has a value and, optionally, domain. Command

creates simple-parameter equal p. Once created, the parameter will be given a number (first available), say m, and later can be referred to as @m. Domain is used only by fitting algorithms, when they need to randomly initialize variable, or change it. In some cases only width of domain is needed. Domain must be enclosed by square brackets. It can be specified in three ways:

If domain of parameter is not defined, it is assumed to be [v - r v, v + r v], where v is the value of simple parameter and r is the value of option default-relative-domain-width.

F-functions, zero-shifts and g-parameters can be created using command:

where t is a letter denoting type of f-function, zero-shift or g-parameter (respectively ^t, <t or $t). It is followed by parameters in number required for given type of object. Parameters can be given as:

In particular, a g-parameter can be defined as a function of another g-parameters. As a result of it, every parameter of f-function (eg. of Lorentzian function) can be an arbitrary combination of simple parameters. You can learn about available types using command s.info described below.

Every f-function, zero-shift, g-parameter or simple-parameter, that was created (added), can also be removed. The only condition is, that there are no references to it. It is obvious - you can not remove a simple parameter, that represents a height of a peak, before removing the peak. The syntax of command is straightforward:

n is the number of f-function, zero-shift, g-parameter or simple-parameter. It can be replaced with an asterisk(*). Command s.remove ^3 will remove ^3, and s.remove ^* will remove all f-functions. To avoid gaps in numeration, all larger numbers are decreased by one. If f-function ^3 is removed, ^9 will become ^8 and ^4 will become ^3. All references are properly updated. If option recursive-remove is set, also parameters, in our example parameters of ^3, will be removed.

Information about every element of the sum can be obtained using command:

And informations about available types of f-functions, zero-shifts and g-parameters can be displayed with:

t is a letter that stands for type, eg. ^G means f-function Gaussian. If type is not specified, list of all available types will be displayed.

The whole sum or selected f-functions can be saved to file in various formats with command:

where t is a letter that stands for format of file: f - mathematical formula, d - xy points, p - peak listing, x - XFIT peak listing (used by some crystallographic software), s - fityk script. + will cause, if specified file exist, appending to the file rather than overwritting.

Value and domain of simple parameters can be changed with command:

p is a new value of @n, or, if % is given, new value is equal to p% of current value. domain is specified in the same way as in command s.add. If domain is not given, it is not changed.

Vector of simple parameters is a subject to change in a fitting process. Sometimes it is useful to refine only selected parameters. The parameters that are not to be refined are called frozen. You can freeze and unfreeze parameters with command:

This command will freeze @n if exclamation mark (!) is not present and unfreeze if it is present. To freeze or unfreeze all parameters, use asterisk(*) instead of n. Example: s.freeze @*, ! @0, ! @3 causes, that only @0 and @3 would be fitted. To display information about currently frozen parameters, just type:

Every change of values of parameters is remembered. It is remembered since last adding or removing simple parameter operation. You can display history of changes with command:

and move to selected position in history with command:

Without sign - you will move to n-th position, with plus(+) or minus(-) - you will go n steps forward or backward. The history is similar to a history in web browser - if you move backward and then add a new entry, eg. by fitting, last entries of history will be erased. In some cases it can be reasonable to fit curve with one method, move back to initial parameters, fit with another method and compare results. To make this comparison easier, you can save result obtained with first method using command:

The command toggles saved and not saved state of history item, what means that using it second time on the same item will allow to delete it. If n is not specified, current history item is used.

Now it is time to explain how value of the sum and its derivatives are computed. You may skip it and read next chapter, if you are not interested.

All fitting methods involve computing value of the sum S(x) for x = x₁ x₂ ..., x_N and such series of computations are repeated multiple times, for different parameter vectors. x₁ x₂ ..., x_N are x coordinates of active data points. In typical cases N >> 1, therefore, when thinking about time consumed by computations, only operations executed for every point matter.

At the beginning of every series of S(x) computations values of g-parameters and some expressions in f-functions, that do not depend on x, are computed. Then S(x) value of every f-function and zero-shift is computed in every point. To speedup computations, tails of peaks, that are very small, can be neglected. This can be controlled with two options: cut-tails and cut-tails-level. If option cut-tails is set, at the beginning of each series of computations, for every peak, points, where the peak has value approximately equal to value of cut-tails-level option, are computed. Outside of these points, value of the peak function is assumed to be zero.

Most commonly used nonlinear fitting methods (in our program Levenberg-Marquard) are using first derivatives of functions. The program knows derivative formulae for all types of f-functions, zero-shifts and g-functions, but it can also use numerical differentiation. If is useful only for testing the program. Numerical differentiation is used when option numerical-d is set. If option numerical-d-both-sides is set, it uses formula df(x)/dx=(f(x+h)-f(x-h))/2h, otherwise df(x)/dx=(f(x+h)-f(x))/h. h is computed as value of option numerical-d-rel-change multiplied by x.

A value either of the whole sum or of a an element of the sum, and all appropriate derivatives can be viewed using command:

eg. s.value ^0 41.5. With asterisk(*), derivatives, computed symbolically and numerically are shown and compared, what is useful for program developers.

This is the core. We have a set of observations (data points), and we want to fit a model (sum of functions), that depends on adjustable parameters, to observations. Let me quote Numerical Recipes, chapter 15.0, page 656 (if you do not know the book, visit http://www.nr.com ):

Our function of merit is WSSR - weighted sum of squared residuals, called also chi-square:

Weights can be are based on standard deviations, . You can learn why squares of residuals are minimized eg. from chapter 15.1 of Numerical Recipes. So we are looking for a global minimum of chi². This large field of numerical research - looking for minimum or maximum - is usually called optimization; it is non-linear and global optimization. Fityk implements three very different optimization methods. All are well-known and described in many book.

You can switch between fitting methods using command:

where f is one of following lower case letters: g - Genetic Algorithm, m - Levenberg-Marquardt gradient method, s - simplex method. If f is not specified, current fitting method is displayed.

To run fit, use commands

and

The first initializes and runs fitting algorithm, and the second starts fitting without the initialization. n is the maximum number of iterations. All non-linear fitting methods are iterative, and there are two common stopping criteria. The first is the number of iterations. If it is not specified with command, value of option default-max-iterations is used. And the second is the number of evaluations of objective function (WSSR), specified by value of option max-wssr-evaluations. It is approximately proportional to time of computations, because most of time in fitting process is taken by evaluating WSSR. There are also other criteria, different for each method. Note that values of all options, even those common for all methods, are set separately for every method.

If you give too small n to f.run command, and fit is stopped because of it, not because of convergence, it makes sense to use f.continue command to process further iterations. [TODO: how to stop fit interactively]

When fit is running, each iteration outputs some informations. If they are scrolling too fast, you can reduce it n times by assigning n to output-one-of option.

Setting o.set autoplot = on-fit-iteration will draw a plot after every iteration, to visualize progress. (see autoplot)

Command ^[1] :

can be used to display WSSR, symmetric errors and variance-covariance matrix.

Available methods can be mixed together, eg. it is sensible to obtain initial parameter estimates using simplex method, and than fit it using Levenberg-Marquard method. Command s.history can be useful for trying various methods with different options and/or initial parameters and choosing the best solution.

Some fitting methods are using random number generator. In some situations one may want to have repeatable and predictable results of fitting, eg. to make a presentation. Seed for a new sequence of pseudo-random numbers is set at the beginning of fitting initialization (when f.run is called). If value of pseudo-random-seed option is set to -1, the seed is based on system time and sequence of pseudo-random numbers is every time different. In other case, if pseudo-random-seed option has a non-negative value, this value is used as a seed. Remember, that like all other options, value of pseudo-random-seed is independent for each fitting method.

It is a standard of nonlinear least-squares routines. It involves computing first derivatives of functions. For description of L-M method see Numerical Recipes, chapter 15.5 or Siegmund Brandt Data Analysis, chapter 10.15 (there is a Polish translation of the second). In a few words: it combines inverse-Hessian method (called Gauss-Newton method?) with steepest descent method by introducing lambda factor. When lambda is equal 0, the method is equivalent to inverse-Hessian method. When lambda increases, the shift vector is rotated toward the direction of steepest descent and the length of the shift vector decreases. (The shift vector is a vector that is added to the parameter vector.) If the better fit is found in iteration, lambda is decreased - it is divided by value of lambda-down-factor option (default: 10). Otherwise, lambda is multiplied by value of lambda-up-factor (default: 10). You can also change a value of lambda-starting-value option.

Marquardt method has one stopping criterium apart from common stopping criteria. If it happens two times in sequence, that relative progress in value of objective function (WSSR) is smaller then value of stop-rel-change option, fit is considered to be converged and is stopped.

L-M method finds a minimum quickly. The question is, if it is the global minimum. It can be a good idea to add a small random vector to the vector of parameters and try again. This small shift vector is added, when value of shake-before option is positive (by default it is 0). Value of every parameter's shift is independent and randomly drawn from distribution of type specified by value of shake-type option (see option distrib-type) in simplex method). The expected value of parameter shift is directly proportional to both value of shake-before option and width of parameter's domain.

This time I am quoting chapter 4.8.3, p. 86 of Peter Gans Data Fitting in the Chemical Sciences by the Method of Least Squares

This method is also described in previously mentioned Numerical Recipes (chapter 10.4) and Data Analysis (chapter 10.8).

After changing current fitting method to Nelder-Mead simplex, what can be done, as you already know, either using menu or using command f.method s you have a few options specific to this method, that can be changed. One of those is a stopping criterium min-fract-range. If value of expression 2(M-m)/(M+m), where M and m are values of worst and best vertices respectively (values of objective functions of vertices, to be precise), is smaller then value of min-fract-range option, fitting is stopped. In other words, it is stopped if all vertices are at almost the same level.

The rest of options is related to initialization of simplex. Before starting iterations, we have to chose set of points in space of parameters, called vertices. Unless option move-all is set, one of these points will be the current point - values that parameters have at this moment. All but this one are drawn as follows: each parameter of each vertex is drawn separately. It is drawn from distribution, that has center in center of domain of parameter, and width proportional to both width of domain and value of move-multiplier parameter. Distribution type can be set using option distrib-type as one of: uniform, Gaussian, Lorentzian and bound. The last one causes value of parameter to be either greatest or smallest value in domain of parameter - one of two bounds of domain (assuming that move-multiplier is equal 1).

[TODO]

For now, fityk offers only primitive algorithm for peak-detection. It looks for highest point in given range, and than tries to find out width of peak. This method is used by crystallographic module, and it is used when adding peaks (semi-)automatically in graphical version of the program. It can be called directly, using command:

If center and margin are not specified, highest peaks are searched in all data, otherwise only a range (center - margin ; center + margin) is considered. Default value of margin is equal to value of option search-width. Option estimate-consider-sum is set by default and causes that program is looking for peaks by analyzing difference of the data and the sum of functions. If not set, only data will be considered. Setting option cancel-peak-out-of-search disables adding peak, when the highest point in the range is found at the border of requested range.

You can find following script and data used by it in samples directory.

## this script is an example of crystallography/diffraction module usage

 d.load 'SiC_Zn.dat' # Load data file from current directory (SiC+Zn data)

 #background was obtained by clicking on plot in "background mode"
 d.background  20.4823 43.3474 , 28.004 24.3128 , 31.4412 23.6984 , 
 d.background  33.7911 36.0138 , 38.6477 30.5403 , 98.9835 23.6079 , 
 d.b 98.9835 23.6079 , 79.5965 22.1979 , 65.1888 17.968 , 49.9335 17.263

 d.range [ 31 : ] # only data with 2theta greater then 31.0 are active

 #Used wavelength in known and we do not want to fit it.
 # If you would like to fit wavelength, replace '_' with '~'
 c.w _1.54051 # Cu Ka1 

 c.set peak-type = Mod-TCHpV 

 #define phases and initial lattice parameters
 c.add %c ~4.35  #SiC, %0 
 c.add %h ~2.66  ~4.92  #Zn, %1 

 #defining some peaks (order does not matter)
 c.add %0 (220) # (220) plane of SiC phase
 c.add %1 (002) # (002) plane of Zn phase
 c.add %0 (111)
 c.add %1 (100), %1 (101) # do you remember how comma(,) works?
 c.a %1 (102) # every command can be shortened with x.x (letter-dot-letter)

 #after adding a few peaks, it can be a good idea to fit it, to obtain
 #better approximation of lattice parameters
 #f.run 10 

 # the rest of peaks
 c.add %1 (103), %1(110), %1(112), %1(200), %1(201), %1(202)
 c.add %0 (311)

 s.add <x ~0 #zero-shift

 f.run # running default fitting method (Lev-Mar)
 #'lambda' that you can see in output window is a parameter used by
 # fitting method, not wavelength.

 c.info # see refined lattice parameters
 
 # if you want to zoom, the easiest way is to use left button on auxiliary
 # plot (this plot with difference). To zoom out, use middle button on it.
 # To see individual peaks, select Show->Peaks from pop-up menu.