• View
• Edit
• History
• Print

IlluminaGuide2014

This document is obsolete

ILLUMINA USER'S GUIDE

Martin Aubé, Ph.D. copyright 2014

General information

In that document, the red text means that you have to adapt that part to your specific case.

Operating system

Illumina should be used with a computer running under linux with a fortran and gcc compilers installed (e.g. gfortran) and mercurial.

Other software dependancies

The following software are needed by the system

• make
• bash
• vim-common
• python-numpy
• python-pyproj
• python-matplotlib
  
• python-gdal
• gri
• bc
• imagemagick
• gnuplot
• mercurial

Installing the code

ILLUMINA model is available from bitbucket:

https://bitbucket.org/aubema/illumina

All sources codes are released under GNU public licence.

To install the model from bitbucket, please follow these steps:

If you are a non developer user:

If you are graphycs1 developer:

Then you must compile the code:

Then edit the $HOME/.bashrc file. This will render the executables accessible from anywere on the file system.

Making light inventory for each spectral line

We have to create zones files. In ILLUMINA, zone files allow the user to define different geographical zones with different type of lamp (different by their relative contribution to the VIIRS-DNB satellite imagery, photometry function or light output pattern (LOP), lamp height). Two or more zones may be in the same geographical region or partly overimposed. Since each zone defines the lamp photometry, the user have to combine both lamp type LOP in an average LOP to define some kind of new combined lamp type. By default, as a first step, all the geographical domain will be defined by 2 zones (urban and contryside). These two cases are discriminated on the basis of a threshold value to the VIIRS-DNB relative radiance. Then additional zones are defined as circle shaped zones with center position and radius. Each new zone overwrite the previous. To create zone you have to edit a .zon file with a simple text editor like kwrite or gedit following the format shown below:

Sample .zon file for a Hg spectral line

First line content

This line contain 3 elements:

1. total number of circular zones (their definition will follow in line 3 and up)
2. percentage of VIIRS-DNB luminance coming from the given element (here Hg) in urban regions
3. urban LOP file name

Second line

1. urban/rural VIIRS-DNB luminance threshold (under this threshold, the pixel will be considered as rural, you can make some test with viirs2lum to validate the best threshold value to use)
2. percentage of VIIRS-DNB luminance coming from the given element (here Hg) in rural regions
3. rural LOP file name

Following lines

1. x' central position of the circular zone. {#x'=x+1#} where x is determined with a wheel click on the pgm file with imagemagick (display)
2. y' central position of the circular zone. {#y'=Nby-y #} where y is determined with a wheel click on the pgm file with imagemagick (display) and Nby is the image height in pixel.
3. radius of the circular zone (pixel).
4. percentage of VIIRS-DNB luminance coming from the given element (here Hg) in that zone
5. zone LOP file name

Find a VIIRS-DNB image in ILLUMINA's standard pgm format

Sample file for southern part of Quebec province

Create a reflectance image in ILLUMINA's standard pgm format

We are using MODIS reflectance product for summer (~mid August) and winter (~mid February) at the nearest neighbourg wavelength according to the following table:

You must choose the MODIS date to fit the VIIRS-DNB date. Example of MODIS band 4 reflectance for year 2009:

Run viirs2lum program

Run program viirs2lum with files created at steps 5.1, 5.2, 5.3 as summarized below:

1. Output root name of the experiment: this name will be used to create the output luminosity files (_lumlp_NN.pgm) will be appended, with NN giving the zone number (01 is for urban and 02 for rural).
2. Fichier viirs (format pgm illumina): file name of step 5.2
3. Fichier reflectance (format pgm illumina): file name of step 5.3
4. Fichier zone: file name of step 5.1

Quick file check

Open all the _lumlp_NN.pgm files to confirm that they are realistics according to the specific experiment. lumlp files are giving the light luminosity for each zone at the given wavelength.

Examples with 546 Hg line but contrast boosted

Zone 1 = cities	Zone 2 = countryside
Zone 3 = Sherbrooke	Zone 4 = 25 km around Obs. Mont-Mégantic
Zone 6 = Cookshire	Zone 7 = Lac Mégantic

Zone 5, wich is the Astrolab parking lot, have not been shown here because there is no Hg lamps. This image was consequently completely black.

Preparation of the batch file

This file is a bash program that will generate a lot of running experiment according to the number of desired cases for each variable.

We must define the following variables in the makeBATCH program:

We must also define the following arrays in makeBATCH program:

• Note that the example given above was for summer case but the data should be define independently for each period.

Preparing a different directory for each period of the year and for each relative humidity

Create a directory for each air moisture analyzed in which two sub-directory will be included, e.g. for summer and winter. In our case, we create three directories or RH50, RH80 and RH70.

Then, we must put in each directory all the files required for the computation.

Submitting the calculations to a linux cluster

To perform the calculations, we now connect to a 'Cluster'. In our case, we connected to 'Galileo' or 'Mammouth serial II' respectively at Cégep de Sherbrooke and at Université de Sherbrooke.

Then it is necessary to recompile the ILLUMINA model using 'makeILLUMINA' or through the following command:

The required directories created in step 7 should now be transferred to the cluster interactive node via the scp protocol.

Preparing the batch execution

Now we need to adapt the program that will run the multiple calculations (i.e. makeBATCH of makeBATCH-territory) for each experiment directory on the cluster. Copy the original file from $HOME/svn/illumina/trunk to the directories and then edit the makeBATCH file in each directory to reflect the particular situation (season or period, relative humidity). Then execute the script:

Notes:

• output_batch_file take any value. If no name is provided then makeBATCH will choose the default name (TortureMammouth).
• Use the command qstat or bqmon -u to verify the status of 'clusters' (compute nodes) before or during execution.
• To delete a task, use the qdel followed by the job number to delete.
• To delete all your jobs use the following command:

To execute the calculations, perform the following command:

This step should be repeated for each season or period and each relative humidity value.

Extracting results

ILLUMINA generates two image files by calculation, a file showing the contribution (PCL) and the other illustrating the sensitivity to light pollution (PCW). It also produces the radiance value in the calculated direction.

Now it's time to extract the data and rename files obtained since the names assigned by % blue% output_batch_file do not indicate all the required values for the files to be indexed correctly on the portal. Currently this information is listed in the directory names containing the results.

To extract the data, you have to go to each execution directory (each one must contain a gridmap directory)

create a list of files to extract. To do so go to the $HOME directory and type:

here word is a string contained in all the execution experiment. Cop the .list file in the execution diretory and extract the data:

At the end of the execution of extract-output-data.bash a Results directory has been created and it will contain all PCL files (ex .: PCL-x302y58-2005-ta0 .05-wl436-el15-az0.pgm), all PCW files (ex.L PCW-x302y58-2005-ta0.05-wl436-el15-az0.pgm) and .out files (the log of each execution).

Add some color to your life

Il peut être plus facile d'analyser une carte lorsque celle-ci est en couleur. Pour ce faire, il faut d'abord savoir quel est la résolution numérique de l'image (8bit ou 16 bit) à l'aide de la commande suivante:

• Si le nombre indiqué est 255, la résolution est de 8 bit.
• Si le nombre indiqué est 65535, la résolution est de 16 bit.

Selon la résolution de l'image, on utilisera le programme pgm8bit2color.bash ou le programme pgm16bit2color.bash.

Donc, on entre le programme approprié suivi du nom du fichier à mettre en couleur. Par exemple:

Renommage des fichiers pour le portail

Pour être bien indexés par le portail les fichiers doivent être nommés comme suit:

PCL-Astrolab_b_midnight_summer_2005-ta0.05-wl436-el15-az0.pgm

Signification: PCL = type de données Astrolab = nom du site d'observation b_midnight_summer_2005 = période (b_midnight est pour before midnight, période par défaut des données VIIRS-DNB) ta0.05 = épaisseur optique des aérosols de 0.05 wl436 = longueur d'onde de 436nm el15 = 15 deg d'angle d'élévation az0 = 0 deg d'angle d'azimuth

Exemple de commande pour renommer les fichiers ci-haut correctement:

Essentiellement cette commande changera x302y58-2005 pour Astrolab-b_midnight_summer_2005. I.e. avant le tiret, c'est le nom du site d'observation qui remplace la position et après on met la période de la nuit suivie de la saison et de l'année. Les tirets séparent les champs utilisés par le portail alors que les underscore servent à séparer les mots d'un champs donné. Par ex., le champ période contient 4 mots (b, midnight, summer, 2005). Notez que si la modélisation a été faite pour après minuit on utilisera les mots a et midnight (a_midnight) et si la modélisation correspond à l'hiver on utilisera le mot winter. Le nom du site est à la discrétion de l'utilisateur du présent guide mais devra ne pas porter à confusion pour les futurs usagers du portail.

Quick using guide for mercurial

To publish your locally modified codes on the server

To update your local version of the codes