This document is obsolete

ILLUMINA USER'S GUIDE

General information

In that document, the red text means that you have to adapt that part to your specific case.

Operating system

Illumina should be used with a computer running under linux with a fortran and gcc compilers installed (e.g. gfortran) and mercurial.

Other software dependancies

The following software are needed by the system

make
bash
vim-common
python-numpy
python-pyproj
python-matplotlib
python-gdal
gri
bc
imagemagick
gnuplot
mercurial

Installing the code

ILLUMINA model is available from bitbucket:

https://bitbucket.org/aubema/illumina

All sources codes are released under GNU public licence.

To install the model from bitbucket, please follow these steps:

If you are a non developer user: (:source lang=bash:) cd mkdir hg cd hg hg clone https://bitbucket.org/aubema/illumina (:sourceend:)

If you are graphycs1 developer: (:source lang=bash:) cd mkdir hg cd hg hg clone https://graphycs1@bitbucket.org/aubema/illumina (:sourceend:)

Then you must compile the code:

(:source lang=bash:) cd illumina bash makeILLUMINA (:sourceend:)

Then edit the $HOME/.bashrc file. This will render the executables accessible from anywere on the file system.

(:source lang=bash:) echo "PATH=$PATH:$HOME/hg/illumina/:$HOME/hg/illumina/bin" >>$HOME/.bashrc cd source .bashrc (:sourceend:)

Making light inventory for each spectral line

We have to create zones files. In ILLUMINA, zone files allow the user to define different geographical zones with different type of lamp (different by their relative contribution to the VIIRS-DNB satellite imagery, photometry function or light output pattern (LOP), lamp height). Two or more zones may be in the same geographical region or partly overimposed. Since each zone defines the lamp photometry, the user have to combine both lamp type LOP in an average LOP to define some kind of new combined lamp type. By default, as a first step, all the geographical domain will be defined by 2 zones (urban and contryside). These two cases are discriminated on the basis of a threshold value to the VIIRS-DNB relative radiance. Then additional zones are defined as circle shaped zones with center position and radius. Each new zone overwrite the previous. To create zone you have to edit a .zon file with a simple text editor like kwrite or gedit following the format shown below:

Sample .zon file for a Hg spectral line

(:source lang=fortran:) 5 14 95Cobra_5Sentinel.dat ! Number_circular_zones [City:86Hg] photometry-file-name-city 0.98 34 66Cobra_34Sentinel.dat ! city-threshold [Countryside:66Hg] photometry-file-name-countryside 241 50 8 10 93Cobra_7Helios.dat ! x-pos_1 y-pos_1 r_1 [Sherbrooke(zone1):90Hg] photometry_file_zone1 306 58 25 11 67Helios_22Cobra_11Sentinel.dat ! x-pos_2 y-pos_2 r_2 [IDSR(zone2):89Hg] photometry_file_zone2 305 55 2 0 A210.2.dat ! x-pos_3 y-pos_3 r_3 [Astrolab's parking(zone3):LPS 0Hg] photometry_file_zone3 264 52 4 20 40Helios_40Cobra_20Sentinel.dat ! x-pos_4 y-pos_4 r_4 [Cookshire(zone4):80Hg] photometry_file_zone4 321 73 3 20 40Helios_40Cobra_20Sentinel.dat ! x-pos_5 y-pos_5 r_5 [Lac-Megantic(zone5):80Hg] photometry_file_zone5 (:sourceend:)

First line content

This line contain 3 elements:

total number of circular zones (their definition will follow in line 3 and up)
percentage of VIIRS-DNB luminance coming from the given element (here Hg) in urban regions
urban LOP file name

Second line

urban/rural VIIRS-DNB luminance threshold (under this threshold, the pixel will be considered as rural, you can make some test with viirs2lum to validate the best threshold value to use)
percentage of VIIRS-DNB luminance coming from the given element (here Hg) in rural regions
rural LOP file name

Following lines

x' central position of the circular zone. {#x'=x+1#} where x is determined with a wheel click on the pgm file with imagemagick (display)
y' central position of the circular zone. {#y'=Nby-y #} where y is determined with a wheel click on the pgm file with imagemagick (display) and Nby is the image height in pixel.
radius of the circular zone (pixel).
percentage of VIIRS-DNB luminance coming from the given element (here Hg) in that zone
zone LOP file name

Find a VIIRS-DNB image in ILLUMINA's standard pgm format

Img:olsQC2009.jpg Sample file for southern part of Quebec province

Create a reflectance image in ILLUMINA's standard pgm format

We are using MODIS reflectance product for summer (~mid August) and winter (~mid February) at the nearest neighbourg wavelength according to the following table:

Element	Wavelength	nearest MODIS band
	±0,3
Mercury	435.5 nm	band 3: 470 nm
Sodium	498.0 nm	band 3: 470 nm
Mercury	546.0 nm	band 4: 555 nm
Sodium	569.0 nm	band 4: 555 nm
Sodium	615.5 nm	band 1: 648 nm

You must choose the MODIS date to fit the VIIRS-DNB date. Example of MODIS band 4 reflectance for year 2009: Img:MODISb4QC.jpg

Run viirs2lum program

Run program viirs2lum with files created at steps 5.1, 5.2, 5.3 as summarized below:

Output root name of the experiment: this name will be used to create the output luminosity files (_lumlp_NN.pgm) will be appended, with NN giving the zone number (01 is for urban and 02 for rural).
Fichier viirs (format pgm illumina): file name of step 5.2
Fichier reflectance (format pgm illumina): file name of step 5.3
Fichier zone: file name of step 5.1

Quick file check

Open all the _lumlp_NN.pgm files to confirm that they are realistics according to the specific experiment. lumlp files are giving the light luminosity for each zone at the given wavelength.

Examples with 546 Hg line but contrast boosted

Img:zone1.jpg	Img:zone2.jpg
Zone 1 = cities	Zone 2 = countryside
Img:zone3.jpg	Img:zone4.jpg
Zone 3 = Sherbrooke	Zone 4 = 25 km around Obs. Mont-Mégantic
Img:zone6.jpg	Img:zone7.jpg
Zone 6 = Cookshire	Zone 7 = Lac Mégantic

Zone 5, wich is the Astrolab parking lot, have not been shown here because there is no Hg lamps. This image was consequently completely black.

Preparation of the batch file

This file is a bash program that will generate a lot of running experiment according to the number of desired cases for each variable.

We must define the following variables in the makeBATCH program:

Fixed variables	Definition	Example
pixsize	Pixel size in meter	1000
exp_name	Experiment name	ete
pressure	Atmospheric ground pressure in kPa	101.3
est_time	Maximal computing time limit (hours)	80
mna_file	Numerical elevation model file	srtm4.pgm

We must also define the following arrays in makeBATCH program:

Array	Definition	Example
x_sites	x coordinate of the observing site in pixel	302
y_sites	y coordinate of the observing site in pixel	58
z_sites	vertical level corresponding to the observing site elevation	33
d_reflect	Averaged distance between obstacles (m)	13
h_obstacle	Averaged obstacles height (m)	9.2
saut_dif	Skipping factor to accelerate the 2nd scattering calculation (e.g. 71 means that one computation per 71 will be made	71
r_dif	Second scattering action radius	4000
elevation	Elevation viewing angle (deg)	15 20 30 40 60 90
azimut	Azimuthal viewing angle (deg)	0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
tau	Aerosol optical depth	0.05 0.1 0.2 0.5 1.0
wav	list of wavelength	436 498 546 569 616
reflect_files	Ground reflectance file names for each wavelength	summer_436_reflect.pgm summer_498_reflect.pgm summer_546_reflect.pgm summer_569_reflect.pgm summer_616_reflect.pgm
mie_files	Aerosol optical properties files for each wavelength	rural_RH70_0.4400um.mie.out rural_RH70_0.5000um.mie.out rural_RH70_0.5500um.mie.out rural_RH70_0.5843um.mie.out rural_RH70_0.6076um.mie.out
r1	minimum distance with full resolution	25
imp	add a weight function to the probability function which is proportional to light luminosity	0.5
lamp_l	Luminosity files created by viirs2lum N.B. $exp_name"_"$wav will be added before $lamp_l	_lumlp_01.pgm _lumlp_02.pgm _lumlp_03.pgm
lamp_h	post height for each lamp/zone	megantic_altlp.pgm megantic_altlp.pgm megantic_altlp.pgm
lamp_p	Angular photometry file of each lamp/zone	cobrahead_fctem_01.dat 42Cobra3Helios.dat A210.2.dat

Note that the example given above was for summer case but the data should be define independently for each period.

Preparing a different directory for each period of the year and for each relative humidity

Create a directory for each air moisture analyzed in which two sub-directory will be included, e.g. for summer and winter. In our case, we create three directories or RH50, RH80 and RH70.

Then, we must put in each directory all the files required for the computation.

Submitting the calculations to a linux cluster

To perform the calculations, we now connect to a 'Cluster'. In our case, we connected to 'Galileo' or 'Mammouth serial II' respectively at Cégep de Sherbrooke and at Université de Sherbrooke.

Then it is necessary to recompile the ILLUMINA model using 'makeILLUMINA' or through the following command:

(:source lang=bash:) bash makeILLUMINA (:sourceend:)

The required directories created in step 7 should now be transferred to the cluster interactive node via the scp protocol.

Preparing the batch execution

Now we need to adapt the program that will run the multiple calculations (i.e. makeBATCH of makeBATCH-territory) for each experiment directory on the cluster. Copy the original file from $HOME/svn/illumina/trunk to the directories and then edit the makeBATCH file in each directory to reflect the particular situation (season or period, relative humidity). Then execute the script:

(:source lang=bash:) nohup bash makeBATCH output_batch_file & (:sourceend:)

Notes:

output_batch_file take any value. If no name is provided then makeBATCH will choose the default name (TortureMammouth).
Use the command qstat or bqmon -u to verify the status of 'clusters' (compute nodes) before or during execution.
To delete a task, use the qdel followed by the job number to delete.
To delete all your jobs use the following command:

(:source lang=bash:) list=`qstat @ms -u maube`;for i in $list; do if [ `echo $i | grep ms ` ] ; then echo $i;qdel $i;sleep 0.01; fi;done (:sourceend:)

To execute the calculations, perform the following command:

(:source lang=bash:) bash output_batch_file (:sourceend:)

This step should be repeated for each season or period and each relative humidity value.

Extracting results

ILLUMINA generates two image files by calculation, a file showing the contribution (PCL) and the other illustrating the sensitivity to light pollution (PCW). It also produces the radiance value in the calculated direction.

Now it's time to extract the data and rename files obtained since the names assigned by % blue% output_batch_file do not indicate all the required values for the files to be indexed correctly on the portal. Currently this information is listed in the directory names containing the results.

To extract the data, you have to go to each execution directory (each one must contain a gridmap directory)

create a list of files to extract. To do so go to the $HOME directory and type:

(:source lang=bash:) cat word exp_name > exp_name.list (:sourceend:)

here word is a string contained in all the execution experiment. Cop the .list file in the execution diretory and extract the data:

(:source lang=bash:) extract-output-data.bash exp_name exp_name.list (:sourceend:)

At the end of the execution of extract-output-data.bash a Results directory has been created and it will contain all PCL files (ex .: PCL-x302y58-2005-ta0 .05-wl436-el15-az0.pgm), all PCW files (ex.L PCW-x302y58-2005-ta0.05-wl436-el15-az0.pgm) and .out files (the log of each execution).

Add some color to your life

Il peut être plus facile d'analyser une carte lorsque celle-ci est en couleur. Pour ce faire, il faut d'abord savoir quel est la résolution numérique de l'image (8bit ou 16 bit) à l'aide de la commande suivante:

(:source lang=bash:) more ete_...*pgm (:sourceend:)

Si le nombre indiqué est 255, la résolution est de 8 bit.
Si le nombre indiqué est 65535, la résolution est de 16 bit.

Selon la résolution de l'image, on utilisera le programme pgm8bit2color.bash ou le programme pgm16bit2color.bash.

Donc, on entre le programme approprié suivi du nom du fichier à mettre en couleur. Par exemple:

(:source lang=bash:) pgm16bit2color.bash PCL-x302y58-2005-ta0.05-wl436-el15-az0.pgm (:sourceend:)

Renommage des fichiers pour le portail

Pour être bien indexés par le portail les fichiers doivent être nommés comme suit:

PCL-Astrolab_b_midnight_summer_2005-ta0.05-wl436-el15-az0.pgm

Signification: PCL = type de données Astrolab = nom du site d'observation b_midnight_summer_2005 = période (b_midnight est pour before midnight, période par défaut des données VIIRS-DNB) ta0.05 = épaisseur optique des aérosols de 0.05 wl436 = longueur d'onde de 436nm el15 = 15 deg d'angle d'élévation az0 = 0 deg d'angle d'azimuth

Exemple de commande pour renommer les fichiers ci-haut correctement:

(:source lang=bash:) list=`ls -1 *x302y58-2005*`;for i in $list; do o=`echo $i |sed 's/x302y58-2005/Astrolab-b_midnight_summer_2005/g'`;mv $i $o; done (:sourceend:)

Essentiellement cette commande changera x302y58-2005 pour Astrolab-b_midnight_summer_2005. I.e. avant le tiret, c'est le nom du site d'observation qui remplace la position et après on met la période de la nuit suivie de la saison et de l'année. Les tirets séparent les champs utilisés par le portail alors que les underscore servent à séparer les mots d'un champs donné. Par ex., le champ période contient 4 mots (b, midnight, summer, 2005). Notez que si la modélisation a été faite pour après minuit on utilisera les mots a et midnight (a_midnight) et si la modélisation correspond à l'hiver on utilisera le mot winter. Le nom du site est à la discrétion de l'utilisateur du présent guide mais devra ne pas porter à confusion pour les futurs usagers du portail.

IlluminaGuide2014