Recent Changes - Search:

Menu

editer le Menu

IlluminaGuide2014

This document is obsolete

ILLUMINA USER'S GUIDE

Martin Aubé, Ph.D. copyright 2014

General information

In that document, the red text means that you have to adapt that part to your specific case.

Operating system

Illumina should be used with a computer running under linux with a fortran and gcc compilers installed (e.g. gfortran) and mercurial.

Other software dependancies

The following software are needed by the system

  • make
  • bash
  • vim-common
  • python-numpy
  • python-pyproj
  • python-matplotlib
  • python-gdal
  • gri
  • bc
  • imagemagick
  • gnuplot
  • mercurial

Installing the code

ILLUMINA model is available from bitbucket:

https://bitbucket.org/aubema/illumina

All sources codes are released under GNU public licence.

To install the model from bitbucket, please follow these steps:

If you are a non developer user:

cd
mkdir hg
cd hg
hg clone https://bitbucket.org/aubema/illumina

If you are graphycs1 developer:

cd
mkdir hg
cd hg
hg clone  https://graphycs1@bitbucket.org/aubema/illumina

Then you must compile the code:

cd illumina
bash makeILLUMINA

Then edit the $HOME/.bashrc file. This will render the executables accessible from anywere on the file system.

echo "PATH=$PATH:$HOME/hg/illumina/:$HOME/hg/illumina/bin" >>$HOME/.bashrc
cd
source .bashrc

Making light inventory for each spectral line

We have to create zones files. In ILLUMINA, zone files allow the user to define different geographical zones with different type of lamp (different by their relative contribution to the VIIRS-DNB satellite imagery, photometry function or light output pattern (LOP), lamp height). Two or more zones may be in the same geographical region or partly overimposed. Since each zone defines the lamp photometry, the user have to combine both lamp type LOP in an average LOP to define some kind of new combined lamp type. By default, as a first step, all the geographical domain will be defined by 2 zones (urban and contryside). These two cases are discriminated on the basis of a threshold value to the VIIRS-DNB relative radiance. Then additional zones are defined as circle shaped zones with center position and radius. Each new zone overwrite the previous. To create zone you have to edit a .zon file with a simple text editor like kwrite or gedit following the format shown below:

Sample .zon file for a Hg spectral line

5    14         95Cobra_5Sentinel.dat           ! Number_circular_zones [City:86%Na 14%Hg] photometry-file-name-city
0.98   34         66Cobra_34Sentinel.dat          ! city-threshold [Countryside:66%Na 34%Hg] photometry-file-name-countryside
241  50   8  10 93Cobra_7Helios.dat             ! x-pos_1 y-pos_1 r_1 [Sherbrooke(zone1):90%Na 10%Hg] photometry_file_zone1
306  58  25  11 67Helios_22Cobra_11Sentinel.dat ! x-pos_2 y-pos_2 r_2 [IDSR(zone2):89%Na 11%Hg] photometry_file_zone2
305  55   2   0 A210.2.dat                      ! x-pos_3 y-pos_3 r_3 [Astrolab's parking(zone3):LPS 0%Na 0%Hg] photometry_file_zone3
264  52   4  20 40Helios_40Cobra_20Sentinel.dat ! x-pos_4 y-pos_4 r_4 [Cookshire(zone4):80%Na 20%Hg] photometry_file_zone4
321  73   3  20 40Helios_40Cobra_20Sentinel.dat ! x-pos_5 y-pos_5 r_5 [Lac-Megantic(zone5):80%Na 20%Hg] photometry_file_zone5

First line content

This line contain 3 elements:

  1. total number of circular zones (their definition will follow in line 3 and up)
  2. percentage of VIIRS-DNB luminance coming from the given element (here Hg) in urban regions
  3. urban LOP file name

Second line

  1. urban/rural VIIRS-DNB luminance threshold (under this threshold, the pixel will be considered as rural, you can make some test with viirs2lum to validate the best threshold value to use)
  2. percentage of VIIRS-DNB luminance coming from the given element (here Hg) in rural regions
  3. rural LOP file name

Following lines

  1. x' central position of the circular zone. {#x'=x+1#} where x is determined with a wheel click on the pgm file with imagemagick (display)
  2. y' central position of the circular zone. {#y'=Nby-y #} where y is determined with a wheel click on the pgm file with imagemagick (display) and Nby is the image height in pixel.
  3. radius of the circular zone (pixel).
  4. percentage of VIIRS-DNB luminance coming from the given element (here Hg) in that zone
  5. zone LOP file name

Find a VIIRS-DNB image in ILLUMINA's standard pgm format

Sample file for southern part of Quebec province

Create a reflectance image in ILLUMINA's standard pgm format

We are using MODIS reflectance product for summer (~mid August) and winter (~mid February) at the nearest neighbourg wavelength according to the following table:

ElementWavelengthnearest MODIS band
 ±0,3 
Mercury435.5 nmband 3: 470 nm
Sodium498.0 nmband 3: 470 nm
Mercury546.0 nmband 4: 555 nm
Sodium569.0 nmband 4: 555 nm
Sodium615.5 nmband 1: 648 nm

You must choose the MODIS date to fit the VIIRS-DNB date. Example of MODIS band 4 reflectance for year 2009:

Run viirs2lum program

Run program viirs2lum with files created at steps 5.1, 5.2, 5.3 as summarized below:

  1. Output root name of the experiment: this name will be used to create the output luminosity files (_lumlp_NN.pgm) will be appended, with NN giving the zone number (01 is for urban and 02 for rural).
  2. Fichier viirs (format pgm illumina): file name of step 5.2
  3. Fichier reflectance (format pgm illumina): file name of step 5.3
  4. Fichier zone: file name of step 5.1

Quick file check

Open all the _lumlp_NN.pgm files to confirm that they are realistics according to the specific experiment. lumlp files are giving the light luminosity for each zone at the given wavelength.

Examples with 546 Hg line but contrast boosted

Zone 1 = citiesZone 2 = countryside
Zone 3 = SherbrookeZone 4 = 25 km around Obs. Mont-Mégantic
Zone 6 = CookshireZone 7 = Lac Mégantic

Zone 5, wich is the Astrolab parking lot, have not been shown here because there is no Hg lamps. This image was consequently completely black.

Preparation of the batch file

This file is a bash program that will generate a lot of running experiment according to the number of desired cases for each variable.

We must define the following variables in the makeBATCH program:

Fixed variablesDefinitionExample
pixsizePixel size in meter1000
exp_nameExperiment nameete
pressureAtmospheric ground pressure in kPa101.3
est_timeMaximal computing time limit (hours)80
mna_fileNumerical elevation model filesrtm4.pgm

We must also define the following arrays in makeBATCH program:

ArrayDefinitionExample
x_sitesx coordinate of the observing site in pixel302
y_sitesy coordinate of the observing site in pixel58
z_sitesvertical level corresponding to the observing site elevation33
d_reflectAveraged distance between obstacles (m)13
h_obstacleAveraged obstacles height (m)9.2
saut_difSkipping factor to accelerate the 2nd scattering calculation (e.g. 71 means that one computation per 71 will be made71
r_difSecond scattering action radius4000
elevationElevation viewing angle (deg)15 20 30 40 60 90
azimutAzimuthal viewing angle (deg)0 15 30 45 60 75 90 105 120 135 150 165 180 195 210 225 240 255 270 285 300 315 330 345
tauAerosol optical depth0.05 0.1 0.2 0.5 1.0
wavlist of wavelength436 498 546 569 616
reflect_filesGround reflectance file names for each wavelengthsummer_436_reflect.pgm summer_498_reflect.pgm summer_546_reflect.pgm summer_569_reflect.pgm summer_616_reflect.pgm
mie_filesAerosol optical properties files for each wavelengthrural_RH70_0.4400um.mie.out rural_RH70_0.5000um.mie.out rural_RH70_0.5500um.mie.out rural_RH70_0.5843um.mie.out rural_RH70_0.6076um.mie.out
r1minimum distance with full resolution25
impadd a weight function to the probability function which is proportional to light luminosity0.5
lamp_lLuminosity files created by viirs2lum
N.B. ⚠ $exp_name"_"$wav will be added before $lamp_l
_lumlp_01.pgm _lumlp_02.pgm _lumlp_03.pgm
lamp_hpost height for each lamp/zonemegantic_altlp.pgm megantic_altlp.pgm megantic_altlp.pgm
lamp_pAngular photometry file of each lamp/zonecobrahead_fctem_01.dat 42Cobra3Helios.dat A210.2.dat
  • Note that the example given above was for summer case but the data should be define independently for each period.

Preparing a different directory for each period of the year and for each relative humidity

Create a directory for each air moisture analyzed in which two sub-directory will be included, e.g. for summer and winter. In our case, we create three directories or RH50, RH80 and RH70.

Then, we must put in each directory all the files required for the computation.

Submitting the calculations to a linux cluster

To perform the calculations, we now connect to a 'Cluster'. In our case, we connected to 'Galileo' or 'Mammouth serial II' respectively at Cégep de Sherbrooke and at Université de Sherbrooke.

Then it is necessary to recompile the ILLUMINA model using 'makeILLUMINA' or through the following command:

bash makeILLUMINA

The required directories created in step 7 should now be transferred to the cluster interactive node via the scp protocol.

Preparing the batch execution

Now we need to adapt the program that will run the multiple calculations (i.e. makeBATCH of makeBATCH-territory) for each experiment directory on the cluster. Copy the original file from $HOME/svn/illumina/trunk to the directories and then edit the makeBATCH file in each directory to reflect the particular situation (season or period, relative humidity). Then execute the script:

nohup bash makeBATCH %blue%output_batch_file &  

Notes:

  • output_batch_file take any value. If no name is provided then makeBATCH will choose the default name (TortureMammouth).
  • Use the command qstat or bqmon -u to verify the status of 'clusters' (compute nodes) before or during execution.
  • To delete a task, use the qdel followed by the job number to delete.
  • To delete all your jobs use the following command:
list=`qstat @ms -u maube`;for i in $list; do if [ `echo $i | grep ms ` ] ; then echo $i;qdel $i;sleep 0.01; fi;done

To execute the calculations, perform the following command:

bash  output_batch_file

This step should be repeated for each season or period and each relative humidity value.

Extracting results

ILLUMINA generates two image files by calculation, a file showing the contribution (PCL) and the other illustrating the sensitivity to light pollution (PCW). It also produces the radiance value in the calculated direction.

Now it's time to extract the data and rename files obtained since the names assigned by % blue% output_batch_file do not indicate all the required values for the files to be indexed correctly on the portal. Currently this information is listed in the directory names containing the results.

To extract the data, you have to go to each execution directory (each one must contain a gridmap directory)

create a list of files to extract. To do so go to the $HOME directory and type:

cat ''word'' ''exp_name'' > ''exp_name''.list

here word is a string contained in all the execution experiment. Cop the .list file in the execution diretory and extract the data:

extract-output-data.bash exp_name exp_name.list

At the end of the execution of extract-output-data.bash a Results directory has been created and it will contain all PCL files (ex .: PCL-x302y58-2005-ta0 .05-wl436-el15-az0.pgm), all PCW files (ex.L PCW-x302y58-2005-ta0.05-wl436-el15-az0.pgm) and .out files (the log of each execution).

Add some color to your life

Il peut être plus facile d'analyser une carte lorsque celle-ci est en couleur. Pour ce faire, il faut d'abord savoir quel est la résolution numérique de l'image (8bit ou 16 bit) à l'aide de la commande suivante:

more ete_...*pgm
  • Si le nombre indiqué est 255, la résolution est de 8 bit.
  • Si le nombre indiqué est 65535, la résolution est de 16 bit.

Selon la résolution de l'image, on utilisera le programme pgm8bit2color.bash ou le programme pgm16bit2color.bash.

Donc, on entre le programme approprié suivi du nom du fichier à mettre en couleur. Par exemple:

pgm16bit2color.bash PCL-x302y58-2005-ta0.05-wl436-el15-az0.pgm

Renommage des fichiers pour le portail

Pour être bien indexés par le portail les fichiers doivent être nommés comme suit:

PCL-Astrolab_b_midnight_summer_2005-ta0.05-wl436-el15-az0.pgm

Signification: PCL = type de données Astrolab = nom du site d'observation b_midnight_summer_2005 = période (b_midnight est pour before midnight, période par défaut des données VIIRS-DNB) ta0.05 = épaisseur optique des aérosols de 0.05 wl436 = longueur d'onde de 436nm el15 = 15 deg d'angle d'élévation az0 = 0 deg d'angle d'azimuth

Exemple de commande pour renommer les fichiers ci-haut correctement:

list=`ls -1 *x302y58-2005*`;for i in $list; do o=`echo $i |sed 's/x302y58-2005/Astrolab-b_midnight_summer_2005/g'`;mv $i $o; done

Essentiellement cette commande changera x302y58-2005 pour Astrolab-b_midnight_summer_2005. I.e. avant le tiret, c'est le nom du site d'observation qui remplace la position et après on met la période de la nuit suivie de la saison et de l'année. Les tirets séparent les champs utilisés par le portail alors que les underscore servent à séparer les mots d'un champs donné. Par ex., le champ période contient 4 mots (b, midnight, summer, 2005). Notez que si la modélisation a été faite pour après minuit on utilisera les mots a et midnight (a_midnight) et si la modélisation correspond à l'hiver on utilisera le mot winter. Le nom du site est à la discrétion de l'utilisateur du présent guide mais devra ne pas porter à confusion pour les futurs usagers du portail.

Quick using guide for mercurial

To publish your locally modified codes on the server

hg commit -u graphycs1 -m "nouveau printout"
hg push

To update your local version of the codes

hg pull
hg update
Edit - History - Print - Recent Changes - Search
Page last modified on March 30, 2017, at 08:04 pm UTC