Menu |
Prof /
IlluminaGuidev22ILLUMINA v2.2 USER'S GUIDEMartin Aubé, Ph.D. Alexandre Simoneau M.Sc. & Hector Linares Arroyo Ph.D. copyright 2022 Latest update: October 28 2024 NewsNew features
Bug fixes
General informationsThis users guide aim to help users of the ILLUMINA sky radiance model to prepare and manage their own simulations. We hope that the document is accurate enough but will be happy to improve it according to some difficulties you may encounter when using it. For any help please contact the PI Dr Martin Aubé (martin.aube at cegepsherbrooke.qc.ca). This guide is the most recent one that incorporate recently added features to the model like the hyperspectral support, the subgrid obstacles filling factor and the contribution of the cloud reflexion. Optimal wavelength rangeILLUMINA can be used at any wavelength between 250nm and 4 000nm. This limitation is mainly due to the fact that the aerosol optical properties are only defined on that spectral range. The molecular absorption features are defined from 200nm to 25 000nm. But most lighting system emits in the 330nm to 730nm range. One may want to extent up to 900nm in order to incluce the near infrared line of the high pressure sodium. This is not a problem at all for the model. Note also that some of the provided reflectance spectra are not defined below 420nm (e.g. asphalt) and then in such a situation, we will assume the nearest neighbour method to determine the reflectance for wavelength lower than that. All provided reflectance spectra are not defined above 14 000nm. Similarly for the provided lamp spectra are defined from 273nm to 900nm. Of course the provided reflectance and lamp spectra may be changed by the user to fit the maximum wavelength range of the model (250nm to 4 000nm) Prepared by Robert A. Rohde for the Global Warming Art project. InstallationOperating systemILLUMINA should be used with a computer running under Linux with Fortran and gcc compilers (e.g. gfortran) and Python (3.8) with the pip versioning system installed. The easiest path to get into illumina is to use of a virtual environment to manage the Python libraries. For a debian based system you can do it that way. sudo apt-get install python3-venv python3 -m venv $HOME/illum We assume here that the virtual environment will be in $HOME/illum. You can change that path to whatever you want but in the following steps we will assume you have done it that way. Activate the virtual environment source $HOME/illum/bin/activate Other software dependenciesThe following software are needed by the system
We also suggest installing some Python libraries that are not needed but are useful.
In all cases, the most recent version of the code should be used. The code is evolving rapidly and then by updating your version frequently, you will benefit of new features and bug fixes. Installing the codeThe ILLUMINA model is available from github: https://github.com/aubema/illumina All sources codes are released under GNU public license. To download the model from github, please follow these steps: cd mkdir git cd git git clone https://github.com/aubema/illumina.git For developers, you must also execute this command: git config core.hooksPath "./.git_hooks" Then you must compile the code: cd $HOME/git/illumina bash ./bin/makeILLUMINA Then modify the $HOME/.bashrc file by typing the following commands in the terminal window. This will make the programs executable from anywhere on the file system. This command needs to be executed from outside the virtual environment if you are using it. echo "PATH=$PATH:$HOME/.local/bin:$HOME/git/illumina/bin" >> $HOME/.bashrc Finally, the model is installed using: pip3 install -e $HOME/git/illumina The user interfaceThe model's user interface is done through the command line in a terminal. The main program name is Entering Preparing an executionIn order to execute the model, some data manipulation is needed to prepare it for the model. It is strongly recommended to separate the data from the code by creating a new directory somewhere on your computer and placing all the relevant data within. When it is done, enter this directory and execute If you re-run that script in the same directory, the files domain_params.in and inputs_params.in will be overwritten. If you do not want to lose them, backup these files first. Downloading and preparing the required satellite imagesILLUMINA requires some satellite data to run properly, namely a digital elevation model and the nocturnal light upward radiance. Theses data also need to be projected in a suitable spatial reference system and clipped to the simulation domain. Domain definitionDefining the simulation domain is crucial in the input preparation, as it directly affects everything afterwards. The first step is to define the location(s) where the simulation of the artificial sky radiance is desired. Then, the projection needs to be defined, as the model need to work with coordinates in meters instead of degrees. The parameters file named domain_params.in defines the domain. It should contain the following parameters: latitude: 20.708 longitude: -156.257 srs: auto nb_layers: 3 nb_pixels: 27 scale_min: 1000 scale_factor: 3 buffer: 10
The (1) Note that (2) As a rule of thumb, we suggest not to exceed 255 for that number. VIIRS-DNB imageryThe night emittance is obtained from VIIRS-DNB imagery that can be found here. One should download the appropriate tile(s) for the period desired (year and month). We suggest the VCMSLCFG configuration in the monthly composite product because of the stray light correction, but the choice is left to the user. You will want to extract the 'avg_rade9.tif' file as it's the actual values, whereas the 'cf_cvg.tif' file contain information related to the quality of the image. The tif file(s) should be placed inside a subfolder named VIIRS-DNB inside your experiment directory. It is also possible to use the VIIRS background method proposed by Coesfeld et al. (2020) for more accurate results. In that case, the VCMCFG product needs to be used instead, and the correction data needs to be downloaded from here and decompressed in the VIIRS-DNB folder. WatermaskWhen used with VIIRS-DNB input, the model need a water mask to calculate properly the light fluxes. You can download it here and save it to the experiment folder. SRTM dataThe digital elevation model is made with the SRTM elevation data that can be found here. One should use the spatial filter to select only the required tiles, and then follow the download procedure. The extracted hgt files should be placed inside a subfolder named SRTM inside your experiment directory. To extract multiple archives at once, one can use Processing the input imagesThe Two files should be produced by this command:
Sample files for Hawaii The standard format used by ILLUMINA is HDF version 5. Theses can be visualized using tools like hdfcompass. We also provide convenience Python functions in the Converting other datasets
illum warp NAME FILELIST where NAME will be the name of the output file (without the extension) and file list is a list of one or multiple files (the use of bash wildcards is recommended here) to be warped. Note that all the files will be warped togheter so they should be tiles of the same dataset. The supported formats are the ones that can be processed by GDAL. Making light inventoryIn order to model the propagation of the light, the properties of the light sources must be defined. There are two way to do this for ILLUMNA: 1- using VIIRS-DNB spaceborne radiance monthly product, 2- using a point source inventory. Both of them can be used together, as long as they are not overlapping. There can not be poing sources where in a pixel already containing sources derived with VIIRS-DNB. Using uniform overlapping circular zonesThe first way is to define overlapping circular zones of uniform properties. Theses zones are defined by a their center point and a radius and specify the mix of lamps assumed to be present in that area (different by their photometry function or light output pattern (LOP), their spectrum, lamp height) as well as the average distance between obstacles, obstacles height and obstacle filling factor. Two or more zones may be in the same geographical region or partly overlapping. Each new zone overwrite the previous in case of intersection between the zones. All the points that are not included in a zone will be ignored. To create a zone you have to edit an ASCII file with a simple text editor like kwrite or gedit following the format shown below: Sample inventory file for the Hawaii territory# lat lon R hobs dobs fobs hlamp Zone inventory Comment 21.4474 -157.9712 50 7 25 0.5 7 90_H_5 10_M_10 # Oahu 21.0052 -157.0123 40 7 25 0.5 7 90_H_5 10_M_10 # Molokai + Lanai 20.7764 -156.1512 64 7 25 0.5 7 18_H_10 72_H_0 10_M_10 # Maui 19.6468 -155.5714 103 7 25 0.5 7 87_L_10 8_H_10 5_M_5 # Big Island 19.2878 -155.2179 23 7 25 0.5 7 0_L_0 # Lava This file can have any number of header lines as long as they are preceded by a '#' symbol. Anything on the same line following this symbol will not be considered by the model. Each data line contains several parameters:
Each lamp characteristics is composed of three fields separated by '_'.
The weighting is applied on the luminous flux of the spectral power distribution of the lamp in lumen. This means that the spectra are weighted by the photopic sensitivity curve.
Example If one zone is composed of 50% of HPS with the angular photometry toto1_fctem.dat, 20% of HPS with phtotometry file toto2_fctem.dat, and 30% of LED4000K with photometry file toto3_fctem.dat and let assume that you use the spectral power distributions provided in the Example/Lights directory HPS_Helios.spct and 4LED_LED-4000K-Philips.spct to create the light inventory. Then you should write 50_HPS_toto1 20_HPS_toto2 30_4LED_toto3 at column 8 of your inventory file for that zone. The data referenced by the last two fields (spectral power distribution and angular power distribution) must be located in a subfolder named 'Lights'. This folder must contain the following files in addition to the ones used to define the lamp inventory :
Theses files can be found in the ILLUMINA installation directory (Examples/Lights). Any additional file used to characterize the lamp must follow the following format :
The normalization of all theses files is not important, as it will be done by the programs. (3)In all cases, any characters preceding the first underscore (_) in the lop or spct file name is the reference word that must be written in the inventory file. All .spct files must have the same wavelength scale. All LOP Files must share the same angle scale. Using a discrete light source inventoryThe second way to describe the lights is to directly specify their properties on a lamp-by-lamb basis. In this case, the file needs to have the following format: # lat lon pow hobs dobs fobs hlamp spct lop 21.295693 -157.856846 2500 20 25 0.9 7 MH 5 21.295776 -157.856782 1500 20 25 0.9 7 LED 0 21.295844 -157.857114 1000 30 30 0.85 7 MH 5 21.286488 -157.845900 1000 50 10 0.3 10 LED 1 where
It is possible to use both methods simultaneously, but in that case all the discrete light sources must fall outside of the zones or inside one with a weight of 0. Defining the experimentThe execution modeYou may be interested to run ILLUMINA for many reasons. By default, ILLUMINA will calculate the artificial diffuse radiance, the part that is produced by the clouds, the direct radiance reaching the observer from a sight to the sources and the direct radiance coming from a sight to reflecting surfaces. If you are more interested by the direct radiance, it may be a good idea to increase to the maximum the resolution near the observer. The calculation of the direct radiance inside the mean free path toward obstacles will not experience any obstacle blocking. The blocking by obstacles only occur when the observer is farther than the mean free path to the ground. This parameter is defined when you specified the "subgrid obstacle properties" with the variable dobs. Actually dobs is twice the value of the mean free path. If you are not interested to obtain the sky or cloud radiance, but only the direct radiance, then you can speedup the calculation by setting off the scattering. Create the input parameters fileThe parameters used by the model for executing the experiment are contained in the inputs_params.in file, as described below: # input parameters exp_name: Chile zones_inventory: inventory_zones.txt lamps_inventory: inventory_INNA_permanent.txt nb_bins: 9 lambda_min: 400 # [nm] lambda_max: 900 # [nm] reflectance: asphalt: 0.8 grass: 0.2 # THIS IS THE TYPICAL SOIL IN THE AREA WHICH WOULD BE PRESENT INBETWEEN THE ASPHALT ROADS. 80/20. RATIO IS A PURE GUESS FOR THE TIME BEING aerosol_profile: D # Aerosol profile. 'CC','CA','CP','U','D'.'MC','MP','ART','ANT','Manual'. layer_type: D # Additional layer. Same choices as aerosol_profile. Set 'layer_aod' to 0 to disable. relative_humidity: 70 # THIS WOULD BE AT THE HARBOUR FACILITIES AT SEA LEVEL.. 60% IS A TYPICAL VALIE ACCORDING TO METEO DATA estimated_computing_time: 10 # estimated computing time per case [h] batch_file_name: batch road_orientation: False # The final azimuth angles will be determined with azimuth_angle (see below) relative to the direction between the observer and the nearest point of the nearest road instead of relative to the north. (WARNING: Can be slow for large domains) observer_obstacles: False # parameters after here can be lists observer_elevation: 20 # elevation above ground level [m] 15m for VLT at PARANAL, 30 m for ELT at ARMAZONES, between 5 and 30 m for different CTA telescopes. Can we use 20 m as typical value air_pressure: 101.3 # lowest domain level atmospheric pressure [kPa] - THAT WOULD BE AT SEA LEVEL. THEREFORE 1013 mbar SHOULD BE OK reflection_radius: 9.99 # radius around light sources where reflections can occur (m) cloud_model: 0 # cloud model selection 0=clear, 1=Thin Cirrus/Cirrostratus, 2=Thick Cirrus/Cirrostratus, 3=Altostratus/Altocumulus, 4=Stratocumulus, 5=Cumulus/Cumulonimbus cloud_base: 0 # height of the cloud base [m] cloud_fraction: 0 # Cloud cover fraction (0-100) stop_limit: 5000. # Stop computation when the new voxel contribution is less than 1/stoplim of the cumulated flux (suggested value = 5000.) single_scattering: True # Activate single scattering (True/False) double_scattering: True # Activate double scattering (True/False) elevation_angle: [90,45] # 45 and 90 at 6 azimuth angles would be a nice grid to sample. azimuth_angle: [0,60,120,180,240,300] direct_fov: 5 # Field of view for the direct radiance calculations [deg] aerosol_optical_depth: 0.101 # AOD value at 500 nm THERE IS AERONET STATION IN PAPOSO WHICH IS A GOOD REPRESENTATION OF THE LOWEST ELEVATION: https://aeronet.gsfc.nasa.gov/cgi-bin/data_display_aod_v3?site=Paposo&nachal=2&level=1&place_code=10 angstrom_coefficient: 1.0 # angstrom exponent value aerosol_height: 2000 # Aerosol scale height [m] layer_aod: 0. # Layer's AOD value at 500 nm layer_alpha: 1.0 # Layer's angstrom exponent value layer_height: 2000 # Layer's scale height [m] Illumina allows to create 2 different layers of aerosols with different properties: compositions, scale height, AOD and angstrom coefficient. The aerosol types that Illumina includes are extracted from OPAC data (Hess, Koepke and Schult 1998). Aerosol types:
Illumina allows the user to combine any component using their Particle density (particles/cm³). Moreover, there are some typical combination that are already defined:
The reflective surface types are ASTER (ECOSTRESS spectral library) files located in the `Lights` folder. We remove the header and keeping only the spectral data of the files. Notes:
Illumina allows to create 2 different layers of aerosols with different properties: compositions, scale height, AOD and angstrom coefficient. The aerosol types that Illumina includes are extracted from OPAC data (Hess, Koepke and Schult 1998. See http://cds-espri.ipsl.fr/etherTypo/?id=989&fbclid=IwAR0Ome7uT9JDK3OlfgwV3Psyja4BZIxedNtojLT1-twEgc23SOEZ1bUKSac). Aerosol types:
Illumina allows the user to combine any component using their Particle density (particles/cm³). Moreover, there are some typical combination that are already defined:
The aerosol components are mixed in order to obtain the resulting Single Scattering albedo and Scattering Phase Function. The equations used are the following: (1) SSA = SUM(N_i*scat_coef_i)/SUM(N_i*ext_coef_i) (2) PF(angle) = SUM(N_i*PF_i(angle)) (normalized) Prepare the inputsOnce all the data is obtained as the input parameter file is created, the Quick file checkThe command produces a directory named 'Inputs'. It should contain:
lumlp files are in units of W/nm. Example run at 605nm with contrast boosted
Zone 5, which is the lava lakes, is all black because its light isn't considered in the model. To achieve this, its lamp inventory was empty. You can verify that in the 'inventory.txt' file. Alternative scenariosYou may be interested in simulating alternative scenarios based on the current situation. For example, artificially replacing all light sources to a new photometry. This is done with the illum alternate --help If used with an alternative zones inventory, a replacement inventory needs to be in the same directory and contain a set of lamp characteristics for each zone. For example, 1_AMBR_0 # Oahu 1_AMBR_0 # Molokai + Lanai 1_AMBR_0 # Maui 1_AMBR_0 # Big Island 0_L_0 # Lava If used with an alternative lamps inventory, a replacement inventory needs to be in the same directory and contain a set of lamp characteristics in the same format as the initial inventory. The command will then generate a folder named Inputs_NAME containing the relevant data. Submitting the calculations to a Linux clusterTo perform the calculations, we now connect to a 'Cluster'. In our case, we connected to 'Mammouth serial II' located at Université de Sherbrooke. The task scheduler used is Slurm. You may need to manually adjust some files to match the execution environment you are using. Then it is necessary to recompile the ILLUMINA model using 'makeILLUMINA' or through the following command: bash makeILLUMINA The `Inputs` directory created for each experiment in step 6 should now be transferred to the cluster interactive node via the scp protocol. Preparing the batch executionNow we need to execute the On a Slurm cluster, you may use theses commands to keep an eye on and manage the executions.
To execute the calculations, simply execute the bash file(s) produced by the Find failed calculationsIn many cases you will probably have a lot of calculations to be done to complete your modeling experiment. Each calculation going to a given core and/or node (if you run on a cluster). Then for some reasons there is some chance that some of your calculations can fail. Finding the failed calculations can be a difficult task. For that reason we provided a command called illum failed -e > your_final_run.bash Then simply start the aborted runs by running this script bash ./your_final_run.bash Note that the script is assuming that you are using a system running slurm. You will see in the script that the execution begin by sbatch. If you are not using slurm, then just remove «sbatch --time=XX:XX:XX» from the script. In such a case you will also probably need split the file into many execution script to be sure that you will not use too much RAM memory. You can use the unit split command for that. Extracting resultsILLUMINA generates two different output per calculation:
To extract the data, the illum extract --help The command will output the data directly, so it should be redirected to a file with illum extract > results.txt If you are not only interested in the total diffuse radiance (clouds + atmosphere) and want also to extract the cloud contribution to the radiance and the direct and direct reflected radiance, you will need to run the command in the full mode. The command will output the data directly, so it should be redirected to a file with illum extract -f > results.txt There will be a column for each radiometric value. As stated in the documentation, the contribution map can be extracted using the illum extract -c > results.txt Units of the radiances are W/sr/m^2/nm. To get the radiance of a spectral bin, one must multiply the radiance delivered by Illumina with the bandwidth (in nm). Units of the irradiance are W/m^2/nm To get the irradiance of a spectral bin, one must multiply the irradiance delivered by Illumina with the bandwidth (in nm). PCL binary files (XXXXX_pcl.bin) do not have any units. The values represents the fractional contribution of a pixel to the total diffuse radiance. The sum of all pixels gives 1. PCL files at different resolution are combined into a HDF5 file to create the total diffuse radiance contribution file in units of W/sr/m^2/nm. These files shoud be named the following way: elevation_angle_XX-azimuth_angle_YY-wavelength_ZZZ.Z.hdf5 Analyzing the resultsThe analysis can be done with your favorite tools. We strongly recommend the use of python and provide convenience functions in the pytools and MultiScaleData packages provided with ILLUMINA. Transforming to magnitude per arc sq seconds (for astronomers...)Transforming diffuse radiance to sky brightnes (SB) in units of mag/sqarcsec is not an simple task. First of all you have to consider that illumina is only dealing with the artificial component of the SB. If you are using illumina in a relatively dark site, the artificial SB can represents only a small part of the total SB. To transform radiance to total SB, you will need a relevant estimate of the natural contribution to the total SB. The natural SB is highly variable with time, altitude, season, observing direction etc. It is composed of many sources like the zodiacal light, the starlight, the sky glow, the Milky Way etc. Given that complexity, we suggest to determine it experimentally for the modeled site and period you are interested in. To do it, you need an in situ measurement of the total SB from which you will be able to extract the natural component and then eventually consider this component as a constant natural contribution to the SB for your specific site and period, no matter the viewing angle or light inventory, obstacles properties etc. Lets call the radiance responsible for that natural contribution the background radiance ({#R_{bg}#}). Lets assume that you have an in situ measurement of the total Johnson-Cousins SB. You need to accomplish the following steps to convert your artificial modeled radiance to total SB. Integrate your radiance {#{R_a}#} and according to the Johnson-Cousins filter. {#{R_a}#} being the modeled radiance you want to convert to {# SB #}. The sensitivity curve of Johnson-Cousins filters are provided in the Example/Lights folder (e.g. JC_V.dat). {#R_{bg}#} the radiance corresponding to the natural level of the sky brightness ({#SB_{bg}#}) can be estimated using the SB data provided by Benn & Elison 1998, except for the R band that was taken from La Palma P99 ASTMON (2018-2019) measurements. We simply added 0.03 from the B and V values provided for La Palma (as recommended by Benn & Elison 1998) and then used the formula below to calculate the radiance.
{## R_{bg} = R_0 10^{-0.4 SB_{bg}} ##} For a value of modeled artificial radiance ({#{R_a}#}), use the following formulae to convert to total SB: {## SB = -2.5 log10 \left( \frac{{R_a} + R_{bg}}{R_0} \right) ##} {#R_0#} are derived from zero points obtained by Bessell 1979 calibration (DOI 10.1086/130542) and given in the table below.
|