1. SEP: Making ATOMS toolbox development simpler
Contents
2. Abstract
Currently (as of beginning of 2016) creating a toolbox is a quite complex process for users. They need to copy a skeleton that contains many (all?) the possibilities there is for a toolbox. Users have to manually modify many scripts to insert the name of their toolboxes and remove - again manually - all the unwanted file from the toolbox_skeleton. This process is long and error prone. While there exists some workaround and tools to speed up this process this could be handled automatically through Scilab scripts for most needs from users.
The goal is to make the development of toolboxes much simpler by:
detecting the structure of a toolbox without needing to resort to many builder.sce scripts for simple cases
having a simple build command build mytoolbox
2.1. Rationale
The toolbox structure and creation process is described in howto/Create a toolbox and Scilab Module Architecture wiki pages.
Most toolboxes only use macros/ and help/ which does not really need many specialized processing.
Toolboxes needing sci_gateway/ and src/ are rarer and even though they need some specialized processing for compilation, there are many builder_something.sce scripts to modify to perform the compilation.
Even rarer are toolboxes modifying the GUI through preferences in etc/.
For all these reasons, creating toolboxes should be made simpler through automation.
The tbx_build_ and tbx_builder_ filenames are obscure:
- the profusion of separated functions are too obscure, often do very little work and could for the most part be entirely automatized;
- they will be kept for backward compatibility reasons.
The features should focus on the process of creating a toolbox and associate functions with each important step:
name |
step |
Role |
tbx_config() |
creation/setting |
name of the toolbox, creation of the necessary folders to make sure the convention is respected |
tbx_make() |
compiling/building |
generate the macros binaries, the help, compile native source code and create the ATOMS point of entry |
tbx_publish() |
packing/publishing |
create the archive of the toolbox and publish it on the atoms server |
3. Requirements
3.1. Toolbox configuration
3.1.1. Requirement 1: setting/getting the toolbox configuration
There should be a way to set rapidly the mandatory configuration elements for a toolbox. Mandatory configuration elements are used to identify the toolbox:
name: a short name for the toolbox, no space allowed
title: a long name for the toolbox
author: name of the author(s)
e-mail: e-mail of the author(s)
version: version of the toolbox, must be in the format x.y[.z]
dependencies: other toolboxes used with this toolbox
dependencies
Currently most (99%) toolbox users only use the name of the dependencies and set the version to any. I believe the use of the version is not useful at all. It may become relevant when a lot of backward compatibility issues are reported. The rule of the thumb is:
- use the most recent version of a toolbox available for your scilab version
I believe this is sufficient...
I suggest tbx_config() could implement getter/setter functions for a structure managing all configuration elements. All config elements would be recorded on a config.ini binary file containing a scilab struct with all fields.
tbx_config("name") // returns the name of the toolbox (getter)
tbx_config("name", "mytoolbox"); // sets the toolbox name in the config file
3.1.2. Requirement 2: providing the conventional layout
Toolboxes have a conventional layout identical to Scilab internal modules. This allows automatization for the compilation phase. Users should not refer to an external source for conventional layout but the tools should be provided to generate the default layout. This should be done to initialize the toolbox.
Most toolbox creators will start with macros, help (and maybe tests/unit_tests), though we could create all necessary layout, then maybe prune the empty folders at the packaging/publishing phase.
Some files will be mandatory for the distribution of the toolbox and should be added at initialization. The bare minimum should be :
a default license.txt file, toolboxes are after all made to be shared, it is sound to require a license file;
Do users really need more? If so, adding options may handle more cases. My suggestions as follows:
tbx_config("init") // creates a default layout with macros/, help/ and tests/ folder and a license.txt
tbx_config("init", "default") // same as above
tbx_config("init", "code") // adds the src/ includes/ and sci_gateway/ directories
tbx_config("init", "xcos") // adds the default mandatory xcos directories
tbx_config("init", "all") // all possible directories as taken from the skeleton
3.1.3. Requirement 3: GUI for setting fields
Instead of manually setting all the configuration fields, it could be interesting to have a GUI for the settings of the toolbox.
My suggestion here is to open a web browser page and have the ATOMS portal manage the GUI for the settings. This will avoid code duplication inside scilab. The server will have a scilab that will create the config.ini file and send it back to the user. This has many advantages:
- the server can check immediately if other toolboxes with the same name exists
- the server can give the list of versions for a toolbox
- the edition possibilities of a web based application is greater and more easily maintained than those of a local GUI
- the management of mandatory fields will be easier on the web page
The validation on the web page will lead to an upload of the config.ini file.
Local GUI
Alternately, the GUI could be entirely Scilab based but this will imply more maintenance and should not be preferred, but alleviate the need for an internet connection or a private ATOMS server
tbx_config("gui") // opens the Toolbox description
3.2. Toolbox compilation
Working once the default configuration and layout has been put in place become easier. The compilation of the toolbox follows the default layout and can be superseded with a builder file for backward compatibility.
3.2.1. Requirement 1: Macros compilation
Macros compilation is straightforward and applies genlib() to the macros folder with the library name being the name + lib where name is the name of the toolbox under config.ini. The default behaviour should be the construction of the lib xml and the .bin files from all sci files present in macros/ subfolder.
Previously a file called buildermacros.sce was required to compile the macros (and was simply doing the genlib() mentionned. For specific toolbox compilation, the builder file should be used but this should not be the default method.
Users might have requirements when there are numerous macros kept under subdirectories, an improvement could be to build the library for all Scilab macros found under macros/ tree (any depth). This could be disscussed.
tbx_make("macros") // compiles the macros
3.2.2. Requirement 2: Help generation
The help generation should be straightforward. The default behaviour should be the construction of the help with xmltojar() under each la_LA/ folder in the help/ directory:
scan the help/ directory for any language subdirectory with la_LA naming convention;
xmltojar each subfolder
There should be a possibility to specify the generation of each language independently.
tbx_make("help") // generates the help
tbx_make("help", "en_US") // generates the en_US help
3.2.3. Requirement 3: Source compilation
The default builder script under src calls the builder scripts under src/c/, src/cpp/ and src/fortran/ and src/java/ folders that know how to compile.
I suggest the default behaviour (in absence of builder script) to be:
scan the src/ directory for subdirectories
call any builder*.sce file there
tbx_make("src") // compiles the sources under each subfolders of src/
tbx_make("src", "foo/") // compiles the sources under src/foo/
Default building of the sources
The compilation of the sources is in general independent of Scilab specific libraries (such as the scilab API). Relying on other build management tools such as make, cmake and the autotools could be a solution to avoid writing much scilab script to manage the compilation. The default behaviour could become "execute the make file" or "execute the cmake then make files".
This proposal could also be a good opportunity to have a look at the behaviour of the numerous ilib_ functions.
3.2.4. Requirement 4: gateway compilation
This current behaviour is slightly trickier than with src/. The default script in toolbox skeleton executes the builder for each sci_gateway/c/, sci_gateway/cpp/ and sci_gateway/fortran/, then create a loader_gateway.sce and a cleaner_gateway.sce script.
The builder, loader and cleaner scripts can be quite specific, however there is limited interest in calling a script at the sci_gateway/ level if its sole purpose is to call the loader from subdirectories.
I suggest the default behaviour (in absence of builder scripts) to be:
scan sci_gateway/ for subdirectories;
call any builder*.sce file there.
The default loader script at the root of the toolbox should call the loader scripts from the subdirectories (by default) if there is no loader script at sci_gateway/ level
Contrary to src/ the gateways do need specific scilab libraries to compile, and loading them requires for scilab to have a linker available. This makes the usage of generic build tools less likely to be generic here.
tbx_make("sci_gateway") // compiles the sources under each subfolders of sci_gateway/
3.2.5. Requirement 5: Loader file generation
Current behaviour is to create a loader.sce and unloader.sce script that will call etc/something.start (this is the behaviour of tbx_build_loader()).
The loader.sce script is the entry point to the ATOMSLoad() function, therefore I suggest moving all default behaviour to the loader file instead of the .start file. Alternately ATOMSLoad() could be modified to scan directly for the .start file.
The unloader script calls the .quit file, default .quit does not do anything.
A requirement was suggested on the mailing list here to render the loading script less verbose ("Loading Macros... Loading sci_gateway... etc"). This should be studied in this proposal.
tbx_make("loader") // generates the loader script for the toolbox
3.2.6. Requirement 6: Global generation
The default make should generate the compilation for each aforementionned target
tbx_make() // calls tbx_make for all possible targets
3.2.7. Ideas to solve the problem
The skeleton provides an example builder file that defines the module name and title and call the builder functions in the following order:
tbx_builder_macros(toolbox_dir);
tbx_builder_src(toolbox_dir);
tbx_builder_gateway(toolbox_dir);
tbx_builder_help(toolbox_dir);
tbx_build_loader(TOOLBOX_NAME, toolbox_dir);
tbx_build_cleaner(TOOLBOX_NAME, toolbox_dir);
Each of these functions can and will fail if the directory they are probing does not exist, or if the build scripts in each directory does not exist, or if the build scripts in the directory has been changed properly with the toolbox name. This process is error prone. Besides, most of these functions merely call a builder script (with name set in the stone) under the appropriate directory, this could be automatized.
I suggest these functions should:
- if their builder file is present, execute it normally (this ensures backwards compatibility)
- if the directory is not present, do nothing
- if it their directory is present but not builder file is found, perform a default duty
tbx_make should zip all those functions in one, I suggest deprecating all the aforementioned tbx_build* functions and render them private to tbx_make():
- This reduces the documentation
- This avoids code duplication of functions only executing build scripts...
3.2.8. start and quit files
The default load and quit files must be present and must have the name of the toolbox. The one present in the toolbox skeleton loads all subdirectories and fails very easily if a toolbox has only macros, or diverge in content from the skeleton. This is annoying and time consuming.
The default load behaviour (moved to the loader) should be to call for a .start if present, and if not perform the load as in the toolbox_skeleton toolbox. I suggest doing nothing in case a subdirectory planned in the toolbox_skeleton is not present. This will allow handling most gateways without removing the possibility to customize the load.
The unloader should not do anything by default (execpt for calling a .quit file if it exists).
3.3. Toolbox Publication
The ultimate goal for a toolbox is to be shared! Scilab itself should have a way to check a toolbox made and publish its content.
3.3.1. Requirement 1: Pack the content of the toolbox
The publish function should be able to produce an archive with the content of the toolbox properly named (depending on the configuration):
- name of the toolbox
- version
tbx_publish("archive") // generates the archive to be uploaded for the toolbox
Requirements for the binaries upload is currently missing
3.3.2. Requirement 2: Test the toolbox installation
Perform a check by using using atomsLoad() on the packed archive, this should include:
- getting the dependencies;
building the sources (as per tbx_make);
loading the toolbox as per atomsLoad();
- executing the toolbox tests;
- generating a report.
This function should also be used on the server side before "releasing" the toolbox.
tbx_publish("check") // performs the check on the toolbox archive
3.3.3. Requirement 3: Publish the toolbox
As per tbx_config(), there should be a GUI to upload the packed file to the Atoms Server with last check. This should include:
- proper identification of the user to have the right to publish;
- page for the source upload already filled with
tbx_publish("upload", "myname@something.org") // opens navigator on the upload for the toolbox archive for user myname@something.org
3.4. Code Review Work In Progress
The following work is available for the exploration of the concepts mentionned above:
https://codereview.scilab.org/#/c/18071/ removes the need for default build files and beginning of a tbx_make function
https://codereview.scilab.org/#/c/17039/ generates the toolbox skeleton from scilab
Loading a toolbox really requires only one thing, a loader.sce file at the root directory of the project. The default loader file currently loads the etc/mytoolbox.start file. It seems sound again that a default loader file should be provided with the same feature as the default start file, handling the directory structure automatically.
4. Related topics
4.1. Bugzilla links
4.2. Mailling list threads
4.3. Other software
5. Examples
5.1. Creating a basic toolbox with only macros and help
Create the folder for your toolbox (in this example mytoolbox)
Create a macros/ folder and move your .sci in it;
Create the help/ folder and en_US subfolder and create your .xml help scripts
Add a license.txt file indicating the license of the toolbox
- Call the build function
tbx_config name mytoolbox
tbx_config version 0.0.1
tbx_config author "John Doe"
tbx_config email "j.doe@example.com"
tbx_config init // will generate a default simple toolbox skeleton
/* Edit your macros, help, tests */
tbx_make
This page has a Discussion page. See the menu hereabove.