[Contents] [TitleIndex] [WordIndex

1. SEP: Making ATOMS toolbox development simpler

2. Abstract

Currently (as of beginning of 2016) creating a toolbox is a quite complex process for users. They need to copy a skeleton that contains many (all?) the possibilities there is for a toolbox. Users have to manually modify many scripts to insert the name of their toolboxes and remove - again manually - all the unwanted file from the toolbox_skeleton. This process is long and error prone. While there exists some workaround and tools to speed up this process this could be handled automatically through Scilab scripts for most needs from users.

The goal is to make the development of toolboxes much simpler by:

2.1. Rationale

The toolbox structure and creation process is described in howto/Create a toolbox and Scilab Module Architecture wiki pages.

Most toolboxes only use macros/ and help/ which does not really need many specialized processing.

Toolboxes needing sci_gateway/ and src/ are rarer and even though they need some specialized processing for compilation, there are many builder_something.sce scripts to modify to perform the compilation.

Even rarer are toolboxes modifying the GUI through preferences in etc/.

For all these reasons, creating toolboxes should be made simpler through automation.

The tbx_build_ and tbx_builder_ filenames are obscure:

The features should focus on the process of creating a toolbox and associate functions with each important step:

name

step

Role

tbx_config()

creation/setting

name of the toolbox, creation of the necessary folders to make sure the convention is respected

tbx_make()

compiling/building

generate the macros binaries, the help, compile native source code and create the ATOMS point of entry

tbx_publish()

packing/publishing

create the archive of the toolbox and publish it on the atoms server

3. Requirements

3.1. Toolbox configuration

3.1.1. Requirement 1: setting/getting the toolbox configuration

There should be a way to set rapidly the mandatory configuration elements for a toolbox. Mandatory configuration elements are used to identify the toolbox:

dependencies

Currently most (99%) toolbox users only use the name of the dependencies and set the version to any. I believe the use of the version is not useful at all. It may become relevant when a lot of backward compatibility issues are reported. The rule of the thumb is:

  • use the most recent version of a toolbox available for your scilab version

I believe this is sufficient...

I suggest tbx_config() could implement getter/setter functions for a structure managing all configuration elements. All config elements would be recorded on a config.ini binary file containing a scilab struct with all fields.

tbx_config("name") // returns the name of the toolbox (getter)
tbx_config("name", "mytoolbox"); // sets the toolbox name in the config file

3.1.2. Requirement 2: providing the conventional layout

Toolboxes have a conventional layout identical to Scilab internal modules. This allows automatization for the compilation phase. Users should not refer to an external source for conventional layout but the tools should be provided to generate the default layout. This should be done to initialize the toolbox.

Most toolbox creators will start with macros, help (and maybe tests/unit_tests), though we could create all necessary layout, then maybe prune the empty folders at the packaging/publishing phase.

Some files will be mandatory for the distribution of the toolbox and should be added at initialization. The bare minimum should be :

Do users really need more? If so, adding options may handle more cases. My suggestions as follows:

tbx_config("init") // creates a default layout with macros/, help/ and tests/ folder and a license.txt
tbx_config("init", "default") // same as above
tbx_config("init", "code") // adds the src/ includes/ and sci_gateway/ directories
tbx_config("init", "xcos") // adds the default mandatory xcos directories
tbx_config("init", "all") // all possible directories as taken from the skeleton

3.1.3. Requirement 3: GUI for setting fields

Instead of manually setting all the configuration fields, it could be interesting to have a GUI for the settings of the toolbox.

My suggestion here is to open a web browser page and have the ATOMS portal manage the GUI for the settings. This will avoid code duplication inside scilab. The server will have a scilab that will create the config.ini file and send it back to the user. This has many advantages:

The validation on the web page will lead to an upload of the config.ini file.

Local GUI

Alternately, the GUI could be entirely Scilab based but this will imply more maintenance and should not be preferred, but alleviate the need for an internet connection or a private ATOMS server

tbx_config("gui") // opens the Toolbox description

3.2. Toolbox compilation

Working once the default configuration and layout has been put in place become easier. The compilation of the toolbox follows the default layout and can be superseded with a builder file for backward compatibility.

3.2.1. Requirement 1: Macros compilation

Macros compilation is straightforward and applies genlib() to the macros folder with the library name being the name + lib where name is the name of the toolbox under config.ini. The default behaviour should be the construction of the lib xml and the .bin files from all sci files present in macros/ subfolder.

Previously a file called buildermacros.sce was required to compile the macros (and was simply doing the genlib() mentionned. For specific toolbox compilation, the builder file should be used but this should not be the default method.

Users might have requirements when there are numerous macros kept under subdirectories, an improvement could be to build the library for all Scilab macros found under macros/ tree (any depth). This could be disscussed.

tbx_make("macros") // compiles the macros

3.2.2. Requirement 2: Help generation

The help generation should be straightforward. The default behaviour should be the construction of the help with xmltojar() under each la_LA/ folder in the help/ directory:

There should be a possibility to specify the generation of each language independently.

tbx_make("help") // generates the help
tbx_make("help", "en_US") // generates the en_US help

3.2.3. Requirement 3: Source compilation

The default builder script under src calls the builder scripts under src/c/, src/cpp/ and src/fortran/ and src/java/ folders that know how to compile.

I suggest the default behaviour (in absence of builder script) to be:

tbx_make("src") // compiles the sources under each subfolders of src/
tbx_make("src", "foo/") // compiles the sources under src/foo/

Default building of the sources

The compilation of the sources is in general independent of Scilab specific libraries (such as the scilab API). Relying on other build management tools such as make, cmake and the autotools could be a solution to avoid writing much scilab script to manage the compilation. The default behaviour could become "execute the make file" or "execute the cmake then make files".

This proposal could also be a good opportunity to have a look at the behaviour of the numerous ilib_ functions.

3.2.4. Requirement 4: gateway compilation

This current behaviour is slightly trickier than with src/. The default script in toolbox skeleton executes the builder for each sci_gateway/c/, sci_gateway/cpp/ and sci_gateway/fortran/, then create a loader_gateway.sce and a cleaner_gateway.sce script.

The builder, loader and cleaner scripts can be quite specific, however there is limited interest in calling a script at the sci_gateway/ level if its sole purpose is to call the loader from subdirectories.

I suggest the default behaviour (in absence of builder scripts) to be:

The default loader script at the root of the toolbox should call the loader scripts from the subdirectories (by default) if there is no loader script at sci_gateway/ level

Contrary to src/ the gateways do need specific scilab libraries to compile, and loading them requires for scilab to have a linker available. This makes the usage of generic build tools less likely to be generic here.

tbx_make("sci_gateway") // compiles the sources under each subfolders of sci_gateway/

3.2.5. Requirement 5: Loader file generation

Current behaviour is to create a loader.sce and unloader.sce script that will call etc/something.start (this is the behaviour of tbx_build_loader()).

The loader.sce script is the entry point to the ATOMSLoad() function, therefore I suggest moving all default behaviour to the loader file instead of the .start file. Alternately ATOMSLoad() could be modified to scan directly for the .start file.

The unloader script calls the .quit file, default .quit does not do anything.

A requirement was suggested on the mailing list here to render the loading script less verbose ("Loading Macros... Loading sci_gateway... etc"). This should be studied in this proposal.

tbx_make("loader") // generates the loader script for the toolbox

3.2.6. Requirement 6: Global generation

The default make should generate the compilation for each aforementionned target

tbx_make() // calls tbx_make for all possible targets

3.2.7. Ideas to solve the problem

The skeleton provides an example builder file that defines the module name and title and call the builder functions in the following order:

tbx_builder_macros(toolbox_dir);
tbx_builder_src(toolbox_dir);
tbx_builder_gateway(toolbox_dir);
tbx_builder_help(toolbox_dir);
tbx_build_loader(TOOLBOX_NAME, toolbox_dir);
tbx_build_cleaner(TOOLBOX_NAME, toolbox_dir);

Each of these functions can and will fail if the directory they are probing does not exist, or if the build scripts in each directory does not exist, or if the build scripts in the directory has been changed properly with the toolbox name. This process is error prone. Besides, most of these functions merely call a builder script (with name set in the stone) under the appropriate directory, this could be automatized.

I suggest these functions should:

tbx_make should zip all those functions in one, I suggest deprecating all the aforementioned tbx_build* functions and render them private to tbx_make():

3.2.8. start and quit files

The default load and quit files must be present and must have the name of the toolbox. The one present in the toolbox skeleton loads all subdirectories and fails very easily if a toolbox has only macros, or diverge in content from the skeleton. This is annoying and time consuming.

The default load behaviour (moved to the loader) should be to call for a .start if present, and if not perform the load as in the toolbox_skeleton toolbox. I suggest doing nothing in case a subdirectory planned in the toolbox_skeleton is not present. This will allow handling most gateways without removing the possibility to customize the load.

The unloader should not do anything by default (execpt for calling a .quit file if it exists).

3.3. Toolbox Publication

The ultimate goal for a toolbox is to be shared! Scilab itself should have a way to check a toolbox made and publish its content.

3.3.1. Requirement 1: Pack the content of the toolbox

The publish function should be able to produce an archive with the content of the toolbox properly named (depending on the configuration):

tbx_publish("archive") // generates the archive to be uploaded for the toolbox

Requirements for the binaries upload is currently missing

3.3.2. Requirement 2: Test the toolbox installation

Perform a check by using using atomsLoad() on the packed archive, this should include:

This function should also be used on the server side before "releasing" the toolbox.

tbx_publish("check") // performs the check on the toolbox archive

3.3.3. Requirement 3: Publish the toolbox

As per tbx_config(), there should be a GUI to upload the packed file to the Atoms Server with last check. This should include:

tbx_publish("upload", "myname@something.org") // opens navigator on the upload for the toolbox archive for user myname@something.org

3.4. Code Review Work In Progress

The following work is available for the exploration of the concepts mentionned above:

Loading a toolbox really requires only one thing, a loader.sce file at the root directory of the project. The default loader file currently loads the etc/mytoolbox.start file. It seems sound again that a default loader file should be provided with the same feature as the default start file, handling the directory structure automatically.

4. Related topics

4.2. Mailling list threads

4.3. Other software

5. Examples

5.1. Creating a basic toolbox with only macros and help

  1. Create the folder for your toolbox (in this example mytoolbox)

  2. Create a macros/ folder and move your .sci in it;

  3. Create the help/ folder and en_US subfolder and create your .xml help scripts

  4. Add a license.txt file indicating the license of the toolbox

  5. Call the build function

tbx_config name mytoolbox
tbx_config version 0.0.1
tbx_config author "John Doe"
tbx_config email "j.doe@example.com"

tbx_config init // will generate a default simple toolbox skeleton
/* Edit your macros, help, tests */

tbx_make 


CategorySep


2022-09-08 09:27