Contributor - ATOMS Portal

Description

The https://atoms.scilab.org ATOMS portal now homes around 200 external modules (2016). Yet, it still misses many useful or even mandatory features, and have pitfalls. Most of them apply as well for the FileExchange Portal (&FE).

The following bugs are examples of such shortcomings:

14436: It is not possible to filter modules by OS, by supported scilab versions, or look for an author or maintainer.
14117 (&FE): Formatting of some contents is turned very hard due to a badly placed htmlization. Lost hours...
11615 (&FE): When we subscribe to a module, it is not possible to receive only users comments. We also receive a copy of all detailled administrative operations done on the package, such as upload or validation or cancellation of each file, etc. It is so boring that subscribers are rare. It is mostly counter productive. In the ATOMS user preferences, it should be possible to subscribe only to users comments, excluding all administrative notifications (provided that the visitor is not one of the module administators).
11439 (&FE): For a module, it is not possible to see users comments merged over all module versions. We have to browse each version one by one to do so.
6723: The help pages of an external module are neither published nor publishable from/on its ATOMS page. Implementing that would be a major improvement.
8489: There is no way to indicate in a standard way the list of functions proposed by a module. This list could be built from the short descriptions given in the help pages.
14246: There is no way to mask modules that are in the Test category, and modules that have no file. This should become possible with a checkbox "Ignore test or empty modules" either in the user ATOMS Preferences or in the ATOMS portal, with other filters. see below for a better way to fix it: separate the test environment from the live environment
11590 (&FE): When we have written a comment and then we click on the "Add me to the CC list of this toolbox " checkbox, the page is posted without the comment: It is lost!
13812: The list of files of a module is not always published, or is sometimes corrupted.
13821 (&FE): For the module administrator, when a new file is uploaded, the notification mail does not indicate the file checksums.
13818: The form to create or update a module misses contextual help. Many tooltips would be very helpful should be fixed now Moreover, some major features are not documented 13819 see below

Here's also a list of the tasks that we (Scilab Enterprises) have defined, or have started working on. These tasks could get some help too:

Summary	Details
Ensure that Atoms builds are done in the same environments as the builds of Scilab	Currently we have specific machines for the builds, but they are not exact clones of the build machines. They should be; this is about setting up (VirtualBox / vagrant) VMs to run the builds.
Remove the "ATOMS packaging system" field, it's useless	- or at least give a better description - it doesn't mean anything: ATOMS packaging system -> "Packages availability" et "This module aims at being included into the ATOMS packaging system ? -> "Should Scilab Enteprises packages this toolbox ?" this is 13819
Prevent renaming files, which is causing weird errors, and forcing the user the do a AtomsSystemUpdate()	when a file is renamed on the server, and a new file is created with the same (previous) name, it causes an error when any user install the the toolbox's renamed file, or any toolbox that depends on this file. The only workaroud is for the user to do an AtomsSystemUpdate(). This is because suddenly the MD5 sums on the local files don't match the MD5 sums from the server. It should not be possible to rename files.
Integrate the DESCRIPTION from server when building (on the server)	Currently: in the Toolbox Skeleton, there is a DESCRIPTION file to be filed. However, this information is also on the website. The information on the website SHOULD be the one used; and the DESCRIPTION file from the skeleton is only to be used for toolboxes not published on the website. So, the last thing that needs to be changed, is to ensure that when uploading a binary (or when downloading a source file ?), the DESCRIPTION file is replaced on the server, by the file generated by the server (based on the info in the DB). Not a perfect solution but the only correct one to avoid this information duplication issue
Remove the "t" element of the "x.y.x-t" versions	The version after the "-" is currently a workaround to allow multiple sets files to be released for one release. It should not be displayed in the version on the website (instead, it should be displayd as a version of the file, most likely) This is also the "PackagingVersion" field in the DESCRIPTION file generated on the server - but it's barely (and useless) in the Scilab client. This should be cleaned and removed. this would mean present it differently on the website ; and do not use it on the client in the version dependencies (always use the biggest one)
Get the server to only send the latest version for each toolbox	Currently, the server (in the TOOLBOXES.gz files) sends a list of all valid toolboxes for the version of Scilab. Most likely (to be verified) this list could be much shorter, by only sending information about the latest version of each toolbox. This would provide a big performance improvement, which is not yet a prb on scilab 6, but will be relatively soon; and is already one on 5.5. It would also alow simplifying code in Scilab (see next itesm)
Also, get the server to verify that dependencies are handled in the TOOLBOXES.gz file	Currently, there is no check that dependencies are correct in the TOOLBOXES.gz file (i.e. that all toolboxes needed as dependencies for all toolboxes are present): this is only done by Scilab once it has received the file. This could be done once for good when creating the TOOLBOXES.gz, and the removed from the client.
Once the two previous items are done, remove useless code in Scilab	simplify dependency handing in Scilab (remove tree-building code). Use the fact that the server only sends the latest version to simplify that code. This would mean, however, that local (non-downloaded from atoms.scilab.org) versions of dependencies would not be handled: but this is not a usual case, and would be OK (to replace by warnings)
Once the three previous items are done, simplify massively the code to handle the dependencies in Scilab	in particular remove support for "<=", ">", etc.. ranges, which are not used (possibly non-compatible, i.e. only on 6.0)
Simplify the build scripts for atoms.scilab.org	This has been mostly done, but need more testing. In particular, be clear on whether it goes to the server to get the DESCRIPTION or not; clarify the options for that.
Better presentation for 5.5 vs. 6.0 on the website: to design.	Currently need two versions (5.5. and 6) for each toolbox; would be better to use the same, when possible (knowing that it's not always possible). Probably means a change in DB design (and a data migration); because of that, probably to keep to a minimum (because migrating data and app is always complex)
Don't show old versions of old toolboxes so proeminently on the website	On the list on the left side, all versions of all toolboxes appear. This makes it a very long list, with mostly irrelevant information. Probably this list should only only present the latest one(s), or one for 5.5. and one for 6.0. We could in fact entirely remove the older versions.
Add ability to post comment without rating, and rating without comment	it doesn't make sense currently to be required the user to add both, probably we are loosing a lot of comments (and rating) because of it
Set up Atoms server for tests only	Currently atoms.scilab.org include toolboxes used for tests, which is very weird to the user. The easiest solution would be to set up a specific website 'tests.atoms.scilab.org' for the tests, with its specific DB, and use that for tests. It would also require editing the tests to use this new server. That would be a simple solution (setting up a new atoms site should be easy - we already did it for dev.atoms.scilab.org) this is 14246 mentioned above
A way to publish the toolboxes' help on the website (reported by S. Gougeon)	This was probably what the DESCRIPTION_FUNCTION had been done for, but it was not used anywhere (neither on the atoms.scilab.org website nor in scilab), so totally useless - so I recently removed it from the toolbox_skeletton, and removed any mention of it. But still, would be really nice for the doc to be published (automatically ?)
Remove or fix list of files, and build logs	List of files (as well as tree - but it's not working) and build, and test logs, are kept with the files. This is not very useful (end-users shouldn't need to look at build logs...) and could be removed. Removed build logs, and file tree; kept file list when it is present, and keep it in the build process
Security: risk of SQL injection on atoms.scilab.org	I checked a little, and there is no obvious risk - I couldn't find a way to actually modify data on our server though SQL injection (because we ony use mysql_query which only allows one query at at time) nor access user data (they are not stored in mysql but on the LDAP server). However, the site should use prepared statement everywhere http://php.net/manual/en/pdo.prepared-statements.php - shouldn't be complex to change it, just some work
TOOLBOXES.gz are getting regenerated	this is for the 5.5. files in /home/web/fileexchange.scilab.org/5.5 on the server: there's some dup data, probably, that are causing them to be regenated all the time. the prb is: "sourceName: number--1-src.zip" vs. "sourceName: number-1.3-1-src.zip". Probably some dup data fixed
Move rarely - or unused - Atoms functions from Scilab into a specific, optional, toolboxes.	Most users only use very few functions from Atoms: atomsInstall, atomsList, atomsRemove... Many other functions for Atoms currently in Scilab are not useful to the end-user, only used for Scilab Atoms build; in particular: atomsDepTreeShow ; atomsTest (used only in atomsCheckModule - we don't use it anymore for our builds) ; atomsCheckModule (probably not used by anyone);... and probably many more. These functions could be removed, or at least moved to an toolbox that people would only install when needed. It would make Atoms much easier to maintain (for us), much easier to use (for the user).
Create Jenkins jobs to run the builds	the old build website http://nightly.scilab.org/ hasn't been working for a while. Instead of fixing it, we should set up Jenkins tasks on http://build.scilab.org to build Atoms toolboxes; it should be easy, although it may require modifying the build chain to ensure that the output is kept properly.
Check load time and performance	Some pages - for example https://atoms.scilab.org/toolboxes/Mathieu - seem to take a long time to load. We should see if SQL or page is inefficient (missing SQL indexes ?)
Clean up the wiki pages	There are two pages explaining how to create/build toolboxes: https://scilab.gitlab.io/legacy_wiki/ATOMS?highlight=%28atoms%29#ATOMS_for_developers, et il y a https://scilab.gitlab.io/legacy_wiki/howto/Create%20a%20toolbox . They should be at least linked together; if possible clarified; and ideally merged. (this should be a small task). There is also a private page ( https://scilab.gitlab.io/legacy_wiki_project/Compiler%20une%20toolbox%20dans%20Atoms ) which should be made public.
Release the AtomCC toolchain to the community	Internally, Scilab Enterprises has a toolchain, written in Perl, to build and release the binaries for toolboxes. There is no reason for this toolchain not to be public, in fact to help toolbox developers to tests their toolboxes, and even release the (all ?) binaries to atoms.scilab.org . Currently the toolchain requires SQL and ssh access to http://atoms.scilab.org to release the files, but it should be easy to replace that to make it an HTTP upload - that toolbox maintainer could run with their own maintainer accounts
It should be possible to see who released a file	summary says it all

Many other tedious bugs are pending. Their full list only for the ATOMS portal is here. The full list including resolved bugs is there

For the FileExchange Portal, it is here. Most of its bug apply as well for the ATOMS portal.