General Module Testing
The main aim of this document is to define general module testing - what kind of
testing must be executed before any module can be exposed on the NetBeans Beta
Update Center.
One of the main intention of NetBeans plugin developers is to upload their
modules to the NetBeans Beta Update Center. This action can be done in case the
module passes the quality criteria. The
stable module criteria
document describes the criteria and process to add module to Auto Update. There
are three entities that are supposed to participate in the quality certification
of modules provided by 3rd party:
Developer or the team (module vendors) provide modules as well as they are
responsible for the quality of their modules - they execute testing defined
below. After they test the module they pass it to the NetBeans
community/quality engeneering team. Then the NetBeans community considers
such module, verifies the quality and decides whether the module will be
available via NetBeans Beta Update Center. Module vendors are supposed to execute testing of
module in these parts (general test scenarios for all contributions):
-
Module update
- test whether the module can be downloaded correctly from Update Center (according to the
module installation testspec document)
- test that the module appears correctly in Modules list
- test that it is possible to install/uninstall module in module manager
-
Options compatibility
In case the module uses settings storage then they must appear in correct
category of Tools | Options:
- test if it is possible to change each option related to module
- check that new values influence the ide according to specification
- try to set invalid values - they should be refused
- assure no change can break the module/IDE
-
Dependency and integration testing
Module depending on other module(s) don't break the functionality of
dependent module(s). All features of the module are working and don't break
IDE functionality.
-
Layout testing
Test layout of all module dialogs:
- resize of dialog must not break the look of dialog components
- dialogs specified as modal must block other dialogs
-
Menu testing
If the module adds the item to main/popup menu then the add of new item must not
change the order of original items. Menu accelerators/mnemonics of new items must not be
in conflict with other items.
-
Accessibility
Assure that the module is accessibility compliant. See NetBeans accessibility web-site.
-
Hardcoded strings
Assure there are not hardcoded strings in the code - they must be stored in
resource bundles.
-
Help sets availability
Try to invoke contextual help on all dialogs/views according to specification.
-
Testing on various platforms
Module vendors will execute tests on as much platforms as possible. Tests on
remaining platforms will be delegated to NetBeans community/QE team.
The 3rd party must provide to NetBeans community/quality engineering team following resources:
- Final module as .nbm.
- Functional specification of the module (not mandatory).
- Test specification for the module (not mandatory).
NetBeans Quality Engineering team assures it is possible to install module without
problem - IDE will be tested briefly to assure there is no major troubles with
integration of module to NetBeans IDE.
QE needs
3rd party resources in order to
execute tests.
QE could help to test module in this area:
- Module update (install/uninstall).
NetBeans community members can help with module testing. Volunteers can
participate on testing before the module is provided to QE team. The vendor should
inform community on mailing list (
nbusers@netbeans.org would be the best place) and provide
final module to the persons interested.
Volunteers testing is grateful and valuable especially due to the execution
on various developer environment (platform/OS configuration and JDK version).
Issues found in 3rd party modules ought to be reported to the module vendors
and should not be filed into the IssueZilla bugtracking system.
