HSQLDB binary format import

Tender approved in the budget by the TDF board


In order to remove the legacy Java / HSQLDB database completely, and move fully to Firebird in LibreOffice 5.4 it is necessary to be able to import old document data with high fidelity from the HSQLDB binary file format which we have used (for performance) inside so many of our existing ODB files. This task involves reading the existing (reasonably simple) Java serialization code, and writing an (import only) filter to import this data safely into base.

LibreOffice bug #106866 is now fixed : When I change the field length of the result is multiplied by 4 each time (firebird, varchar)

Bunth Tamás committed a patch related to issue #106866: query character size instead of bytes

It has been pushed to "master":

It will be available in 5.4.0. The patch should be included in the daily builds available at 

http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours.

More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds 

Affected users are encouraged to test the fix and report feedback.

Firebird bug CORE-5452 is fixed : Segfault when engine's dynamic library is unloaded right after closing worker threads

Firebird bug CORE-5452 is fixed http://tracker.firebirdsql.org/browse/CORE-5452

The issue was reported multiple times in Firebird devel list, I will mention here Damyan Ivanov (Debian) and Stephan Bergmann. (RedHat)

And related commit in Firebird 3.0 branch

https://github.com/FirebirdSQL/firebird/commit/40f782ae3e918c4f3842571ff8064be1c4f54961

Stephan Bergmann contributed the patch to LibreOffice in external/firebird

LibreOffice bug #103074 : Implement Boolean Type for Firebird driver is now fixed

Bunth Tamás committed a patch related to issue #103074 : Implement Boolean Type for FB driver .
It has been pushed to "master":

It will be available in 5.4.0.

The patch should be included in the daily builds available at
http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours. More
information about daily builds can be found at:
http://wiki.documentfoundation.org/Testing_Daily_Builds

Affected users are encouraged to test the fix and report feedback.

ps:
From the list a of Firebird integration blockers there are only a few left

Listed blockers:

   70425: EDITING: Embedded Firebird - Copying integer values from Calc to Base impossible  See dependency tree for bug 70425
   70506: EDITING: Embedded Firebird - Alias in queries are ignored  See dependency tree for bug 70506
   104316: FIREBIRD 3 : attempting to open a Firebird 3 embedded ODB database file fails to draw all UI elements, causes general LibreOffice UI malfunction

Firebird 3 by default in LibreOffice 5.4

Lots of missing features & big bugs were fixed recently .
All of the blockers that were initially mentioned on tracking bug are now fixed.

Lionel mentioned that he added a few more blockers that needs to be fixed until Firebird will be switched by default :

I went over
https://bugs.documentfoundation.org/buglist.cgi?quicksearch=firebird

and added a few open bugs to the blocker bug
https://bugs.documentfoundation.org/51780

based on their description (and NEW instead of UNCONFIRMED status)
only, without reading the bug log in great detail.

Tamás Bunth (Wastack) agreed that 6 months time frame is more than enough to fix bugs that will appear

I agree, it would be reasonable to wait a half year. Until that we can
fix the already found and the upcoming bugs, solve the backward
incompatibility. Even if those can be solved quickly we can upgrade to
the c++ API. 

Another issue is importing old hsqldb databases without java installed

It requires making a read-only parser for the HSQLDB "cached" table
data format in another language than Java. Most definitely possible,
the format may even be better documented than "read the source code".

Service for migrating Firebird 2.5 format to Firebird 3.0

On Libreoffice Devel there is a thread about migration of the already created odb files with 2.5 databases inside (fdb) but also about the forward and backward compatibility of the Firebird backup archive (fbk).
https://lists.freedesktop.org/archives/libreoffice/2016-November/075953.html

Lionel added notes about upgrading in the wiki :

People trade privacy for convenience or the other way round all the time. People that in this case favor convenience can use the hypothetical web service. People that favor privacy can follow the instructions at
https://wiki.documentfoundation.org/Development/Base/FirebirdSQL#Older_Firebird_databases(which the web service should point towards for information).While I'm not exactly lobbying for this web service, we are a volunteer-driven organisation and if Tamás feels like doing it, all the merrier.

Lionel Elie Mamane committed a patch related to this issue tdf#101789 : Firebird 3 - fails to build on OSX 10.11.x with clang

Lionel Elie Mamane committed a patch related to this issue tdf#101789 Firebird 3 - fails to build on OSX 10.11.x with clang.

It has been pushed to "master": tdf#101789 work around DYLD_LIBRARY_PATH limitations on newer MacOS X.

An alternative fix has been in place ever since Stephan Bergmann commit " Allow external/firebird to be built with a custom SHELL under Mac OS X 10.11".

Bunth Tamás commited a few patches related to Firebird driver

Tamás committed a patch related to tdf#74172 issue.
Ensure prepared while setting numeric
It has been pushed to "master":
It will be available in 5.3.0. The patch should be included in the daily builds available at http://dev-builds.libreoffice.org/daily/ in the next 24-48 hours.
More information about daily builds can be found at: http://wiki.documentfoundation.org/Testing_Daily_Builds
Affected users are encouraged to test the fix and report feedback.

Another commit is just code refactoring : get table name from resultset simply. There is a much simpler way to access table name in ResultSetMetadata in Firebird sdbc.