AbiWord Weekly News #95, (2002, week 23, released 2002.06.10)

Welcome to issue 95 of the AbiWord Weekly News.

This week there's been some nice progress made on both table and GTK2 support. We also got a new baby AbiWord TWiki running on www.abisource.com (please help feed it!), and finally, Tomas Frydrych has written an excellent Special on AbiWord's BiDi support. Oh, and we released AbiWord 1.0.2.

In this issue:

• CVS Stats
• Bug Update
• Latest Releases
• On the Mailing List
• Special Interest

Editor(s) of this issue: Jesper Skov

CVS Stats

Bug Update

Here's the AbiWord bug update for the last week. If you can spare some time, please help us triage the bugs - you don't need to be a programmer to do this.

Bug Statistics

Bug Votes

These are the week's Top 20 Bugs in the categories problems and requests for enhancements. Influence next week's results by casting your own votes.

Verified Bugs

In the past week, the following Bugs have been verified as fixed. The listed Bugs have all been put in RESOLVED mode as either FIXED or WORKSFORME and have then been verified by the people listed below. Other causes of resolving a Bug (e.g. as INVALID) are not tracked since they usually do not represent a fixed problem.

To get your name in neon, help QA the Bugs. If you want your name rather than your email address to appear, drop me a line.

Latest Releases

Here are links to the latest official releases of AbiWord for various architectures and operating systems.

These are links to snapshot builds of AbiWord for a subset of the supported architectures/operating systems.

Note that the snapshot builds may not work (at all!), but are likely to include more features and have fewer bugs than (older) official releases. Use the official releases for "production systems" and the snapshot builds for testing and when you want to help with Bug triaging.

On the Mailing List

Traffic on the developer mailing list has settled at about 200-400 postings per week.

You may also find interesting threads on the user and documentation lists (unfortunately the archive for the latter is broken at the moment).

This week, interesting topics on the developer list included:

1. AbiWord 1.0.2 Released!: This week Mark Gilbert announced AbiWord 1.0.2.

2. Code Formatting: John W. asked what code formatting style to use. The answer can be found in the file abi/docs/AbiSourceCodeGuidelines.abw. Unfortunately, not everybody follows the guidelines. Programmers are a lazy lot, and even though there's been suggestions for adding layout instructions in every file for emacs, not everybody uses emacs. So we'll probably have to live with the current sad state of affairs.

3. Zeroth order table Table support.: Martin has made great progress, and posted the first snapshot of an AbiWord document with a table.

4. Commit: GTK2 branch compiles, links, (sort-of) works: Dom also posted a snapshot of the GTK2 progress he has made.

5. AbiWord Twiki engine: I set up a twiki engine on the AbiWord server. The primary goal is to allow for all AbiWord users to help add to the FAQ. But hopefully it'll aquire a life of its own. If you don't know what a wiki is, I strongly suggest you spend an hour or so looking around here. I found it very interesting. I will try to make a Special on the AbiWord Wiki in some future issue.

6. PATCH: WordPerfect lists: William Lachance keeps working on the WordPerfect importer, and has added support for lists.

7. Xft front status -- WANTED: win & postscript hackers: Joaquín Cuenca posted a summary of his Xft work, including a snapshot that looks very nice (to a Linux user anyway).

Special Interest -- AbiWord and Bi-Directional Support - Tomas Frydrych

How I got to be involved with AbiWord

As a would-be scholar of the Hebrew bible, I have got word-processing requirements quite different than most 'normal' users; in particular I have been confronted by the need to include Hebrew, Greek, and Syriac into my documents, and I have not been able to find a suitable wordprocessor that would let me do that. There are some specialized wordprocessors for linguists out there, but they tend to be expensive, and judging from the comments of those who invested into them, they are not always satisfactory. Consequently, there are workarounds used in my field of work that make it possible to use standard wordprocessors. These start with specialized fonts, that are really Latin or symbol fonts just pretending to be Hebrew, Syriac or Greek fonts; they are usually cumbersome to use, sometimes giving the impression that they were designed by someone who did not fully appreciate what they were for, they use a myriad of arbitrary encodings, and they often cost arm and leg. There are then workarounds to circumvent the cumbersomeness of these fonts, keyboard utilities of various kinds, but these too do not solve our problems. Then there is the fact that what worked with version x of a given wordprocessor will rarely work with version x+1, and that sticking with the older version quickly becomes impossible -- it is this pain that a biblical scholar has to put up with, that sets the background for my involvement with AbiWord.

I stumbled on AbiWord entirely accidentally. As a Linux novice, I was keen to free myself from the tight grip of Micro$oft, but quickly discovered that there was no wordprocessor that I could use for my work under Linux. Initially I hoped to use StarOffice, but it did not take long to find out that it was not able to cope with my specialized fonts. Not willing to give up, I searched the net for "free wordprocessor", and landed at abisource.com. What struck me there was that AbiWord felt quite like The Other Wordprocessor I was used to, but with it being Open Source I could make it do what I needed it to do, that is, handle bi-directional (bidi) text, which gets me to the subject of this article.

Introduction to the Bidi Problem

There are some languages, such as English, Czech or Greek, that write from left to right (LTR), and other languages, such as Hebrew, Arabic or Syriac, that write from right to left (RTL). You are probably thinking, OK, so what's the great deal. Well, as long as you are dealing with uni-directional text, i.e., pure LTR or pure RTL, nothing at all, you simply draw these characters on screen in one direction or other. However, when you start mixing the two, resulting in bi-directional text, things get more complicated than might seem at first; for a wordprocessor there are two issues in particular: presenting the text, and navigating through the text.

Presenting text

The nature of the problem of presenting text is best illustrated by examples (here lower case letters stand for LTR text and capitals for RTL text).

Say the user types:

abc XYZ klm

How should it look on the screen? That depends. If the whole line is considered to be LTR line with embeded RTL quote in it, you will get:

abc ZYX klm

but if the line as a whole is considered to be RTL with embeded LTR text you get

klm ZYX abc

So from this it can be seen that the layout of a line of bi-directional text depends on (1) the directional properties of the individual characters; (2) the direction of the overall context. Unfortunately, the situation is complicated by the fact that not all characters have clearcut directional properties. Some characters are directionally neutral, i.e., their actual direction is dictated by the immediate context. Consider, for instance, punctuation:

If you type

abc XYZ klm.

and the overall direction is LTR, you will get

abc ZYX klm.

but if the overall direction is RTL, you will get

.klm ZYX abc

that is, the ' .' behaves here like an RTL character.

Then there are characters, like European numbers, that behave even stranger. While in themselves these characters are LTR, their direction is weak. If you type

1234 567

on a line that is overall RTL, you could get two possible arrangements,

1234 567

or

567 1234

In the former case the whole sequence is treated as regular LTR (for instance a phone number embeded into a Hebrew text), in the other case it is treated as weak LTR text, with the word order being RTL.

As you should see, ordering bi-directional text is not a trivial matter. The relationship between the logical sequence of characters, i.e., what the user types, and the visual sequence, i.e., the way it is displayed on screen, is described by the Unicode Bidirectional Algorithm, which is fairly complex.

Navigating through text

The fact that the visual sequence of characters on the screen can be quite different from the logical sequence in which the characters were input, creates complications once you start navigating through the text. If we take the example from above, with logical sequence

abc XYZ klm.

which, if treated as RTL text with embeded LTR segments, we get visual sequence

.klm ZYX abc

The HOME key should then position the insertion point (indicated by |) like this

.klm ZYX | abc

the END key needs to take me to

| .klm ZYX abc

and if I now press the right arrow, it will take me to

.klm | ZYX abc

further 3 presses of the right arrow will take me to

. | klm ZYX abc

and a further press to

.klm | ZYX abc

In other words, a sequential movement in the logical plane results in a non-sequential movement in the visual plane and vice versa. This can sometimes have rather interesting consequences. Consider the visual sequence above with the the insertion point just after the letter 'c'. Where should the insertion point, which indicates where the next letter will go, appear on the screen? Well, that is a good question. If the next character will be for instance 'd' the result would look like

.klm ZYX abcd

so we would expect the insertion point to be like this:

.klm ZYX abc |

However, imagine that the next letter will be 'Q'. In that case the result will be

.klm ZYX Qabc

and so we would expect the insertion point to be positioned like this

.klm ZYX | abc

Obviously, in bi-directional text the visual position of the insertion point can be dependent on the next, yet unknown, character. A possible solution to this is to display two insertion points, each with a little flag indicating which direction it applies to. This is what we do in AbiWord, and that brings me to some more details of AbiWord's bidi implementation.

Bi-Directional Support in AbiWord

When I first stumbled on AbiWord in the spring of 2000, it did not have any bi-directional support, but after familiarizing myself with the sources, it seemed to me that it should not be too difficult to add this. With hindsight I was somewhat optimistic (naive?), at the end it took hundreds of hours of work spread over 18 months to get the bidi support to a reasonable shape.

The whole process can be separated into two parts: (1) calculating the visual ordering of characters; (2) separating calculations that need to be done in logical plane from calculations that have to be done visual plane. The first task can be delegated to some third-party library; in AbiWord we use the excellent Fribidi library, which now implements the entire Unicode Bidirectional Algorithm. Integrating Fribidi with AbiWord was not entirely straight forward to start with, partly because we had some needs that were not foreseen in Fribidi design, and partly because it operates on raw Unicode characters that are not readily avaliable in AbiWord (its layout code operates with more abstract notion of runs of text, where a run is a sequence of characters of arbitrary length). It is only fair that I acknowledge here the assistance of the current Fribidi maintainer Behdad Esfahbod, who repeatedly implemented changes that I requested of him, so that now we have reached a stage where we should be able to use a Fribidi library installed on the system rather than having to rely on a patched version built into AbiWord.

The second part of the bidi implementation, the separation of logical and visual calculations is what took most of the effort. While the changes in themselves were not too profound, there was lots of them; the initial bidi patch was some 50kB in size after compression with gzip, and that made most of the existing developers reluctant to review it, with the exception of Jesper Skov. Further, because the translation from the logical to visual planes and vice versa is entirely dependent on the wider textual context, the bidi code was prone to crashes, often trying to query things that were yet not there. Identifying and debugging these problems was time consuming, and could not have been done without the few faithful testers; I should mention Uri Elias, Uri David Akavia, and AB), as well as assistance from Tzafrir Cohen who pointed me to useful resources and brought some of these testers to AbiWord.

There was yet another issue. Compared to handling uni-directional text, bidi text requires non-negligeable amount of extra processing. As most AbiWord users would not process bidi text, there were concerns among the developers about adverse effects on AbiWord performance (this, together with the stability problems, was the main reason for maintaining separate bidi and non-bidi versions of AbiWord). Consequently, I have spent a lot of effort on optimizing the performance of the bidi layout engine. In the first stage, logic was built in which completely circumvents the full bi-directional processing for pure LTR lines of text, and only uses extremely simple algorithm for pure RTL lines. In the next stage, I have managed to reduce the memory requirements directly associated with bidi processing to a level where the amount of resources used is not much greater than that of the non-bidi version of AbiWord; suggestions from Yaacov Akiba Slama were instrumental in this. Further, in the past 7 or 8 month, I have put lot of work into optimizing other aspects of the layout code, not necessarily related to the bidi processing. As a result at one stage the screen drawing performance of the bidi version was noticeably superior to that of the non-bidi version, so that the changes were later moved out of the bidi-only code to speed up the regular version as well.

The Present and the Future

With the release of AbiWord 1.0 we have crossed an important milestone -- in the future there will be only one version of AbiWord, one capable of handling bidi text. This should give us shorter bug-identification - fix cycle, as in the past lack of testing on mass scale was one of the problems for the bidi development. However, this will have only a limited effect unless more developers get actively involved in the bidi work. There are still outstanding issues, among them adding, testing and fixing bidi support for the various importers and exporters; at the moment there are known problems with the critical MS Word importer that require further work.

The next crucial step related to bidi support is to add proper glyph shaping so we can handle scripts, for instance Arabic. We currently have a very basic shaping support, which is capable of providing elementary support for Arabic, but work has already started on using an external shaper Pango to do the job properly, but I shall leave that for a different article.

In closing I would like to say that as much as I enjoy working on AbiWord, my time and resources are limited by the need to earn living; for the long-term future of the bidi support in AbiWord it is necessary to expand the one man team. I appreciate that the whole notion of bidi text can seem quite esoteric at first, but it really is not that complicated and even those with no or limited prior knowledge of the problems can quickly get into it -- give it a try!