Performance Enhancements with DBServer and DataWindow (part 2)

This is part 2 of this article, see here for part 1!

Server Side Processing
This is a very important topic of its own, but we do not have time to investigate it fully today. It is often overlooked in the peer to peer environment that most of the client/server mechanisms that make RDBMS and ADS Systems superior, are still available to us. For instance, it is quite a simple exercise to write a GUI-less NT service to accept and execute server-side processing. For instance, record packets for update can be sent using cSocket classes and compressed XML (or simply, raw data) for execution on the server. Similarly you could send requests for reports, indexing or batch processing and printing. At no stage am I suggesting that you can achieve the full performance of a purpose driven RDD or RDBMS because these things use low level file manipulation. You would need to do the same to come close. However, we can still improve upon our base VO model.

Imagine if every update (reading is never the problem) was sent to a socket for the server to update; rollback could be maintained effortlessly and your worries over index corruptions would automatically be a thing of the past. I am not advocating the duplication of an rdd or of the full functionality of ADS etc but rather, I do advocate the selective transfer of bulk and repetitive activities to the server. Be warned, however, should you not maintain the same functionality at the work station level then you lose all the native advantages of a peer to peer system. This is not to be given up lightly.

In general, our discussion here today will concentrate on client-side processing, that means that the user’s application is executing all activity on the local machine and generally, connecting with files residing on another machine “somewhere” on the network.

The Nature of Data Aware Classes
Data Aware controls are those which are able to communicate automatically with a server attached to their owner window. Although this is not a strict definition, it will do for the purposes of our discussion. We will lump together all data aware edit controls, list boxes, radio button group boxes, check boxes, data browsers and data windows of all flavours. Please note that it can certainly include Fixed text. The raw power of VO over most other languages, including VB, C++ and Delphi is direct connection of a control to a field in a data source. You do nothing except drop the control onto a form with a server and its done! No extra coding.

Of course there is a huge amount of processing going on behind the scenes so once our windows become complex, we can end up overloading ourselves with this complexity. Without reverting entirely to the static control approach (which is totally possible in VO), we need to find some middle ground. The middle ground exists in the form of what is commonly termed “buffered datawindows”. Take a look at Figures 1 and 2.

Directly Connected Controls
The nature of a directly connected window and its controls is essentially that as you edit and change focus from a control, the connected server is updated immediately (…well, almost, see the discussion on concurrency). There is a limited amount of recall or reversal using Refresh() and Update() but once you move the record pointer in the server, the changes become permanent. The other issue is that during the whole time this window is active, you maintain a live connection with the server. Edits can occur at any time and other users may or may not be attempting to use the same record, or parts of it. In a busy network environment, this becomes a complex issue to manage and issues of contention and update visibility become important.

Under the Clipper model we generally locked a record prior to editing, did our thing and unlocked it again. Critical file-wide updates were handled with a file lock or even a progressive record lock. Either way, we could often roll onto or past other users in the same record set. So, is this wise? What if the user dwells in the window for minutes at a time and you have 100 concurrent users of the same server? Under DBFCDX, this probably represents three open files (server, memo and index) but under DBFNTX, it could be many more. In VO we were much better off with various concurrency modes available to us (ccOptimistic, the premium mode, will be discussed a little later) because instead of their being a network transaction for every field edited, we could restrict activity to just record change events. However, there is a different model worth investigating.

Let’s have a look at the code of the standard app to which we have added a listing and edit window to which we added controls automatically from our server. We now compile this and check its physical size and its runtime requirement. We will make this a comparison base for our future code.

The Buffered Server
Of itself, buffering a server only yields a small improvement in the above situation by keeping latent record locks to a minimum. This might in fact be a significant improvement in a busy network but buffered servers open the door to other techniques of value and that is the purpose of our discussion today. So what are we doing with buffering? Essentially we create a bunch of controls to hold and edit our server field values for the current record but the controls operate in isolation to the server. Now we need some scatter/gather code to read in or refresh values and to save them. At the outset it seems we need now to add additional code to our app but access into the network is only momentary: system wide, the number of file handles is reduced, user dalliance over editing eradicated and network load significantly reduced.

But how far do we take this? It depends on the manner of user access. If the user is in and out of windows like a yo-yo then perhaps the server should be left open for the duration but with reading/saving explicitly enacted. If instead the user opens the edit window and spends minutes there, the server need only be opened for the act of reading or writing and be closed at all other times. The model doesn’t matter if contention issues don’t exist.

Concurrency Models
There are four models available: ccOptimistic, ccStable, ccRepeatable and ccNone. Below is the VO On-line help information on these modes and the descriptions are reasonably self-explanatory:

The most common modes are of course ccNone and the default, ccOptimistic. ccNone is effectively Clipper style - you must make and release all locks which is totally tedious and totally unnecessary but not without its performance benefits. The trade-off is that you can reduce network traffic but only at the expense of record availability. Always remember, however, that VO’s GUI classes are based around the ccOptimistic model and so if you do not take advantage of this, you may be adding to overall inefficiency where you least expect it.

So, what is ccOptimistic? Firstly, what its not! Its not OPPORTUNISTIC_LOCKING. This is a network issue where the operating system decides when and how it will lock and unlock files for writing. This process is entirely beyond our control and is overlaid onto our application execution. ccOptimistic is a mode whereby we only lock records for the exact duration it takes to save a field and we only save those fields which have changed. It is in fact a form of double buffering, even if a little imperfectly executed. Consider Figure 3.

Basically, we separate reading and writing from moving. A move (commit, skip, goto, etc) causes our network updates but the mere act of getting and putting only works within our buffers. The initial state is to read the current record into the aOriginalBuffer. From then on, all reading and writing is done from aCurrentBuffer (if a change has been made) or from aOriginalBuffer if none have. When a move occurs, VO first re-reads the current record (we’ll come back to this in the next section) and decides if there are changes to be made to the physical file. If so, only the changed fields are sent and the next record read, starting the process over. Contrast this with ccNone. Evey single fieldput causes a network transaction, regardless of whether it is changed or not. But equally, we can move around the database without building buffers at every step of the way.

Figure 3: ccOptimistic Buffering by VO

Let us quickly look through the Notify() method and the FieldPut and Fieldget to review the difference.Clearly there are modes of operation which suit ccNone and those which suit ccOptimistic. Lots of record editing with varying fields changing: ccOptimistic. Heavy editing of entire records or heaps of movement without editing: ccNone. Generally, most accounting, finance and inventory applications benefit from ccOptimistic. Retrieval and view applications benefit from ccNone. Fortunately, we can flick from one mode to the other if we are careful. We will discuss which methods benefit most from this a little later.

Figure 4: __OptimisticFlush() Method of DBServer

Contention
We need to discuss the issue of contention and this is one problem that ccNone cannot help us with. Consider the situation where two users pull up the same record for editing. Both start editing the same record and one takes longer than the other. The first user saves an address change and moves on. The second user only wants to update the phone number. His screen (and controls) contain the original address but the new phone number. ccNone will allow him to save the record and thus overwrite the address change. Bad news! Here, ccOptimistic comes to the rescue. At the expense of another network read (review Figure 3), VO re-reads the same record and compares it with the aOriginalBuffer. If they are different, the second user is locked out of the record! Have a look at the code in the SDK which executes this:

Figure 5: __RLockVerify() Method of DBServer

Thus we can see that ccOptimistic completely solves the contention problem for us. This is great news. The penalty is a network transaction but if you were to implement your own contention code, you would have to do nothing less so at least you are spared the effort. There is a small error here in VO and I will show you how to repair this. Let us review a small part of the __OptimisticFlush() method. Take a look at Figure 6. Essentially VO sends an error object to tell the user the record has changed but what can the user do? To attempt to move off the record just gives the same error. We can’t re-edit the record so we are stuck. Instead, we decide to clear the current buffer, refill it with the newer data and offer it back to the user for re-editing. Importantly, you have the best solution for contention.

Figure 6: Repairs to __OptimisticFlush() Method of DBServer

Now, imagine a little. Because you have the changed record at your disposal, you could easily report the change to the user and ask you wish to proceed. You can even identify the updated data and offer the option to mix your data (because you also know from aCurrentBuffer which fields YOU changed) with the newer data. It is very easy to provide a high degree of sophistication here only found in RDBMS models.

A Further Look Into the SDK
So far we have investigated how VO’s native concurrency model works and the impact it has on the physical data. To enhance this picture we now need to fit the various window and control methods into the picture. Firstly, let’s take a look at the key control methods we may need to tweak for our benefit.

Linking Data Aware Controls to the DataWindow
Below is the method by which your controls become bound to a server. The key is the array aControls and it features heavily in the server control portions of DataWindow code. In the same box is the corresponding control method

Figure 7: Repairs to __RegisterFieldLinks(oDataServer) Method of DataWindow

Figure 8: Repairs to LinkDF(oDS, siDF) Method of Control

How does the control get here? Well, take a look into its Init() method and you will see how.

So there is lots of incestuous to’ing and fro’ing between the control and the window in the process of instantiation. Without pasting in too much code, let us just say that __SetupDataControl(SELF) of the DataWindow class merely adds this control to its aControls array and there is sits for processing. So in summary, here is the chain we are going to break with our performance enhancements:

• The DataWindow instantiates
• Controls instantiate one-by-one
• Each control decides it is Data Aware and appends into aControls
• The DataWindow connects to a server
• The DataWindow links each aControls member to the same server

What Happens with DataBrowser?
Essentially our DataBrowser is no different. Again let’s look to a generated Init() method and we will simply see more controls and usually, DataColumns as well. Essentially, they all get the same treatment and without going too deeply into the code, Class DataColumn has its own LinkDF() method, although it does not enter the aControls array, the SubDataWindow has special handling to ensure it doesn’t miss out.

One particular nicety about our browser, though, is our ability to make it Read Only and hence turn off all the buffer checking that goes with regular controls. Thus, if our aim is to buffer the datawindow, we only need to be concerned with SLE‘s and other Data Aware controls. Not the columns. However, the Form View SLE’s are linked into the DBF via the SubDataWindow in just the same way as they are for the main datawindow. Of course, you say, because both are datawindows and inherit from the same super class.

Please stay tuned to www.sdgn.nl to read part 3 of this article!