XML in, with, and for VFP

Overview

As a VFP Developer, you have plenty of uses for XML. This article is a slightly-updated excerpt from a longer document I wrote more than five years ago. (You can read the full document, with a lot of still-useful specifics at http://spacefold.com/lisa/lisa_xml). All the reasons I gave then are still true today, and have been borne out by common practice:

• XML provides a platform- and language-neutral syntax for interchanging information between applications and business partners, so it is ideal for VFP applications that function as mid-tier components, in web as well as non-web-enabled architectures.
• It is ideal for multilingual applications, because XML was designed from the ground up to handle Unicode.
• It provides raw content for unlimited forms of display, so it is an ideal tool to complement the shortcomings of the VFP FRX and reporting systems.
• It is well-equipped to express relational data as well as to describe objects, so it is ideal for use in an environment, like VFP, that use both normalized databases and object hierarchies.

I don’t think you should magically believe that “XML is where it’s at”, no matter where Microsoft or anybody else, including me, says you should want to go today. Still, I want to let you know why I think XML is so important. I will tell you something about my experiences and extrapolate some general meaning from them.

This is not about “the Internet”

This is not about “the Internet”. These days, it’s true, “distributed” often automatically means “working over the Internet”. Many of my examples have an Internet component or “background”, too. However, data interchange and data presentation are problems you have to solve whether you work with the Internet or not. My examples and advice are intended to show you how VFP and XML work together within any application.

No app is an island

All of you have written “collaborative software” already. Your VFP app is dependent on the outside world, every time you send a REPORT FORM TO PRINT out through the Windows printing subsystem. If your report contains a letterhead image embedded in the FRX, VFP calls the image’s host application when rendering the graphical portions of the page. You make use of external components in your VFP applications every time you use GETFILE(). And, of course, many of your applications use COM components or manipulate external data from other database servers.

XML gives us a way to have a maximum number of components talking to each other with a minimum of fuss

By writing VFP applications that expose XML interfaces, you make your work accessible to more types of developers, who can hook together more types of functionality, than ever before. XML makes it easier to communicate through firewalls, to transcend configuration hassles, and to alleviate character set and codepage issues.

Publishing everywhere

Since 1997, I’ve been talking about something I call “the one document process". I’ve been looking for ways to make it work, for much longer than that. Lots of other people were searching for ways to do this, too, although they may have called it something else.

In a one document process, we store information only once across an enterprise, no matter how many business units need it, no matter how many formats they want for views and reports, even data input screens. When it comes to the metadata that describes our application, the data that describes our work as developers we also have one document. This one document, starts with requirements, goes through testing, and proceeds to end-user help files. We generate all output materials as customized views and formats of one set of metadata.

We can store all the necessary data in VFP tables, if we want, but that’s irrelevant. No matter where the data is stored, in VFP our abilities to interchange data and present data are not complete. We might have to extrapolate Visio diagrams indicating process flow and functionality from the metadata we have stored, for developer documentation. We might extract help files and use cases in .DOC, HTML, and PDF format. 

XML gives VFP the tool to write once, publish every-where, every-what, and every-how.

XML gives VFP the tool to write once, publish every-where, every-what, and every-how

What VFP applications do, and will always do

Think about what all your applications do, and by extension what you do every day for a living (see figure 1 below).

Fig. 1: An oversimplified, but comprehensive, view of VFP activity.

Essentially, you receive data input, you take care of it, and you provide data output. VFP is great at doing these things, but that’s all it really does. Looks simple and straightforward, but as you know, the “devil is in the details”.

If the data comes in from an IMPORT process, you have to write certain types of procedures to clean it and import it into your database, error handle, et cetera. If the data comes from a human typist, you have to write entirely different procedures – to validate, provide feedback, and handle any visual cues and infelicities of the GUI controls in your forms and toolbars – to bring essentially the same rows and columns into your database. If your client wishes data to come into the application from a new source – say, a Palm device or a bar code reader – you have to handle different cues, different feedback, a different interface. You will also have to write some code to talk to the Palm device, which is code of a type that VFP is not well prepared to provide.

If your data output or presentation is going to an EXCEL spreadsheet, you may write some automation procedures, whereas you use a REPORT FORM to go to a printer. If you send the data out to a fax, you face a different set of problems once again. When your client says she’d like browser-based reports as well, you go back to the drawing board to start generating HTML in yet another, distinct, output procedure.

What XML-enabled VFP applications can do

Suppose all these input and output devices could accept and provide the same message format. Each device would still have different limitations and different needs, but the procedures you used to address them all – to provide feedback, to deliver data, to communicate errors – would be the same. In a client-neutral strategy, your application activity and the code you planned to write would be simplified in structure, looking more like figure 2 below.

Fig. 2: A “client-neutral” VFP application has a simpler job.

In fact, we could take this a step further and erase the distinction between data input and data output, from the point of view of structuring the code you have to write to collaborate with other applications. Collaboration and messaging between components would be reduced to a series of requests and responses, as you see in figure 3.

Fig. 3: A “messaging view” of VFP applications simply accepts requests and provides responses, with no distinction between input and output.

What should the common messaging standard be like? There are several reasons why XML is a good choice for messages between disparate clients and servers. Among them:

• XML is a format easy for multiple types of participants to create and decode, because it is not binary. Yet it makes provision for multiple extended character sets and encodings, and does not require the participants to share these character sets or encodings.
• XML easily handles relational data, because of the way elements nest  Lookup elements can be handled with external references to other documents, the use of ID and IDREF attribute types, and many other schemes. Yet it is not restricted to a regular format like relational databases. Binary data of any imaginable rich data type can be designated using NOTATION-type attributes or the XLINK standard. Like Lotus Notes and object databases, XML can “stretch” to irregular formats that easily represent real-life needs without following a pre-designed structure.
• Unlike EDI, XML messages can be broken up with separate sections sent in parallel, for better performance. Also unlike EDI, it is not expensive to get started in XML and participate in XML exchanges.
• XML passes easily over HTTP transport, and can pass through firewalls. Yet it doesn’t execute or trigger code directly, even though an XML message may describe code to be executed. The receiver is in charge of what code can be described and executed on the receiving system.

There are several reasons why XML is a good choice for messages between disparate clients and servers

With standardized XML requests and responses as your collaboration strategy, you do not plan separately for separate clients. You define the following elements, the same for all clients:

• Define a message, or as many message formats, as you wish, to expose your public interface, your way of working with other applications. (Don’t go it alone – investigate SOAP or Hewlett Packard’s e-speak, or one of many available standards for defining document formats. But if you choose to invent your own format(s) instead, as you’ll see, you haven’t precluded any opportunities. )
• Define a way or ways in which you will allow the request messages to be passed to your applications (will you use a COM message call accepting an XML string, or a filename, or both? Will a command-line argument indicate a file to be processed? Will your application look for files in a particular physical location representing a task queue?).  
• Define your application’s method of passing back a response back to its collaborators (usually, responses are provided according to a similar plan as the requests were accepted).

When you see your work in these terms (a series of uniform requests and responses), it almost looks as if anybody could do database development work! Not so. We’re just removing the distractions, getting you back to the work you used to do when you wrote FoxPro applications, before collaborative applications. Data is still complex, and proper data storage and analysis takes your professional skills. From what I’ve seen in the last couple of years, businesses and organizations of all types sorely need your skills focused on the work of data management, not siphoned off into understanding how a serial port or some other interface to an input or output device can be coaxed to work with VFP.

You don’t have to change all at once

Suppose your applications currently have a standard Windows GUI interface, do you have to stop using it or change how it works to start providing this common message interface?  Suppose your applications currently use REPORT FORMs to provide printed output, do you have to stop using reports? Of course not.

Define your XML requests and responses based on the kinds of input your applications already need, not replacing them. XML messaging can exist in parallel with your current activities, while making room for new clients, without a problem.

For example a database update XML request would include the same elements currently provided by your data input screen. Your data input screen continues to function as before, while the other clients used the XML request-processing to handle updates.

Whenever you’re ready, you can retrofit the data input “save” button to construct and send the same XML update request to your database procedures as all other clients. The native language for your application to accept updates is now XML. Your save button uses a proxy to translate the form and send that XML. You use the same strategy if your customer asks you to handle something new, such as an EDI exchange: translate the exchange via proxy into XML and continue with consistent, “EDI-unaware”, processing.

The native language for your application to accept updates is now XML

How we have used XML in VFP

For the past 6 years, XML has allowed the applications that Colin and I have authored to be completely “client neutral”. They receive requests and send out responses to Java, Flash, VFP, VB Script, Perl, or Palm clients… you name it. As described above, we sometimes have to add a thin translation layer when the client doesn’t speak XML natively, but most of the processing continues exactly the same for each exchange.

Fig. 4: Acxiom architecture shows XML and XSLT at work both in internal and external exchanges.

Our XML-handling mid-tier has changed backend data resources more times than you can imagine. These back end data resources are not accessed through the same interfaces or mechanisms -- we have been through ODBC, sockets programming, and CORBA – and they don’t even have the same data structures. The client receives the same XML response, all the time, no matter what.

The client is completely unaware when our mid-tier changes, too. So sometimes the mid-tier includes VFP in the solution and at other times it does not. Our goal has been to maintain what the client experiences, through each change.

You have probably heard people say that XML doesn’t do much good unless people agree on document formats (DTDs or schemas). In fact, this is unnecessary when you add XML’s sister-standard XSLT (eXstensible Stylesheet Language: Transformations) to the processing. As shown in figure 4 above, we have an architecture in which different parts of our own enterprise don’t share the same format, and we use XSLT to move between them. On the client side, our XML-enabled partners have their own requirements for document format, and XSLT comes to the rescue again.

Starting to do the work

The real XML-handling job for your applications is to figure out how VFP will handle these three main tasks:

• Designing and creating the documents we send to partners, clients, and servers
• Parsing the documents they send to us, to determine the required actions
• Transforming between XML formats, as necessary, in between.

You usually start by designing your document, just as you’d design a database. You create a DTD or schema to define the document. You put sample data in it, just as you’d append records interactively in a newly-created Visual FoxPro table, to test the design.

Creating the XML documents we’ve designed

After design and interactive document creation comes programmatic document creation. In VFP we are used to creating strings on the fly, and they can be very large strings. There is no reason why you can’t create a couple of classes designed especially for this purpose to create XML documents.

The method you’ll use to build up a document manually in VFP code varies depending on the content of the document. However, no matter which VFP techniques you use to create a manual XML document, make sure your documents are well formed before passing them to another application or a user. An XML document is well-formed if it follows the syntax rules of XML, including the necessity of closing all tags and having a single root node.

It is the responsibility of the application formulating the XML to make sure the XML will be parse-able, without any further adjustment, by the receiver of the document.

It is the responsibility of the application formulating the XML to make sure the XML will be parse-able, without any further adjustment, by the receiver of the document.

Don’t confuse well-formedness with validity (conformance to a schema), which is not always required, or with balance (ensuring that all tags are closed), which is required for xml document sections, or fragments, but does not automatically indicate a well-formed document.

How can you check for well-formedness?  The easiest way to test the XML you’re generating manually is to load your results into a parser conforming to the XML standard, at least for debugging purposes. To check from the command window, simply CREATEOBJECT(“Microsoft.XMLDOM”), which gives you a reference to the Microsoft XML parser. Now load your document into the parser as a string using the LoadXML(tcString) method, or use the Load(tcFile) method for a file on disk. If the method returns .F. (or errors, depending on your version of the Microsoft parser and/or your version of FoxPro!), you should be able to check the .parseError.reason property to see what went wrong.

If you don’t mind leaving the VFP environment for your tests, load your XML output into _CLIPTEXT and move over to an XML-oriented IDE like XML Spy. In this type of editor, you will get more exact information about your error, and usually more help.

An important error that people often make when creating documents manually is to omit translating characters used in expressing XML syntax when they include data elements. There are only five such characters, but if your data includes them the document will not parse, because the parser won’t be able to tell where tags and entity references begin and end. Here’s how I handled the problem in my pre-VFP 9 FRX2XML class:

PROC XMLTransform(tvElement)
  LOCAL lcElement
  lcElement = TRANSFORM(tvElement)
  lcElement = STRTRAN(lcElement, '&', '&'   )
  lcElement = STRTRAN(lcElement, '<', '<'    )
  lcElement = STRTRAN(lcElement, '>', '>'    )
  lcElement = STRTRAN(lcElement, '"', '"'  )
  lcElement = STRTRAN(lcElement, ['], '''  )
RETURN lcElement

Another way of resolving errors arising from XML control characters in data is to surround the data in a CDATA block, which tells parsers not to parse the contents of the block at all. In this way the offending characters are masked from the parser. I think this is less flexible, however.

Ensuring well-formedness (and validity if your scenario includes a DTD or a schema) is quite a bit of work if you create documents manually. In VFP you can also use CURSORTOXML() or the XMLAdapter. (Check the source to FFC’s XML ReportListener, which uses all three methods!)  If you use these VFP-centric tools, the resulting document will be more closely tied to your database structure than might be appropriate for the contents of your data. In XML it’s more important for the document to model your actual data correctly than for it to be elegant in a relational sense. That means that every node might have a completely different format – the equivalent of a relational database made up of thousands of tables with one record each.

A better alternative is to use an XML-standard parser object model to construct, or finish constructing, your document. For document construction, you will usually use the DOM (document object model) parser we’ve already mentioned briefly. Take a look at how the DOM protects you from the issues you have with manual document creation:

ox = CREATEOBJECT(“Microsoft.XMLDOM”)
oy = ox.createElement("xxx")
oy.text = "<>&'"+["]
? oy.text
* result: <>&'"
? oy.xml
* result: <>&'"

See the difference?  The XML for this newly-created element object is appropriately transformed to include the entity references. I have not yet appended this element into an XML document, using the parser – but when I do, this element will load, and become a valid part of a parseable document without any problems.

The DOM has rich syntax for creating nodes of different types and using them to build documents, of which the above is a small sample. Within the standard, other parsers beyond Microsoft’s may use different method names for the same functionality. Basically, however, they all work the same way.

If you take a look in your TOOLS\XSOURCE\VFPSource\Coverage directory, VFP Coverage engine’s GetStackXML(tcLog) method gives you a much more extensive example of DOM parser use. The code is not complicated, given a little time spent with an Object Browser and the MSXML parser. The only part that will be strange to you, and take some practice, is the XPATH syntax:

loThisNode = loXML.SelectSingleNode( ;
   ""+loParent.NodeName+"/"+ ;
   lcTag+"[@StackLevel="+TRANSFORM(FStack)+"]")

XPATH is a kind of query syntax, which you can use directly in the MS parser to select a single node or value (as used above, with the SelectSingleNode(cXPATHExpression) method) or to get a reference to a collection of nodes sharing some characteristics (use the SelectNodes(cXPATHExpression) method.

XPATH is odd looking but very powerful, something like “regular expression” syntax. In both direct use with a parser, as here, and within an XSL transformation, XPATH allows you to winnow through very complex document structures, using object references and variables, and setting multiple filter conditions on different levels of your query expression. You can apply the query expression either to an entire document or the portion of a document represented by a single node reference.

XPATH is odd looking but very powerful, something like “regular expression” syntax

The DOM syntax and object model has a few other concepts you may find strange at first. For example, it’s often difficult for people to realize that the text surrounded by two element tags (for example the word “whatever” in the document fragment whatever) is actually an object, a node of type text, not just a string. 

Consider the following Javascript snippet, which changes the text for a particular element if it has been previously filled, and otherwise just adds new text. If we were editing the document fragment above, this code would change the “whatever” text to some new text. If had no current text, this code would simply add it. In this code you can see how the text is being treated as an object and also how it exists as a child node of another document element:

var oNode =
  httpXML.selectSingleNode(tcQualifiedNodeName) ;
if (oNode) {
  var oTextNode = httpXML.createTextNode(tcNodeValue) ;
  if (oNode.hasChildNodes()) {
    // we should be at the bottom level
    // at this point, in this particular document
    // so there should only be one child, the text node.
    // If oNode could have other children, such  as
    // child elements, I might
    // write this code a little differently:
    var oOldChild = oNode.firstChild ;
    oNode.replaceChild(oTextNode,oOldChild) ;
  }
  else {
    // no text yet, just add it
    oNode.appendChild(oTextNode) ;
  }
}

Handling documents (parsing)

You can use the DOM parser described above to handle the documents you receive from partners in XML exchanges. Using XPATH query expressions, you can easily grab one or two crucial values from the document, and discard the rest.

Be aware, however, that DOM parsers load documents into memory, so this may not be the best choice for large result sets. This is not a necessary limit of the DOM, it is just a limitation of DOM parsers currently released. (It is easy to imagine a DOM parser swapping parts of a document to and from memory, just as VFP swaps cursors to and from disk depending on their size and available resources.)

Be aware, however, that DOM parsers load documents into memory

Normal VFP string-handling procedures might be more appropriate here, given our ability to handle long strings and less concern with the possibility that you might unwittingly do something “non standard” when you are parsing, rather than creating, a document. You can grab single values with an AT() search, or use low level file functions,  MLINE() (don’t forget the performance-enhancing _MLINE offset!) or ALINES() to scan through an entire document efficiently, inserting rows into a cursor as you go.

You might also decide to XSL-transform the document so the result was “cursor-shaped”, matching one of the formats understood by the XMLToCursor(cExpression|cFile, [cCursorName, [nCursorType | lStructureOnly]]) function or XMLAdapter. After you ran XMLToCursor  on your transformed document, it is easy to manipulate and store the data using normal VFP methods. This can be a very efficient approach, especially for documents containing multiple rows of results.

Document transformations and exchanges through XSLT

Once you can create and parse XML documents, you get to the crux of the problem: how do people share these documents? If everybody creates their own internal documents to fit their own processes, as I’ve recommended, what happens then?

Throughout this paper I’ve been saying “you have a document and then you transform it using XSLT to meet somebody else’s needs” or “you receive a document and transform it so it matches your requirements”. Why do I think it’s a good idea, what exactly does the transformation do, and how does it work?

The “why” part is probably easiest for me to answer: The truth is that people will create their own formats, and they will not cooperate on one standard format. Whatever the reasons, even when you have simple row-and-column-shaped data, people will not agree on how that data should be represented. Therefore, “what” you have to do is find a way to map, or transform, dissimilar types of XML, not try to legislate consistent schemas.

So… get used to it. You’re going to map between XML exchange partner formats, even when they are doing exactly the same kind of job (showing rows and columns in a table). When they are doing something more specialized than showing rows and columns, the mapping gets a little more complicated to do but it is just as necessary, if not more.

The “how” is harder to explain, and would take more space and time than the rest of this paper (!). However, you already benefit from XSLT’s abilities when you use VFP 9’s HTML Report Listener’s base output. You will see more evidence of these abilities as time goes on.

It just provides a conduit – both for data exchange and data presentation

XML is not the end of the journey

XML isn’t really a product or an end result, in itself. It requires some application, some device, and some intelligence (yours) to give it meaning. It just provides a conduit – both for data exchange and data presentation. 

But, in spite of that caveat, it has so much to offer! I expect, for as long as I work with computers in the future, I’ll be working with XML. When I’m using VFP to extract the meaning and apply the format to XML, I know I’m working with two well-matched technologies, and one of the most creative partnerships that the world of computers can offer any developer today.