Every Document is an Expression: The Reach of Mathematica 6

This notebook is Copyright ©, 2008 Scientific Arts, LLC,  all rights reserved.  It may be redistributed only in full and in its original form without modification; and it may not be used for any commercial purpose without permission from Scientific Arts.  A non-exclusive royalty-free license is given to reproduce and use all the of the Mathematica code in this notebook for any purpose whatsoever, as well as any of the ideas.   No guarantee whatsoever is given as to the  accuracy or functionality of anything presented in this document.  Use at your own risk. 

Enjoy: I hope that it's useful. For further information on  A WorkLife FrameWork, please click on the Scientific Arts banner above.

Version of 24 July 2008

Mathematica 6 exposes a huge landscape of possibilities for automatic report generation, the creation and maintenance of documents with active content, and the development of document-oriented applications.  In this brief review I touch on a few aspects of Mathematica 6 as a system for creating programmable, active, document-oriented applications.

This review is based on Mathematica 6.01: some observations may change as new versions of Mathematica are rolled out.

. Introduction

In this brief review, for lack of a better term, a Document is a very general concept covering a very large space of possible Mathematica document types along with external non-notebook files and data sources.  Typically a Document will contain a set of Mathematica Notebooks with arbitrary user interface elements in them, arbitrary interdependent Dynamic contents, arbitrary graphical and textual content, and arbitrary code that creates and modifies content. And, with this definition, a Document will also contain and interlink with external files and applications. The programmatic aspects of generalized documents in Mathematica and the underlying form of many of their elements as Expressions opens up a conceptual space of broad potential and scope for the creation of systems of documents and document-oriented applications.

This has been true prior to Mathematica 6, and my Mathematica add-on—"A WorkLife FrameWork" http://scientificarts.com/worklife— is, from this perspective, a broad document-oriented application that enhances the reach of Mathematica. This application was built with Mathematica 5.1 and 5.2 and was the first Mathematica add-on package available that was compatible with Mathematica 6 upon its release.  With Mathematica 6, the continued development of "A WorkLife FrameWork"—and the potential for new document-oriented  applications—can now move into areas that were previously inaccessible.

It is now open season for creativity and discovery in the field of document-oriented applications.  Mathematica is ideal for both the creation of such applications: both ones driven by Mathematica as its engine, and the prototyping of such applications that may ultimately be deployed with a technology other than Mathematica.

These opportunities are enabled by several aspects of Mathematica 6: its very wide variety of new user interface elements, its new Dynamic functionality, and a document programming language that extends and sits above the much lower-level functions that first appeared in Mathematica starting at version 3.

This review focuses on the document generation tools in Mathematica 6: Wolfram Research has created these as an essential complement to the new user interface generation tools and Dynamic functionality, and their full power is in concert with these.

For historical reasons there is still the impression amongst non-Mathematica users that Mathematica is "a symbolic computation tool" or a "technical computation tool."  These, of course, are sub-aspects of Mathematica.  What is not appreciated in general is the Mathematica's reach well beyond these narrow scopings.  Understanding and exploring document-oriented applications is one way to get a sense of Mathematica's breadth of scope.

. The Scope of Mathematica and the New User Interface Programming Layer

Something that we, as Mathematica users, often hear is that, in Mathematica, “everything is an expression.” This elemental design decision for Mathematica is a key to what gives it its broad scope and extensibility.  It is also what allows Mathematica to represent entities of arbitrary complexity, whether in historically traditional mathematical systems, or arbitrary other abstract systems, including document-oriented applications.

In versions of Mathematica prior version 6 accessing and programming entities like Cell and Notebook expressions, as well as the then limited set of user interface elements mainly Buttons programmed as ButtonBoxes), required a strong familiarity with their subtle low-level details.

In Mathematica 6 there is now a new layer for FrontEnd programming that sits above the low-level elements (which typically are explicit Box constructs).  This is an enabling design choice on Wolfram Research's part because it gives users powerful,  “simple,” and enabling access to these constructs.

Many functions that exist at the low-level as Box structures, and which are of the form fooBox, now also have counterparts simply of the form foo. Thus, StyleBox now has the counterpart Style and GridBox has the counterpart Grid. And, in many of the cases where those Box functions were present in earlier versions of Mathematica, they are considerably more versatile in their latest incarnations.

. The Mathematica Document Programming Language (MDPL)

From the perspective of this review, one thing that is new in Mathematica 6 is the high-level language-within-the-language for creating and programming documents. While one might engage in debate about what determines whether a function should be included in this MDPL, here is a list of many important top level ones that, in effect comprise a core of the MDPL.

({{"Documents", "Notebooks", "Cells"}, {"CreateDocument", "DocumentNotebook", "TextCell"}, {"CreateWindow", "PaletteNotebook", "ExpressionCell"}, {"PopupWindow", "DialogNotebook", "CellGroup"}, {"Deploy", , }})

({{"Cell\nLayout", "Graphic\nLayout", "Views", "Events", "UI\nelements"}, {"Grid", "GraphicsGrid", "FlipView", "FrontEndEventActions", "Button"}, {"Row", "GraphicsRow", "MenuView", "NotebookEventActions", "ActionMenu"}, {"Column", "GraphicsColumn", "OpenerView", "CellEventActions", "RadioButton"}, {"Item", , "PopupView", , "Dynamic"}, {"Spacer", , "SlideView", , "Many, many,\nmore..."}, {, , "TabView", , }})

. Using the language

Prior to Mathematica 6, you might create a cell and its associated content in a way similar to this (MakeBoxes does not evaluate its argument while ToBoxes does),

aCell=Cell[BoxData[MakeBoxes[1+2+1/x]],"DisplayFormula"]

and then display that cell in this manner,

1+2+(1/x)

In version 6 the function ExpressionCell (there is now also a related function, TextCell) will take care of the lower-level formatting constructs automatically (here the automatic expression evaluation is suppressed using the new function Defer):

1+2+(1/x)

Here is an example of a simple reporting function written in the way one would have done prior to version 6.  It involves explicitly making use of low-level functions such as TextData, BoxData, and ToBoxes as well as care, using string manipulation, in making sure that certain items do not evaluate:

Here is an example of the output:

FibonacciReportCell[54]

And here it is formatted with CellPrint:

The numerical value of Fibonacci[54] is 8.626757127199998`*^10

Here is a version of this function using some constructs from Mathematica 6—low-level Box constructs are not needed in its design:

(The choice of option, CellFrameMargins -> 2, is so that the CellDingbat doesn't print over the CellFrame.)

Executing this creates an output that is a representation of the cell itself.

FibonacciReportCellV6[54]

Note that the CellDingbat does not appear in the Output cell.  This is an important point.  Since the result of executing the TextCell is displayed in an Output cell, it does not exhibit its full formatting as it would when printed as a cell in its own right.  Use CellPrint to create a fully formatted version:

The numerical value of Fibonacci[54] is 8.626757127199998`*^10

Here is a prototypical example of a "report" generating function built using the new function DocumentNotebook (another approach is to use CreateDocument).  The function DocumentNotebook can take any of the options that Notebook can, and these are used to adjust details of the resulting document.

Here is an example. DocumentNotebook creates an in-notebook representation of the document in an Output cell.  As above, some of the formatting is not exhibited in the environment of the Output cell.

FibonacciReport["David",54]

By clicking on the  icon, a full notebook will open with the contents of the document. Here is a screen shot:

In the code for FibonacciReport above I used DocumentNotebook to create an in-notebook representation of the document.  This can be bypassed and you can go directly to the independent notebook by using either CreateDocument instead of DocumentNotebook in that code, or by making use of CreateWindow in this fashion:

CreateWindow[FibonacciReport["David",54]]

. Possibilities for confusion

There are simple approaches to translating from the document programming layer to the lower Box-oriented layer.

For those users that are most familiar with the low-level functions that were available for document programming prior to version 6, there is the possibility of confusion when combining these with the new interface layer.  For new users though, this will usually not be an issue. But it is important to remember that, generally, Mathematica is using Box-oriented expressions at the lower-level.

NotebookWrite is still the means of adding material in to a notebook;  NotebookRead is still the means by which material is read from notebooks; NotebookFind and NotebookLocate are still the means by which material is found and selected in notebooks; and SelectionMove is still the means by which on navigates around within a notebook.

The place where one most often has to make a translation is when one wants to programmatically add new material to an already existing Notebook using NotebookWrite.  In this case one will often have to use ToBoxes or MakeBoxes, and there can be some confusion here.

For example, this doesn't work:

But this does:

So, the general principle here is to convert to Boxes when making use of NotebookWrite.

There is however a case that is not covered by this. The following does not work (the content in the resulting notebook is, internally, an InterpretationBox, not the simple TextCell that is desired) :

Neither  ToBoxes nor MakeBoxes is appropriate for the translation here. What is needed is either an automatic translation mechanism added to the behavior of NotebookWrite (and a proper design for this should then automatically translate things like Grid, Row, and so on...), or a function, perhaps called ToCellExpression, that acts on TextCell, ExpressionCell, and CellGroup to translate them properly so that they can be used in the second argument of NotebookWrite.  Here is an example of such a function (it is limited here to TextCell and ExpressionCell):

With this, the following does work:

In fact, there is a function that is deeply buried in an internal Context that is used by Mathematica 6 to do this sort of thing and which uses the undocumented option StripOnInput. (As with all undocumented “features” this symbol should not be depended upon for any future version of Mathematica until it is officially documented with a usage message.)  Here is its definition (with the context path removed):

The document programming process

One thing that is important to keep in mind, and which we have seen in this review already, is that when functions like TextCell, ExpressionCell, and DocumentNotebook are executed, the result appears within the environment of an Output Cell.  Because of this, what you see in the output may not be in quite the same format as you might expect.  Thus, the process of designing and debugging, for example, a TextCell should involve using CellPrint. Similarly, if you are working with DocumentNotebook, you should use CreateWindow to check the final form.

There is a good argument for this behavior of the in-notebook representation of a TextCell or ExpressionCell within Output Cells: they are creating inline cells within an Output Cell and, therefore, will not show all of the attributes of the cell that they are representing.  However, the fact that DocumentNotebook does not fully represent the form of the cells that it contains is probably not what is desired: the output should be drawn as if it were a mini-window inside the main window.  I expect that the future development path for this will be to have this behavior.  And for later versions of Mathematica I expect that fully formatted notebooks within notebooks will appear.

. Some User interface elements

In a way this review cannot be written linearly, though the constraints of linear print media require it.  There are two topics that I want to cover in brief—user interface elements and Dynamic functionality—which are intimately intertwined. I touch on each, but their real power is in concert with each other. And, in fact, my touch is very pointillistic and I expect the reader of this review to understand that these bits and pieces and others that I am not covering—in the unified system that is Mathematica 6—are examples of what one can make use of to create a document-oriented application driven by Mathematica.

. Palettes

For the purpose of this review I am only going to exhibit some of the many user interface elements that are new in Mathematica 6 in the context of accessing multiple documents at once.

This simplest way to do this is through palettes, and the creation of palettes in Mathematica 6 is very much easier than it was in prior versions. A very simple example is to create a palette that allows you to access the set of notebooks that are currently open.

Note the use here of the new functions Style, Button, PaletteNotebook, CreateWindow, and CurrentValue.  Functions such as PaletteNotebook can take on any of the options that the lower-level function Notebook can, and in the function CurrentlyOpenNotebooks I illustrate some of these. This emphasizes the point that many of the new document creation functions in Mathematica 6 both define a new User Interface Programming Layer while also maintaining a bridge to lower-level notebook programming options.  The advantage of functions such as PaletteNotebook is that they take care of many of the options settings that needed to be set at a low-level in earlier versions of Mathematica. Also PaletteNotebook is properly integrated with functions such as TextCell and ExpressionCell.

This palette is also an example of a mini-application that allows you to navigate amongst your open documents.  A more sophisticated version of this would use Dynamic constructs to continuously update the palette as documents are opened and closed.  Another alternative might have the buttons contain, rather than the name of the Notebook, the abbreviated contents of its first Cell, or a wide variety of other possibilities..

. Menus

An alternative approach to the example CurrentlyOpenNotebooks, is a menu (base on the mathematica 6 function ActionMenu) that you could embed in an application to access currently open notebooks.  Of course this is the function of the Window menu item in Mathematica. But this is a prototype for accessing particular sets of items, whether they be open notebooks, PDF files on your hard disk, web pages, or other items of very general sorts.  And the concept that Mathematica reaches out to interact with all sorts of documents finds a simple expression in such examples. (I comment on deeper expression in the next subsection.)

One example of this is the creation of a pull-down menu that accesses a list of web locations: a bookmarks menu.  It makes use of a special form of NotebookLocate which will open a supplied URL in your system's default web browser.

And here is an example:

ActionMenuWorkLifeWolframWombatsWikipedia"Bookmarks"

And here is what it looks like when clicked on:

This example can be generalized to access other favorite documents through their default applications. (The approach is essentially the same as shown in the code for WebBookmarksMenu; however, where a URL is supplied as an argument to URL, a string containing the file path to the document is substituted.  The correct form for the filepath string depends on the operating system—for Windows it is the full file path itself and for OS X and Linux/Unix it is of the form "file://"<>filepath—and each space in the file path must be replaced by its escaped string, "%20". This is the approach that I take for the function OpenFileOrDirectory in "A WorkLife FrameWork.")

The key point of this simple example is the use of new User Interface Elements to manage and create applications that access a variety of documents—some of which may be Mathematica notebooks, but many that may be of a variety of other types. Further processing can take place upon accessing the documents: for example one can use Import to bring the document into Mathematica and then create the final content by processing the imported material and visualizing the final result (possibly in an applicatoin other than Mathematica by first saving the material to disk using Export).

. Importing, Exporting, and Curated Data

Reaching out to various document types and opening them in their default applications is one simple possible aspect of a document-oriented application in Mathematica.  There is also a converse version of this which is to bring such items into Mathematica to process and then display the results of that processing. In addition, one can use Mathematica to create the content that one wants to export out of Mathematica to a document of a different application type.

For many versions of Mathematica up to the present the functions Import and Export (as well as ImportString and ExportString) have been available for purposes such as this..  In Mathematica 6 the types of data (and hence documents) that are available to Import and Export have been considerably expanded.  The lifeblood of the sciences is the ability to access and create a very wide variety of data files, and Wolfram Research fully appreciates this (and also the fact that the data files may reside at arbitrary locations on the Internet or other network).  This is also essential for document-oriented applications, and increases the scope of what can be created with these.

While the file types that Import and Export can handle have been considerably expanded, Wolfram Research has created an additional direction for data access in Mathematica 6.  These are the Mathematica 6 functions that access "Curated Data."  And generally these functions have the form fooData (though there are other functions in Mathematica that have this form).  Here are the functions of this sort that are available at the time of the writing of this review:

({{"Curated Data Functions", , , }, {"AstronomicalData", CountryData, "IsotopeData", "PolyhedronData"}, {"ChemicalData", "ElementData", "KnotData", ""WordData""}, {"CityData", "ExampleData", "LatticeData", " "}, {"ColorData"
, "FinancialData", "ParticleData"
,  }})

Here is an example:

AstronomicalData["Mars","Image"]

Each data item has a set of available "Properties".

AstronomicalData["Mars","Properties"]

In a sense, these various Curated Data are meta reports—in effect free-form documents—on particular subjects.  So, again, this addition to Mathematica 6 expands the space of possible document-oriented applications.

. Different Views

There are a number of new functions in Mathematica 6 that provide ways of viewing, one at a time, distinct sets of notebook content.  These functions typically have the form fooView:

?*View

Here are three example uses of these involving the function FibonacciReport.  The first uses TabView,  the second, SlideView, and the third, FlipView.

TabView

In this example I am creating a set of documents which are each directly accessible from the tab buttons at the top of the TabView display.  These buttons are tabbed to indicate the argument specifying the "report" desired: the second argument supplied to FibonacciReport.

Note that this example shows an embedding of DocumentNotebooks within a DocumentNotebook.  Each separately is accessible by clicking on its own  icon.  The outermost  icon generates the multi-document "report" in a separate notebook with the full TabView available to make use of. And any of the inner  icons generates an individual "report" for the specific value of the second argument to FibonacciReport that is currently  being displayed in the TabView.

DocumentNotebook[TextCell[
TabView[#->FibonacciReport["David",#]&/@Range[100,150,10]]]]

In a way, TabView is like Random Access Memory.  You can go to any "report" directly by clicking on its associated tab.

SlideView

In contrast with the metaphor of Random Access Memory, SlideView is like a tape: you access the different "reports" sequentially, one at a time, by making use of the slide control.  As in the case of TabView, each internal  icon opens a new notebook containing its associated content; the outer  icon opens a new notebook containing the full SlideView content.

DocumentNotebook[TextCell[
SlideView[FibonacciReport["David",#]&/@Range[100,150,10]]]]

FlipView

In the case of FlipView, each of the "reports" is again accessible sequentially, though only in the forward direction (looping cyclically through the list).  One moves from one "report" to the next by clicking on the "report" itself.  Hence, the content of the "report" in  FlipView is not interactively accessible—in effect it is a simple way to limit interactive access to the material in the reports.

Therefore, in contrast to the cases of TabView and SlideView, in this example only the outermost  icon opens a new notebook containing its associated content (the FlipView material), and in that new notebook none of the  icons are active (because they are contained in the FlipView).  This is not a bug: it is as designed.  So, if I was designing FibonacciReport for a FlipView, I would be better off by simply exhibiting the cells comprising each "report" in a Column rather than within a DocumentNotebook.

DocumentNotebook[TextCell[
FlipView[FibonacciReport["David",#]&/@Range[100,150,10]]]]

. Dynamic Active Documents

The example above is very simple, and illustrates the point that there are now higher-level functions that are much more accessible for developing code that generates and manipulates documents. The scope of what you can do, however is enormous.  Part of the reason for this is "everything is an expression" design of Mathematica, part is due to its Dynamic functionality, and part due to the presence of a full suite of user interface elements.

The example of FibonacciReport that we focused on so far creates a fixed report for each set of argument that are supplied to it. And in the examples of the preceding section the documents for multiple FibonacciReports were precomputed and then accessed using the user interface devices of TabView, SlideView, and FlipView. A paradigmatic example of an active document is a simple calculator that allows you to change the input parameter value arbitrarily. The following function, DynamicFibonacciReport, does gives an example of this making use of the functions Dynamic and InputField, and the scoping function DynamicModule.

The result of executing DynamicFibonacciReport is a "simple calculator" where, based on the value placed in the InputField, material in the "report" is automatically updated.

DynamicFibonacciReport["David"]

This an aspect of this example is an example of how Dynamic parameters in Mathematica 6 can be embedded throughout a notebook to create active documents that change their state depending on parameter values. (In the simple case here, the text in the document responds to the changes in the value in the input field.) Not only can documents be created by designing the expressions that comprise them, but these documents can morph and evolve arbitrarily in response to user-initiated changes within them or by other events associated with the document (via the functions FrontEndEventActions, NotebookEventActions, and CellEventActions).

One further point that is worth mentioning in this context is Wolfram's free Mathematica Player application (http://www.wolfram.com/products/player/), intended for people who do not have Mathematica.  This is Wolfram's replacement for what was formerly knows as MathReader.  The difference is that Mathematica Player permits active notebooks to function—in effect allowing you to deploy applications for use with users that do not have Mathematica. Currently this is only possible with examples that you submit to the Wolfram Demonstrations site (http://demonstrations.wolfram.com). Wolfram has a process in place whereby they digitally "sign" such examples before placing them on the site. It is hoped that this process will be generalized in the near future to allow users with a valid Mathematica license to sign their own examples and distribute them themselves.  Also Wolfram is planning to release Mathematica Player Pro, which will have an extended fuctionality (http://blog.wolfram.com/2007/05/mathematica_player_so_much_more.html).

. Bugs, of Course...

Of course in this short review I have only touched down at a few points in the huge space of possible document-oriented applications that are possible with Mathematica 6. My trial by fire with some of the new features of Mathematica 6 was with bringing the 50,000+ lines of (mostly FrontEnd) code for "A WorkLife FrameWork" into Mathematica 6 compatibility. It is a testament to the responsiveness of the developers at Wolfram Research that, as I worked though this task in the context of changing alpha and beta versions of Mathematica 6, they responded to bug reports quickly.  I'd like to think that this helped with increasing the stability of Mathematica 6, but from my perspective it made my task much easier.  And the process of transitioning 50,000+ lines of code was surprisingly less of a task than I had originally expected.

In the course of writing this review I came across a number of bugs, and it is my practise to refine each bug as much as possible and to report it promptly to Wolfram Research. In some cases what I though were bugs, were intentional design elements, and figuring them out helped with my understanding of the associated Mathematica functions.

When working with the new features of Mathematica 6 you will certainly encounter bugs This is normal for any very large software system—and even more so for one that has experienced the degree of change that has occurred in Mathematica 6. With the prospect of combining the new elements of Mathematica 6 in many novel ways you will certainly encounter situations that are not in the testing suite that Wolfram runs on Mathematica daily. I strongly encourage you to report all bugs that you come across to the Mathematica developers.  The correct email address for this is support@wolfram.com.

. Tutorials and Guides

The new Documentation Center in Mathematica replaces the former Help Browser (which is still present to display legacy package documentation).  Within the Documentation Center there are a large number of tutorials and other materials.  Here is a function for accessing them directly and following it is a table of tutorials and some other documentation items relevant to this review.

({{"Tutorials", "Guides"}, {"tutorial/NotebooksAsDocuments", "guide/NewIn60NotebooksAndDocuments"}, {"tutorial/GridsRowsAndColumns", "guide/DocumentGeneration"}, {"tutorial/TablesAndMatrices", "guide/LayoutAndTables"}, {"tutorial/FormattedOutput", "guide/DynamicInteractivityOverview"}, {"tutorial/ImportingAndExportingData", "guide/ListingOfAllFormats"}})

About the Author

David Reiss (http://scientificarts.com/staff.html) has been using Mathematica since it was born and had watched it through its gestation and birth. He has used Mathematica for most of his technical and other creative work and, like many Mathematica users, first started using it as a computational scratch pad.  He is an accredited Mathematica consultant and is splits his time between various consulting opportunities and the continued enhancement of "A WorkLife FrameWork."

Over the years he came to realize the beauty and power of Mathematica's language, its unified design, and the potential of its FrontEnd. This inspired him to start to create a set of tools that would be useful in his everyday work—whether in his technical computation, research, and writing tasks, or in his broader creative work.  This set of tools eventually evolved into "A WorkLife FrameWork" (http://scientificarts.com/worklife) and now he is lost without it.

David Reiss