Format Introduction

As you have hopefully seen from some of the examples included with Sloup, a transaction with Sloup consists of a sequence of command lines sent from the desktop. The simplest sequence consists of:

  1. a soup name
  2. a entry specification: the order and type of data values
  3. the data itself: a line per record
  4. end of transaction

There are several variations on this: other commands/lines that can be included; and a "line" can consist of several lines, but before getting into that complexity, let's look at a sample transaction in slow motion to see how Sloup's status messages reflect where it thinks it is in this sequence.

you tap Connect in Sloup (and open connection from desktop)
Sloup displays "Connected", "Waiting for Soup Name"
you send a soup name
if the soup exists (or if you've asked to create a soup) and it is successful, Sloup displays "Waiting for EntrySpec". If it is unsuccessful, Sloup returns to "Waiting for Soup Name"
you send an entry specification (entryspec)
if the entryspec compiles successfully, you've just created a NewtonScript frame object that describes the data (order and types) and Sloup displays "Waiting for Data"; if unsuccessful, Sloup returns to "Waiting for Soup Name"
you send a data line
if the data is correct (number of fields, types of values), Sloup should successfully map and convert the data using the entrySpec, store it in the soup, and increment the Entries: count; if unsuccessful, you may see an error notification. Sloup continues to wait for data.
you send "BYE!"
Sloup finishes with the current soup, plays a sound (if selected), and returns to "Waiting for Soup Name"

Here is a preview of some possible variations in the above example of a Sloup file/transaction. These will be explained in more detail later.

soupName
besides an existing soup like Notes or Names or one that you or an application created earlier, there can be several special names and commands. For example, you can use "Sloup" to change global settings such as keyboard mapping or field delimiters; DUMP! lists names of soups; EVAL! evaluates and prints a NewtonScript expression; Package (on NOS 2.x) sets up for import/export a single package; you can create a soup by appending the soup name with !, followed by an array of index definitions.
entrySpec
this is a format frame. in addition to defining the format for incoming data, an entrySpec can be used to define global settings (for the Sloup "soup"), to define output format for exporting (DUMP!) data, and to limit specific data to DUMP (a very limited, non-general query capability)
data
sometimes an initial data line may actually be a command, such as ERASE! to erase entries, or REPLACE! to define how to replace (rather than just add entries). For Notes, a line is raw text concatenated with other lines (until ----- is encountered) to create a single entry; empty paragraph entries are ignored. For other soups, a line is a sequence of tab-delimited fields; a single line represents an entry, unless a line continuation \ is used; empty lines are ignored.
BYE!
a DUMP! might occur here to cause entries to be exported (according to earlier entrySpec); a BYE! might be followed by an array of definitions for Soup Indexes (created at end of process, rather than at beginning after soupName), e.g.,
MySoup![]
{....}
...
BYE![{structure: 'slot, type:...}

This allows the data entries to processed faster; however, when indexes are added at the end, the total time is greater (since entries have to be accessed again). Sloup only adds the indexes if there are none present, e.g., it assumes you have removed the soup earlier, or are using the same existing indexes. (To add just some indexes to some stores involves too much code that relies on undocumented features of how soups are represented). I would generally recommend that you add indexes first, but now you have a choice.

Format Summary

Since there are some differences between import and export, and between Notes and other soups, here is a summary of the four basic kinds of transfer supported by Sloup:

  1. Import text to Newton
  2. Export text to desktop
  3. Import data to Newton
  4. Export data to desktop

Import text to Newton

Here is a simple example of importing text into the Business folder.

Notes
{labels: 'Business, viewFont: 10241}
text
a second line in same paragraph
-----
another para
-----
-----
BYE!

Notes is the soup name for the notepad; 2.x users can also use Newtworks to transfer plain text documents to Newtworks.

The entrySpec for Notes/Newtworks is used to specify several default values.

labels
labels is a symbol corresponding to a folder destination, e.g., 'Personal, 'Business, or nil or 'nil (for Unfiled). If the label contains non-alphanumeric characters, you need to wrap the symbol with vertical bars, e.g., '|two words!|. If you include a symbol for labels slot that does not (yet) exist, be sure to Edit Folders later on your Newton to add it -- otherwise, the only way you'll see these notes is via All Notes
viewFont
If viewFont is not specified, then the Note is created with the current default font set by the user in Styles. If viewFont is specified, it should be a font specification, e.g., a coded integer like 10241 (userFont10) or font frame description like {family: 'espy, face: 0, size: 9} -- see Newt or NTK documentation. I have included viewFont: 10241 in the .nwt files to be consistent with Newt's default
height
If height is not specified, then Sloup creates each Notepad entry with its calculated height. You can specify a fixed height, e.g., {labels: 'Personal, height: 100}. If the Notepad clips a long para, hold down on the note separator bar in the NotePad for several seconds, then drag it downwards (it disappears) and then let go -- the separator should jump to the end
title
NOS 2.x: Sloup uses the first line of the text as a separate title for the Note. Note: NewtDevEnv currently expects the header line to be the first line of the text note (and not stored separately in the title); if for some reason you want to specify a title for Newt files for 2.x, you should add a separate title line (possibly a duplicate of the header line)
class
NOS 2.x: 'list (outline), 'checkList or 'paperroll (default). For lists, tabs in the data lines are used to indicate level.

Notes entrySpec import examples:

ERASE!

If you know what you are doing (and have backed things up first), you can insert an ERASE! command after the entrySpec. This erases all entries from the specified soup; or if you've specified labels in your frame spec (for Names or Notes), it erases just the entries in that folder; if a field has labels, you can specify the symbol _all to erase entries in all folders, e.g., {labels: '_all}.

ERASE! can be handy especially if want to get rid of many existing entries before adding new information (also see REPLACE! for selective replacement). Warning: this does not give you a dialog box asking to Confirm erasing of entries!

For example, if you edited Newt sources for your project on the desktop, and wanted to replace these completely on your Newton, you could do something like:

Notes
{labels: 'Personal, viewFont: 10241}
ERASE!
myApp
{_proto: protoApp,
}
-----
myApp+aButton
{_proto: protoTextButton,
viewBounds: RelBounds(10,20,50,15),
}
-----
BYE!

REPLACE!

Although a soup is like a database and may have an index that treats a particular slot as a "primary key", Sloup does no checking to eliminate duplicates. So, it always adds new entries. Instead of erasing an entire folder and then storing entries, you could instead specify a slot to identify an entry; if an entry with the same name already exists, the new entry would replace the entry. You can also remove individual entries with a soup utility.

For Notes, title is the only slot supported, so REPLACE!title would replace any note that has the same title -- this works on 1.x by using the first line of the text as the "title". For Newt, the first line of a note is typically a unique identifier, e.g., "MyApp+button1"; for Newt's Cape, the first line could be a comment indicating the filename, e.g., <!--foo.htm-->

Note: all entries are added to the default store, e.g., on your memory card if "Store new items on card" is checked. If you "replace" an existing entry, the old entry is actually removed from its existing store, and a new entry is created on the default store.

DUMP!

You can Export text to desktop and Export data to desktop by using DUMP!. You can follow DUMP immediately with a frame, i.e., DUMP!{...}) to specify parameters related to delay, print function, field, record and soup (eof) delimiters. Final field always is followed by a field delimiter (previously, only record delimiter). Here are the slots you can specify (along with defaults):

delay
number of seconds to pause. default: 0 (no delay); handy for setting up a log/capture file
printFunction
NewtonScript function definition with single parameter. default: Functions.SPrintObject
fieldDelimiter
string to output after each field. default: tab ("\t")
recordDelimiter
string to output after each record. default: none
soupDelimiter
string to output after all records. default: "BYE!"
noAddDelimCR
default: nil. controls whether to add crlf after recordDelimiter or soupDelimiter (default or user specified) -- DUMP!{..., noAddDelimCR: true} means do not add cr. (default adds it)

Note Separators

Each note is separated by a line of at least 5 dashes, i.e., -----. The last note separator is optional: BYE! will also terminate the last Note. Empty notes are not saved to the Notepad.

Examples: Notes import

For NOS 2.x systems, you can import text into outlines and checkLists using class. Each line is an entry; tabs indicate level. For example, this would add two checkLists to the Unfiled folder:

Notes
{labels: nil, class: 'checkList}
a checklist in Unfiled
	second level
		a next level item
-----
another checklist
	indented1
	indented2
another main entry
	indented1
BYE!

Add two lists to 2.x Business folder (lists are called outlines in the New picker):

Notes
{labels: 'Business, class: 'list, viewFont: {family: 'casual, face: 0, size: 10}}
an outline in Business
	indented item
		another indented item
-----
another outline
	indented1
	indented2
another main entry
	indented1
BYE!

Export text to desktop

For dumping (exporting) text to the desktop from Notes (or Newtworks for 2.x users), you can specify a folder, and several fields:

labels
if not provided (or '_all), Sloup dumps all folders; otherwise, it's a symbol for an existing folder (or nil or 'nil for Unfiled)
title
if present (type: "string"), Sloup outputs the title of the Note (for NOS 2.x) or a copy of the first line (1.x)
_modTime
if present (type: "dateTime"), Sloup outputs the last modified date using the date format specified
class
if not present, Sloup dumps all notes, including lists. If present (NOS 2.x), Sloup restricts to just that class of note,i.e., 'paperRoll, 'list, 'checkList

Note about field order: Sloup does not output class, and it defers title until just before the text; otherwise, fields (labels and _modTime) should appear in the order specified. Currently, entries are dumped from all mounted stores (internal, card,...).

Examples: Notes DUMP

DUMP! all Notes in Business folder

Notes
{labels: 'Business}
DUMP!

For NOS 2.x systems, that specification would have included all classes of Notes. You can also select just lists (outlines) or checkLists.

Dump just Unfiled checkLists:

Notes
{labels: nil, class: 'checkList}
DUMP!

Dump just plain notes (no checkLists or outlines) from all folders, along with titles and modification dates:

Notes
{class: 'paperRoll, title: "string", _modTime: "dateTime"}
DUMP!

Dump just Business outlines (i.e., lists):

Notes
{labels: 'Business, class: 'list}
DUMP!

Import data to Newton

Importing general data is more complicated than for Notes: there can be many fields and different data types, depending on the soup --these fields and types may or may not be documented. Each data entry is represented as single line of tab-delimited text (unless there are line continuations). For example, see CasioNam.tab. Blank lines are ignored.

Sloup assumes that no more than approximately 20 fields are embedded at one level in one frame (you could have more than 20 fields total if you have nested frames or arrays); the order of values in a line must correspond to the order of slots in the frame spec. For small (<20) number of slots, the frame spec maintains a linear order when it is compiled; however, for larger number of slots this ordering is not preserved (I have no plans to redesign Sloup to avoid this limitation).

Since exporting uses the same entrySpecs, I will discuss these and provide examples a little later.

Export data to desktop

Assuming you have a valid entrySpec, export (DUMP!) is quite simple. The following example works if you have already created "testSoup" (if not, go to the later Soup Indexes section; then come back)

TestSoup
{a: "int", b: "string"}
DUMP!

For example, see CasioNam.dmp. Entries are written to terminal emulator as tab-delimited lines in frame specification order. (If possible, you should have your terminal emulator preserve tabs, wrap lines (rather than overwrite characters), and save/capture to a log file). Any data fields with text values that contain newlines will be transmitted as is -- you may need to re-edit results into a true tab-delimited text format, or modify field delimiters.

If you do not know the fields and value types for a soup, you can specify an empty frame {} as the entrySpec. In this case, Sloup will dump an pseudo-entrySpec based on the first soup entry, and then dump all remaining entries with respect to this entry. (This works only if levels of the soup structure are less than 20 in length, and subsequent entries are similar in structure to the first entry).

TestSoup
{}
DUMP!

Although you can use Sloup to discover soup structure, I would recommend that you use documentation wherever possible (e.g., additional Sloup docs for built-in soups like Notes and Names), or soup utilities (e.g., StewPot) or NTK Inspector. And of course, backup your Newton to avoid possible damage.

Special "soups"

There are several special names that can occur in the soupname position (first line of a transaction) and are handled specially.

DUMP!
If you do not know the names of the soups on your Newton, you can use DUMP! in place of the soupname and Sloup will list names of soups on all stores (there may be redundancy)
EVAL!
EVAL! can be used to evaluate and print arbitrary NewtonScript expressions, e.g., EVAL!3+4. You will have better print results and error reporting if NewtDevEnv is installed.
Package
On NOS 2.x, you can list packages, or import or export an individual package. This does not actually access the "package soup" directly but uses a different API. Import/export examples later under binary:packageEntry. To list package names:
Package
{}
DUMP!
Sloup
If the first line is Sloup (or "Slurpee" in older scripts), the entrySpec is used to specify global settings; it returns to waiting for a soup name -- there is no data. For example, if you want to change some global settings and then restore defaults, here's how it might appear:
Sloup
{fieldDelimiter: ",", stripQuotes: true,...}
Names
{....}
...
BYE!
Sloup
{}

Or if you had earlier made global changes earlier and wanted to ensure that defaults were used:

Sloup
{}
...

Here are the global fields you can set in a Sloup entry:

fieldDelimiter
default: tab, i.e., "\t". string with new field delimiter, e.g., ","
stripQuotes
default: nil. if true, strip any beginning and ending " from each data field
translationTable
default: nil. specifies an array of pairs of strings -- the first is a pattern that you want to replace in String values, the second is the substitution. This can be useful for international users who want to specify special characters but whose character encodings are different from the Newton's. These can be single unicode characters (or anything). These substitutions are applied starting at the beginning of the list. for example, translationTable: ["\u2022", "\u00D1"], would substitute an "enye" character for the first one. Currently not used for DUMP. Note: translationTable is applied to all lines, so this affects Notes as well as tab-delimited data (and may affect other commands, specs)
totalRecords
default: nil. a number. If you know in advance how many records you have, include a number, e.g., totalRecords: 500. Then, "Entries:" will display 1/500, 2/500, etc.; a gauge also appears
convertUnicode
default: nil. controls whether to convert \u unicode sequences.Use \u to start a sequence (must be multiples of 4); sequence ends with \u or end of each line. This might be more convenient that creating a translationTable. One user is already using this to transmit Chinese characters to the Newton. For example,
Sloup
{convertUnicode: true}
Notes
{labels: nil}
first line
upsidedown question mark \u00BF\u embedded
enye at end of line \u00F1
\u00A700A52126\u -- a few in a row
in lowercase: copyright \u00a9\u yen \u00a5\u
BYE!
Sloup
{}
debug
default: nil. if true, Sloup will print the unicode (4 char hexadecimal) code for each keystroke. This can be handy to determine what code is actually getting to the Newton. When specifying a code, you can prefix this unicode character with \u
clearKey
Depending on your terminal emulator and operating system, you might want to remap this or other characters. default: "\u001B" (esc). If you type the Clear key, the current field is cleared (with a poof sound) -- unless the current field is Newt's source editor, in which case, it Reverts to the last Saved entry. After a Clear in the Notepad, you need to tap to reselect the view
evalKey
default: "\u001A" (ctrl-z). Evaluates the current selection or field (in Newt, in the Notepad, or any input field) as a NewtonScript expression. If Newt is installed, Sloup uses Newt's read/eval/print capability for the current expression; otherwise, Sloup does a simpler compile and print (only of immediate value results: strings, numbers, characters). Any errors messages also appear in the terminal window. (Newt, if present, decodes some of the common error codes). After an Eval, you'll hear a "plunk" sound. If the current field is Newt' source editor, the current object or method is Saved (it's checked/compiled) Some expressions to try:
3
3/     //syntax err
3/2
3/x    //undef var
3/0    //div by 0
[2,4,6,8,10][0]
"Hello," && userConfiguration.name
{person: {lastName: "Smith", firstName: "John"}}.person.lastName
GetRoot():Notify(3,"AN ERROR!", "(not)")
enterKey
default: "\u0003" (ctrl-c). same behavior as evalKey -- provided as an alternate key binding
closeKey
default: "\u0017" (ctrl-w). closes 'viewFrontMost. This might be useful for closing Eval Log (after errors in Newt) or your application, though you'll probably want to set Newt's 'closeConfirm preference to avoid inadvertently closing Newt. This does GetView('viewFrontMost):close(), which may not always do what you want/expect. viewFrontMost means the frontmost view that has vApplication set. Another option might be viewFrontMostApp (same but ignores views with vFloating set). viewFrontMostKey might be a possibility, but it typically refers to the currently selected view, which often means Eval Controls
scrollUpKey
default: "\u001E" (ctrl-up). invokes :viewScrollUpScript on current keyView
scrollDownKey
default: "\u001F" (ctrl-down). invokes :viewScrollDownScript on current keyView
keyboardMap
similar to translationTable for keyboard, e.g., {keyboardMap: ["'backspace'*", "\u0008", "'left cursor'*", "\u001C", "'right cursor'*", "\u001D", ],...}

For illustration purposes (since this is redundant), assume you want to set ESC as the clearKey. The unicode value for ESC is 001B. So, to turn off debug printing and override this one key (the others are still defaulted):

Sloup
{debug: nil, clearKey: $\u001B,}

Since these options are sticky across subsequent connects (until the next time you specify global settings), there are several strategies for multiple files:

  1. set them once in a separate file that applies to all subsequent files
  2. make changes at beginning, reset defaults at end of file
  3. set default {} at beginning of each file

Here is an example of using a line continuation in entrySpec, setting several parameters, and resetting at end:

Sloup
{fieldDelimiter: ",", stripQuotes: true,\
translationTable: ["N~", "\u00D1"], totalRecords: 3}
mysoup![]
{a: "string", b: "string"}
field1,"field2 with extra quotes"
"xxx",eN~ye
3rd,entry
BYE!
Sloup
{} // restore defaults (optional)

Soup Indexes

In the default situation, when a soup name does not exist, Sloup ignores it and subsequent lines until it finds a valid soup name.You can also create a new soup by appending ! to the soup name and including an array of soup indexes (this will also find an existing soup; to add new indexes, it is best to remove and reinitialize the soup). Before doing this, you should feel comfortable with transferring entries into existing soups and customizing an entrySpec. I would also recommend obtaining a soup utility so that you can inspect entries, remove entries and remove soups (if necessary).

To create a soup, follow the name with a ! followed by an array of index specifications, e.g., [] is none. The soup is created as a "union" soup; on 1.x, this is created on each store, i.e., internal memory and card, leading to some empty soups; on 2.x, the soup is created only on the current default store.

TestSoup![]
{a: "int", b: "string"}
0	hello
1	there
BYE!

Here is an example of a different soupname line if you had wanted to index on the integer field a:

TestSoup![{structure: 'slot, path: 'a, type: 'int},]

Indexes can be useful to applications for random access or for accessing soup entries in a particular sort order. For examples of other index specs see the NTK docs, or the bitmap and sound examples (next); more docs to follow...

You can add the indexes for a soup at the end (BYE!).

Entry Spec

This is the most complicated part of Sloup. You need to specify the correct slot names and value types. For standard soups, you can discover this from NTK or additional Sloup documentation or examples. In general, you can use a soup utility (like StewPot) to inspect a soup, or print frames with the NTK Inspector or NewtDevEnv. Or, you can DUMP! a soup using an empty frame spec (note: this corresponds to the first soup entry, and requires Newt for complex entries).

Here is a summary of value types that you may encounter or wish to use. These would generally be specified as strings, i.e., enclosed in double quotes, e.g., "int".

string
text
int
integer
real
a floating point number. (this should do the correct interpretation of "decimal point" depending on locale setting, e.g., "," in Europe)
boolean
true or nil (false)
symbol
a NewtonScript symbol. When used as a value in an entrySpec, prefix with a single quote, e.g., {labels: 'Business}. In order to include characters other than alphanumerics and underscore _, surround the symbol with vertical bars, e.g., {labels: '|a funny $ymbol|}
stringWithClass
this is basically the same as symbol, but it is used to signify a type that will be applied later to the entire data object (used only with Names?)
array
comma-delimited elements, surrounded by square brackets. e.g., ["int", "int", "int"]. If the last element in an entrySpec is an array, it can be variable length, i.e., the last type in the array will be applied to any "extra" data fields
frame
slot: value pair, delimited by commas, surrounded by curly braces, e.g., {a: "int", b: "string"}
para
a special text type, primarily intended for Names and Calendar notes. For example, if you'd like to add a text note to a Names entry (the text field you see when you Show:Card&Notes), include notes: ["para"] in the frame spec. In the data entries, the text in this position up to the next tab (or end of line if it's last) will be included as a single paragraph in the note field
class:string
this indicates the type is prefixed in the actual value. For example, "homePhone:555 1234" (the type and colon are removed before the value is stored).
dateTime
on input, parses date&time string using Newton StringToDate function, using current locale; on output, formats date&time using DateNTime and current locale, e.g., mm/dd/yyyy hh:mm. Internally, dateTime is represented as an integer: number of minutes since midnight Jan 1, 1904, so you could use type int. Note: information about seconds is lost.
date
on input, parses date string using StringToDate; on output, formats data using ShortDateStr (mm/dd/yy)
dateTimeSpecs:
follow dateTimeSpecs: with an array of 3 format numbers: [longDateSpec,shortDateSpec,timeSpec], which are used respectively with LongDateStr, ShortDateStr, TimeStr, using the current locale; include nil to omit an element. The array expression is evaluated on the Newton, so can refer to built-in formats. For example, ROM_dateTimeStrSpecs (@66) contains some pre-defined formats; the following table shows which formats (indicated by "L", "S", and/or "T") are applicable to which functions(positions):
slotexample
longDateStrSpec(L)Wednesday, July 22, 1992
abbrDateStrSpec(L)Wed, Jul 22, 1992
yearMonthDayStrSpec(L)July 22, 1992
yearMonthStrSpec(L)July 1992
dayStrSpec(L)Wed, Jul 22
monthDayStrSpec(L)July 22
numericDateStrSpec(LS)7/22/92
numericMDStrSpec(S)7/22
numericYearStrSpec(LS)1992
longMonthStrSpec(L)July
abbrMonthStrSpec(L)Jul
numericDayStrSpec(LS)22
longDayOfWeekStrSpec(L)Wednesday
abbrDayOfWeekStrSpec(L)Wed
longTimeStrSpec(T)10:40:59 AM
shortTimeStrSpec(T)10:40 AM
shortestTimeStrSpec(T)10:40
hourStrSpec(T)10
minuteStrSpec(T)40
secondStrSpec(T)59

So, you could include a date (either long or short) and/or a time.

{..._modTime: "dateTimeSpecs:[@66.longDateStrSpec, nil, @66.longTimeStrSpec]",...}

You could also combine constants (e.g., adding different month + year components). Full details on format constants may be found in the Newton Toolkit Documentation:

bits, bitmap
see Bitmaps
picture
see PICT
sound, samples
see Sounds
resource
see Resources. e.g., IR resource code
binary:
follow binary: with a specific binary class, e.g., binary:pixels