TS-Lang Talks

TSL Language Aos Engine Family SqLite TSL Integration Year-2

David Simmons, Smallscript Corp

Tcl 2017 Conference SqLite Track SqLite: SqLite APIs for Cmd Extensions

• “af” procedures • af: normal • afd: deterministic • afEach: normal tables • afEach: deterministic tables • Deterministic means • cacheable idempotent • SqLite Api Integration • build flags • a few other APIs used as hooks

• UUID

• Time Conversion • %J: Julian days as float • The julianday() function returns the Julian day - the number of days since noon in Greenwich on November 24, 4714 B.C. (Proleptic Gregorian calendar).

• • time-fmt({now +1 day -2 hours time-fmt: tsl-command -7 min, {%J}); • custom time parsing and formatting library • time-fmt(connection.time-created, • parses most text forms {%Y/%m/%d %H:%M:%S.%f UTC}); • time-fmt(uuid-create-sequential(), • parses numbers as either epoch seconds in epoch, uuid or {%Y/%m/%d %H:%M:%S.%f UTC}); FileTime, time 10-7 64-bit time

11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 4 SqLite Track SqLite: JSON Interchange & Ad-Hoc Records • JSON1 Library • Json data in records • jFieldName • Json as tables in sql expressions • CRUD triggers on columns and indexes • JSON1 (flaws) • Empty and Nullable • Caching

• sqlite “lib” • built w/JSON1 • convinced a number of the poplar open source tools to add JSON1 to their nightlies • afm built as a single “lib” • built from a single “cpp” file • Linked as “afm.exe” console subsystem coffbox • Renaming “afm.exe” a .SO or .DLL • Exporting standard DLL API interfaces • AOS/S# COFF binary ffi-thunk bBuilder

• SqLite Header • SiteId, AppId • Common DB extension forms • .afx, .afr, .dpk, .zdb [FsPath VFS model]

• sqlite_master • VersionSchema table concepts • VersionSchemaTrigger table • db-resident scripts • StoredProcedures • Where to place stored procedure code depends whether a vfs-repo-fs table model is available • Undefined sql API hook (NFH) • Allows lazy NFH lookup into existing environment, or lazy load from db stored procedure tables

David Simmons, Smallscript Corp

Tcl 2017 Conference Deployment Track CoffBox: DevOps Concepts

• Executable Components • Registering script types • afm.exe • Windows • afm-symlinked-name.exe • PATHEXT • executable-db-vfs-pkg.afx • ftype, assoc • usually a “.afr” that’s been • registry twiddling for overloading symlinked • nix* • shebang-#!path or pathext-ext • shebang - #!path header registered. • chmod • script-name.afts

• My home office environment (by way of example) • Networks • 10GB Fiber • Mesh WiFi • ZWave, Ethernet-over-power • over • 70TB NAS storage • 20 computers • 200 devices • OS • Linux • OSX • Windows • Others

• Show Files and Locations • Demo App

• .af directory patterns • Packaging Model • .conf customization model • search paths • .af directory pattern • SymLinks • .afws • client system relative • .afr, .afx • symlinks, reparse-points • host system relative • .afts scripts • junctions • absolute and relative paths • repo-treatment • ignored, as-file • xattr archive-flag • Virtual Drives • .dmg, .vhdx, linux sparse file images

• script w/command line arg processing

Customizing a System Reflecting on a System • ChildProcess • ::system::info • pipes • elevated, admin, etc • http(s), message-queues • reg- commands • sockets • http(s) • EventLog mechanisms • telemetry reporting

• Binary build model • TSL scripts exported as native • cpp, hxx, hpp, h COFF library binaries (.so, .dll, .exe) • Single Binary for Exec & DLL • your-code-thunk.dll - thunk- (.so) bridges • Coff Format • Subsystems • Sections • Resources • Read-only Memory Repos

David Simmons, Smallscript Corp

Tcl 2017 Conference iOT Track iOT: Purpose Built Constrained Environments

Constraints Typical metrics • Maybe single app • Memory • Security • Cpu • Storage types • • Hw permission and firmware Storage controls • Power Management • Memory • Os Services • Power Management Lifecycle • CPU Cycle Cost • RTOS Constraints and Models • Embedded toolset contraints

11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 24 iOT Track iOT: engine requirements for iOT deployment • Os Platforms • Cpu Targets • Windows, OSX, iOS, Android, • Arm (32, 64), Intel x86/x64, Linux, Linux Embedded, MIPs FreeRTOS, nuttx, nucleus rtos • see https://www.osrtos.com/

• Networking • UX • Wifi, Bluetooth, Wired- • custom displays Ethernet, ZWave, Zigbee • custom input/sensors • VFS • Web Server/Service • Packaging, Versioning • Telemetry • Resources, executable components • Updates • Power Management Lifecycle • Device Interop (peer, slave, • Device On/Off master) • Device Firmware Update Models • Rollback, Upgrade • Build, Package, Sign requirements • Flashing Lifecycle Constraints • OTA, Wired

• Pal: Host Abstraction • Coff binary build model • Memory – cpu/kernel direct • cpp, hxx, hpp, h • TLS – CPU direct (mirror OS model) • Interrupts, Signals, Exception • embedded r/o “.afr” • Debugger Aware Channels • compression • Pal: Cpu Abstraction • FPU, Vector • Codecs • MP-Sync Instructions • Bit Operations • compression • Special Forms • Pal: FsPath Pluggable Abstraction • Engine Design • Native File System, (tags, versioning) Fs built on SqLite blobs, • Booting Model Registry, Fuse/Dokan, HTTP(S), BuiltIns • Communications • Thread Model • Transports • engine-affinity, fiber-co-operative, • Ethernet, Bluetooth, Serial rpc message streams • Sockets • DNS, DHCP, HTTP (1.0, 1.1, 2.0 ALPN) • pipes, tcp, ssl, http • SSL, Certs (pal and host integrated) • json, html, …

David Simmons, Smallscript Corp

Tcl 2017 Conference Language Track Statements: JS & TCL 12 Rules Disambiguation

Procs and Threads Code Locations • Process Model • FsPath Concepts • multiple engines allowed per process • SymLinks • Engines • Archive XAttr-Flag • engines have thread affinity • co-operative threading within an • Process Loading Model engine • Namespaces • Script Binding Model • ::super • ::system • Web Site and Page Model • ::afm • ::shell • ::app • ::script

• Phase 0: Text Command Statement • Comment Forms • Statement Delimiting • // EOL • /* nestable */ • [command must defer-level1] else rules • # - special command, NFH form • ; eos • Allows #command if NO whitespace • {} eos • Word Grouping Rules • # eos • () first following command • ‘ws’ eos [tcl-mode only] • allows whitespace • () {} … no other tokens • Phase 1: Command • partitions into discrete words whether • Phase 1a: Expr Args whitespace or not • Phase 1b: semicolon args • Deferral Levels 1,2,3 • =, ? ... : • Phase 2: Binary Operators • for() … • do while {} • Phase 3: TCL Parameters • return

• Types • Modes • default • proc • ^ uplevel & native proc^ • ::super namespace • @ modules • protected from re-definition • Prototypes • func (fn) • Tson Declaration/Merge &:{} • context bound • Declaration Features • method • parameter binding • this, prototype bound • binding (*a, &b, ?z, {x {}}) • Invoke named parameters • Mint-Paths • ( key: value, ) • All commands are namespace • Observation and NFH types and get minted • Loader Hooks, NFH

• Operators • Contexts where it is implicit • new … eval • func … • pathx invokes () • = (op-assign) • pathx indices [] • numeric ops • tcs-line • .qualifiers for typing • text command statement • == • ===, ==~, !=, !==, !=~ • Short Circuit Boolean Ops • ||, &&

• PathX • PathX Operators (incl • $$,$?,$!,$: reserved) • &:, &? • . operator • delete & • .*() invoke • && upvar • .. cascade • (expr), {}, [expr] • :: and ! binding operators • “”, {}, [] and tcp • token/macro rules

• Variable References • Name Partitioning • & Operator • … • Stack-Context References • this • &(#) • super • &pathx – deferred binding • @key, key@qualifier • [expr] => {scalar-key} • {literal-key-closing} • && Upvar model • escape rules • Valid Name Patterns

Foundation Concepts • Namespaces • Minting • ::super • namespace command • mint-name concepts • ::system • minting rules • ::afm • Where Procs Live • ::shell • Where System information is • ::app • User owned • ::script • ::global • Names • CamelCase • modules • Hyphenated

• Naming Patterns • Locations • Case Usage • Modules • Uppercase – factory (new) • ::super proc space • Lowercase - variable • ::app, ::lib • CamelCase - methods • reserved • Hyphenated - functions • ::globals – for local composers • ::afm, ::system, ::shell • Special commentary • ::app, ::script • @names • this-… names • Module Naming pattern • super • domain – like package model • uuids – anon modularization

• file-, dir-, fs- • time- • string- • ?put* • mint- • @* • sql-

Key variables Extensions • ::system::path • -ncf • ::app::path • -aei • ::shell::path • ::shell::ext • ::shell::jails

TSL Text Scripting Language AFM Design and demos Aos Engine Family 25th Anniversary Rich Web Servers DevOps Toolset Repo VFS CMS Integrated SqLite Single Binary acl-security, file- Open Ssl, Certs, CLI, Ansi, EventLog, system, registry, 1st class Sql support Config Free Json, Markdown, Services, Pipes, env, time-codec, JSON, Sql Procs HTML, TCP, Sockets Exec, Codecs SymLinks uuids

David Simmons, Smallscript Corp

Tcl 2017 Conference Refresher, demos, discussion

For those I did not meet last year, my name is David Simmons • A year has gone by… • This morning we are going to see how TSL/AFM has evolved. • The beginning of this talk will be a refresher • Then we will go into a series of demos and discussions

Over the last year, the afm system has been actively serving 10,000+/engine PBX cloud telephony users • Single-binary, single db pkg-file • coff-box symlink model, single db-file package system (vfs) • drop install on any machine. • Setting up new servers is only a few minute operation • fail-over, scaling, and versioned upgrades are mostly automated by simple remote check-in/out. • Full server and devops set of stabilized features • More than is possible to cover in a talk.

• What is the acronym “st”? • Site, Smalltalk, Smallscript • What is the acronym meaning of “AOS®”? • Agents Object System, Agile Object System, Actor Object System • What is the acronym root “af” of “afm” stand for? • collates top of the alphabet ☺ • Aos-Family, Aos File Manager, Aos Fossil Manager • But in all seriousness: • Carefully thought through names and naming conventions are critical to a language design and best practices for code libraries.

1978-1997 1997-2017 (Owner/MSFT Architect) • NBS Net/ArpaNet HW w/Basic (6512) • • Forth, APL, Mix of Shell Languages, many Smallscript, S# Language others of the era • Native engine, .NET Runtime • Fortran, Lisp, PostScript, … DLR 2000 PDC • Message-C, SIAL, Informix SQL’s Full Text engine for International Publishing on BRS/ • VBA, Visual Basic.NET Dialog with TeX • • QKS Smalltalk, Prolog, Scheme,… JavaScript • MacOS AOS 1991/1992-1998 • • Embedded in QuickTime .NET Mobile Runtime and XNA • Windows 1994-1998 • PowerShell • Newton, AppleScript, Kaleida/SK8, Taligent • AFM/TSL

Public Cloud VMs $8 Cellular Wristwatch iOT, Edge, Private Clouds DEMO Mesh Networks, 10GB fiber NAS

1997 Flashback to the good old days

2007-2017 (Rise of the mobile iOT Device 1986-1996 (gave life to the era of Java) Net) • Awesomely productive and rich toolset capable of • Everything connected, people message, and talking to almost anything watch/create content on their schedule • Immense integrated frameworks • Compute is cheap $8/full watch cpu • Pioneered many aspects of modern software competitive with 2004 PC development from patterns, agile programming, • Embedded, real-time, devices proliferate unit testing and with it immense scaling challenges • Disaster for working with text and file-based code management and practices • HTTP, JSON, HTML-UI dominate compute systems • Monolithic image system unable to integrate within other systems, execution engine scaling • Text processing and file-based assets issues dominate compute design with massive • Decompositional Model challenge to schema stack-based libraries version, package, and be small and bootstrap • devops challenges and complexity single from nothing largest impediment to evolution

• Let’s begin by talking about the FsPath system Fuse • pluggable and defaults to supporting: Dokan Registry • Disk • fully symlink, xattr and stream aware • Versioned repo (scm) • System core files baked into the afm coff binary as a r/o repo • Web based files • has a specific path syntax for disambiguating content • server-type • file-system (default, current directory) • streams/xattr support • repo vfs • branches, tags, versions support • af web-server generalized

/~:alias/path/file.ext path aliases are located in ::app::path ^ ::system::path maps

① ②

File folders Branch folders Branch folders Branch folders on disk in local repo in r/o coff repo in https repo (versioned) (versioned) /path/file.ext /:branch:repo/file.ext ~:/path/file.exthttps://af.st/:branch:repo/ N:/path/file.ext .afr; afm.exe ./.af/site/ N:/path/dir:xattr- .afx; afx.exe ./.af/sites/domain stream.ext .afws afts.exe

The executable’s File Name is used by kernel.tsl code to determine the execution path as it examines the exec command line; this allows efficient creation of single shared binary tools using symbolic-link naming patterns. ① ② kernel.tsl Rich Web CMS Fossil-SCM .afx .afr Afx Web Servers time-fmt, csv/cst, child- Repo db(s) SqLite process, certs/ssl, sql-procs, many-codecs, regex, debug- TSL Language 1.2 io/events, registry/env Open Ssl

afts Intel, ARM, … COFF (binary bundled) Executable *nix, ms-windows afm

… $& ::identifier .(…) && ^ {…} :: . […] Examples:

$![putl "Hello"]1 $!{string "Hello"} † $!(17 + 5 % 3)3 $::identifier-expr $^identifier-expr $identifier-expr $identifier.{literal-key}.[expr-key] $a.b[3] $a::b[3] $a.b::[3] $a{b}[3]["z"] $receiver-path-expr::namespace-expr(?invoke-expr-param?, …) Cases of Note: &:{ … JSON pojo … } JSON pojo supporting TSL eval expr extensions and auto-conversion. &:[ … JSON poja … ] JSON poja supporting TSL eval expr extensions. var identifier-path = &&var-ref-path; Binds two variables, as opposed to assignment.

Process Exec main Engine Init primary thread TSL Core Registered

.af/.conf.tsl Link-name executed as Not-found Hook Invoked conf located and run command, if available to load command

② ①

afm Command Executed as 1. script relative .af/ Fall-through to fossil- Program 2. exe relative .af/ default 3. coff system pmsp .af/ OR

11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 56 Command re-examined as Message statement word-tokens: literal, macro- $ "…" […] forms {…} Command Params (aka words) cputl $msg; # A TCL command with macro- param cputl(msg) # A TSL function with expr- param this.cputl(msg)11/17/2017 ; TSL Talk by David# Simmons,A TSL Smallscript methodCorp (c) 2017 with an expr-param57 TSL Language Aos Engine Family afm

David Simmons, Smallscript Corp

Tcl/Tk 2016 Conference What we are going to briefly tour this afternoon..

TSL Text Scripting Language AFM Its COFF binary executable host Aos Engine Family

afts Command Line Script

11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 60 DSL: Domain Specific TSL Language As a language TSL is founded on the macro and command patterns used in TCL As C++ can be described in relation to C, TSL can be described in relation to TCL It would be incorrect to describe TSL as a dialect of TCL

TCL JavaScript blended Smalltalk, Lisp, Self Intrinsic C++ Macro & Command syntax & intrinsic messaging behavior object integration Patterns JSON and

Frameworks, IDE, Executable Libraries: Source, Op-Codes, and precompiled Machine CodeImage-Snapshots, Intrinsic-Repos

Object Model: Path Binding Execution Host Integration prototype, mixins Engine: Invocation: Interp- & Lifecycle Model predicate binder JIT, FFI

Perspective Types: Dynamic Composition Versioning, Modularization, AOS Execution Engine and Container System Aspect-weaving and introspection Security 11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 61 AFM executable Self-installing single host executable that uses symbolic-link file-naming to determine its execution behavior

The executable’s File Name is used by afconfig code to determine the execution path as it examines the exec command line; this allows efficient creation of single shared binary tools using symbolic- link naming patterns. ① ② afconfig.tsl aflib Fossil-SCM

TSL Language Files and Directories .afr SqLite Repo db(s) C/C++ implementation BuiltIn FileSystem AOS Execution Engine Intrinsic Resources C/C++ Libraries and Services System and Libraries

afts Intel, ARM, … COFF (binary bundled) Executable *nix, ms-windows afm

Hello, my name is David Simmons • For 25 years I’ve had a particular specialization in high-performance hybrid dynamic-static language runtime systems. • By education, I am an Electrical Engineer and Astrophysist specialized in VLSI design. However, most of my career has been as a Software Engineer and Architect • Working on Operating Systems, Hardware Devices, Real Time Systems and Programming Languages and their Execution Machinery and Infrastructure • I began my career some 40 years ago, in the summer of 1976 at The National Bureau of Standards which is known today as NIST (The National Institute of Science and Technology) working on Fortran Runtime Real Time Libraries, Perkin Elmer Operating System Development, and ArpaNet NBS Net Packet Switch Hardware and Analyzers. • Employed working with or for: • NBS/NIST, U.S. Congress, Air Force, Faculty at UofMd NSF Systems Research Center, Tokyo Gas, Apple, Suse Linux, Microsoft and worked for or owned a number of small to mid-size companies of up to 250 employees. • Worked on, collaborated on or was lead architect on the following language systems: • Apple’s Kaleida, Apple’s AppleScript, Apple’s Quicktime-Track-Scripting, Script on Newtons • Basic, Forth, Message-C, SIAL (1990 ODBC equivalent with full-text search engine), Smalltalk and Prolog, S# • Microsoft Visual Basic, Javascript/JScript, .NET Runtime (desktop, mobile), XNA, Powershell

Process Exec main Engine Init primary thread TSL Core Registered

afconfig.tsl Link-name executed as Not-found Hook Invoked config located and run command, if available to load command

② ① 1. current directory afm Command Executed as 2. ~/ user‘s home Fall-through to fossil- Program directory default 3. ee‘s home directory OR 4. coff-builtin directory

TCL TSL • Statement • Statement • Composed of Words • Composed of First Word and • First Word is Command Rest • Rest is Command Macro Params • First Word is Message • Expressions • Command, Function or Method • Rest is Message DSL Params • Things • Literal, Macro, Eval • Scalars • Things • Key-Value Containers • Scalars • Mixin Prototype Dynamic 11/17/2017 TSL Talk by David Simmons, SmallscriptObjects Corp (c) 2017 65 Operators and Keywords Syntax, Semantics and DSL patterns Operator Language Type Patterns {…} tcl Literal text patterns. "…" '…' `…` tcl+ Macro prefix-names. $(…), $[…], ${…} forms. (…) invocation. Quirks for "[…]". † […] tcl Command proc, func, method patterns. Unary and binary symbol method patterns. (…) tsl Expr (eval) implicit eval pattern. -- $ tcl+ MX path Operator macro-expand operator for variables and other tokens. & tsl Ref path Operator reference operator for variables and path expressions. -- \ tcl+ Escape Operator enables UTF8, ascii and inline-operator escapes. ; tcl † End-of-statement TCL quirks mode for newline and curly-brace EOS. , tsl End-of-expression Expr eval param delimiter. -- # /* */ // tcl+ † Comments JavaScript/C++ compatible comments. Quirks for TCL #.

Keywords11/17/2017 DescriptionTSL Talk by David Simmons, Smallscript Corp (c) 2017 66 super this † tsl inheritance operator and contextual variable for accessing inheritance Command re-examined as Message statement word-tokens: literal, macro- $ "…" […] forms {…} Command Params (aka words) putl $msg; # A TCL command with macro-param putl(msg) # A TSL function with expr- param stream.putl(msg); # A TSL method with an expr-param stream.putl()11/17/2017 $msgTSL ;Talk by David# Simmons,A TSL Smallscript methodCorp (c) 2017 with a macro-param67 Variables re-examined

Declaration set varName valueParamWord;# TCL assignment form set varName;# TCL get command var varRef = eval-expr;# TSL assignment form var varRef;# TSL decl command form

Variable Reference is a path

Path roots are: Global, or in Stack Frame Context down-level of global-root- frame.

… $& ::identifier.(…) && ^ {…} :: . […] Examples:

$[putl "Hello"]1 ${putl "Hello"} † $(17 + 5 % 3)3 $::identifier-expr $^identifier-expr $identifier-expr $identifier.{literal-key}.[expr-key] $a.b[3] $a::b[3] $a.b::[3] $a{b}[3]["z"] $receiver-path-expr::namespace-expr(?invoke-expr-param?, …) Cases of Note: &{ … JSON pojo … } JSON pojo supporting TSL eval expr extensions and auto- conversion. &[ … JSON poja … ] JSON poja supporting TSL eval expr extensions. var11/17/2017 identifier-path = &&var-ref-pathTSL Talk by; David BindsSimmons, two Smallscript variables Corp (c), 2017 as opposed to assignment. 69 DEMO

var sharing Script Example

11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 70 Object Model Object Logically, objects reference other C/C++ Structured objects using key-value associations Header Storage Slots which can be shared. An association can also be called a variable. TSL variables and commands play a key role in how In practice, the internals are perspective types are used for versioning and dynamic security more highly optimized and only Variable (aka Association) ObjRef’s are tagged used associations as an object pointers. Certain scalar proxy indirection when sharing Key Value Symbol ObjRef values, like IntPs objects, requires it. actually have their value Expando Type proto-super mixin behavior encoded directly in the pointer. Object Object

C/C++ Structured C/C++ Structured Header Storage Slots Header Storage Slots

11/17/2017 TSL Talk by David Simmons, Smallscript Corp (c) 2017 71 Object Space One per engine created within a process. There can be an unlimited number of engines within a process. Each engine takes microseconds to create or release. The variable predicate path- binder plays a key-role in mixin Global Root ::system ::system::info and path resolution By default, Stack Frame proc thisFrame #1 Stack Frame declarations are ::system::command placed here. thisFrame s #2 Stack Frame thisFrame While crucial concepts, as we will see #3 By default, next, the DSL for intrinsic command func declaration constructor forms mean declarations are that and are rarely 11/17/2017upvar uplevel TSL Talk by David Simmons, Smallscript Corp (c)placed 2017 here. 72 needed. JSON Given a uniform internal object model, all objects can be converted to or from JSON via serialization. When TSL networking protocols for HTTP/ The original technology name for this in AOS HTTPS with JSON are combined with SQL 1992 was PIPOs. Platform Independent Portable TRIGGERS in SqLite using JSON1, powerful Objects, which contained TOCs, DBs, versioning, robust systems can be easily built. schema-migration features and supported cross- machine migration of threads and UX components with automatic re-wiring.

Object Object

Scripts are lambda commands

• Modules are commands with additional metadata.

• Source has provenance, which plays a role in perspective-type binding of versions and access security.

• Since objects are pathable and commands are global or contextual objects, commands are commonly used as namespaces.

More demos offline and BOF