Feature Exchange ­ Better FFI

Background

The biggest new part of LiveCode 8 is LiveCode Builder, a new English­like programming language which is ​ ​ designed to be used to extend the feature set of LiveCode itself. LiveCode Builder has been designed in such a way to ensure that it can eventually replace C++ as the implementation language for all the LiveCode features you use and enjoy today.

Compared to LiveCode Script, LiveCode Builder is a stricter language with a more expressive type system allowing it to be used to build the very­high level features you use every day. It has been designed to ensure that, in the future, LiveCode Builder code can be compiled to native code with similar performance to C++ code that does the same thing.

One of the key features of LiveCode Builder is its ‘foreign function interface’, or FFI. A ‘foreign function’ is a function which has been written in C and compiled into a loadable shared library (or DLL, on Windows). LiveCode Builder lets you define ‘foreign handlers’ that let you call these functions directly from LiveCode Builder code. This feature is used extensively in the implementation of LiveCode Builder itself! Most of the features exposed through syntax in LiveCode Builder are implemented in C++ (with C function signatures) and then referenced from the LiveCode Builder standard library modules.

The ability to hook into foreign code in this fashion enables LiveCode Builder to be used to implement very high­level features (which you access using LiveCode Script) which directly use operating system APIs or pre­existing third­party libraries.

Foreign functions are designed to be called by the language they are implemented in. This means that they know nothing about the high­level types which LiveCode Builder uses, and you are familiar with from using LiveCode Script. To make such functions easily usable requires a bridge between the types of value which Builder understands to the types which the foreign language understands.

A good example of this can be seen by considering perhaps the most used value type you see in LiveCode Script and Builder: the string. In the LiveCode world a string is a sequence of characters which you can easily manipulate ­ you don’t have to worry about representation or memory management as the LiveCode VM takes care of all of that for you. In contrast, many C functions expect strings as a sequence of bytes or shorts with either a separately­specified length, or a NUL terminating byte. So, if you have a C function which expects a string as a parameter, you need to map the high­level string abstraction LiveCode uses to the specific kind of string that particular C function understands. The C function might also want to ‘keep’ the string you give it rather than just look at it temporarily, and this means ideas of lifetime and ownership of the underlying representation come into play. In addition to these considerations, the conventions in terms of the actual type of string a C function might expect, or what it expects to do with it can’t be detected automatically by looking at the C function! The information often only exists elsewhere, such as in the reference documentation for that particular function, or the documented conventions established for the library as a whole.

The mapping of types of value from one programming language’s world to another programming language’s (or ‘bridging’) is perhaps the hardest problem to solve when you want this level of interoperability.

Currently, LiveCode Builder tries to solve the bridging problem to some extent but the reality is that you still have to do a lot of work to utilise all but the simplest of C functions. The foreign function interface in LiveCode Builder is perfectly usable, but it is still a lot harder to use then we’d like, and a lot harder than it ​ ​ should be in a programming language which bears the name LiveCode.

It is this ‘bridging’ problem we would like to work on to solve, and which has prompted this Feature Exchange proposal.

Bridging to C

Here are some examples of C APIs and how they ideally would map to Builder. libuuid ­ libuuid_generate

This really useful function generates a UUID and is present on most UNIX based systems in a library called ‘libuuid’. In C, the signature (description of parameter types and return type) of ‘libuuid_generate’ is:

typedef unsigned char uuid_t[16]; ​ ​ ​ ​ ​ void libuuid_generate(uuid_t uuid); ​

These two lines of C basically do the following: ● Declare a new type ‘uuid_t’ which is a sequence of exactly 16 ‘unsigned chars’. ● Declare that the ‘libuuid_generate function’ takes a parameter of type ‘uuid_t’ and has no return value.

This sounds all well and good, and surely that should be all you need to know to be able to call the function from a higher­level language such as Builder. However, unfortunately, it is not! There are two pieces of missing information.

1. What is the content of those 16 ‘unsigned chars’? It is a matter of informal convention that you typically use ‘unsigned char’ in C to denote a byte, and as such the ‘uuid_t’ type is most likely describing a sequence of 16 bytes. However, this is just an informal convention: whether this is the case or not depends on what the intent of the function is and the conventions that the writers of libuuid decided on. They could have equally well have chosen ‘char’ and not ‘unsigned char’ and it would be just as correct. The intended meaning will be part of the documentation of the function within the library. It is not part of the language syntax itself. 2. What does the function actually do with its uuid parameter? Does it want to read from it, does it want to write to it, or does it want to do both? Again, this is something which is only explained it its documention. In this case it is quite easy to figure out. Because this is generating a UUID, it would ​ ​ be reasonable to expect that the function will write a new uuid to the parameter it has been given.

With the actual C syntax and these two extra pieces of information, it becomes clear that the ideal Builder handler would look like::

handler libuuid_generate() returns Data ​ ​ ​

Once you understand both sides — the handler we’d like in Builder, and the C function definition and extra information — you can work out how to ‘glue’ the two sides together so Builder can call the C function seamlessly and safely. libsqlite ­ sqlite3_prepare

This complex function prepares a piece of SQL for execution by SQLite. It has the following C signature:

typedef struct sqlite3 sqlite3; ​ typedef struct sqlite3_stmt sqlite3_stmt; ​ int sqlite3_prepare( ​ sqlite3 *db, const char *zSql, ​ ​ ​ int nByte, ​ sqlite3_stmt **ppStmt, const char **pzTail); ​ ​ ​

There are several pieces of extra information which can be foind in the documentation for the function: ● The zSql parameter should be a string which has been encoded using UTF­8 ● The nByte parameter is the maximum number of bytes to consider in the zSql parameter, up to the first zero byte (if present) ● ppStmt is the memory address where the created sqlite3_stmt reference should be stored ● pzTail is the memory addresswhere the point in zSql that the parser got to should be stored, because the function only compiles the first SQL statement in zSql ● The return value is an error code. If it is SQLITE_OK then it succeeded, and otherwise it failed and the return parameters will not have been touched.

The trickiest part here is pzTail return value. At the C level, this return value will be a memory address inside the zSql string, but since you don’t really want to be dealing with memory addresses directly this needs to end up being an offset (in characters) into the input SQL string.

Taking all this information into account, a reasonable Builder signature for the function could be:

handler sqlite3_prepare( ​ in db as Pointer, ​ ​ ​ in zSql as String, ​ ​ ​ out ppStmt as Pointer, ​ ​ ​ out pzTail as Integer) returns Integer ​ ​ ​ ​ ​

Of course, this still leaves the question of how to glue the two levels (Builder and C) together. Windows API ­ GetCurrentDirectory

The conventions used in different foreign libraries vary and many have their own specific patterns which are usually uniformly applied. A good example of this is the way the Windows Win32 API typically handles returning variable length strings. Consider the GetCurrentDirectoryW function from the Win32 API:

typedef uint32_t DWORD; ​ typedef wchar_t *LPWSTR; ​ DWORD GetCurrentDirectoryW(DWORD nBufferLength, LPWSTR lpBuffer);

As before, there are a few bits of further information you need to understand how this function works: ● The caller must provide a buffer big enough to include the NUL terminating char to lpBuffer, and pass its length in nBufferLength. ● The caller can determine the size of the buffer needed by calling the function with nBufferLength of 0 and lpBuffer as NULL. ● If the function fails, 0 is returned and the error code can be gained by calling GetLastError(); otherwise it returns the number of chars written into the buffer.

There is a pattern here which is used a lot in the Win32 API. You call an API which returns a variable length buffer with 0 for the length and NULL for the buffer, and the function tells you how big it needs to be. This essentially leads to a ‘double­call’ approach for many of these APIs ­ you first call them to find out how much memory you need to allocate, and then you call again to actually get the result.

Thus, a nice Builder signature for this function would be this:

handler GetCurrentDirectoryW(out rDir as String) returns Boolean ​ ​ ​ ​ ​ ​ ​

We would want the need to call the foreign function twice to be hidden at the Builder level, and the return value to simply indicate whether the call was successful.

Gluing

The examples so far have shown some C APIs and then given some ‘ideal’ Builder handlers that make it possible to use them while hiding as many of the low­level details as possible. However, the examples haven’t explained how to connect the two together. Some ‘glue’ code is needed which maps the C world of memory addresses, bytes and memory to the Builder work of Data, Strings, Arrays and other high­level types.

One option would be to write the glue code in C or C++.You would create a C function which has a signature which Builder hooks into more easily. However, then you then need to compile that C code, which can be quite inconvenient, especially when you need to support several platforms for the same library. A better approach would be to make it possible to write the glue code in Builder itself.

To be able to write the glue code in Builder requires the addition of a set of ‘unsafe’ features to the language. These ‘unsafe’ features would allow you to do things with pointers and memory directly. The idea is that these very low­level and unsafe features can be used to build high­level and safe features, in a way very similar to how C is currently used to provide the high­level features you use in the LiveCode engine everyday. The kinds of features that are needed are things like being able to fetch a sequence bytes from a location in memory, or convert a (byte­ptr, length) pair into a Builder Data value.

Let’s now look at what some of this glue code might look ­ assuming we had such ‘unsafe’ features. libuuid ­ libuuid_generate

As explained above, you might map the C function:

void libuuid_generate(unsigned char uuid[16]); ​

To the Builder handler:

handler libuuid_generate() returns Data ​ ​ ​

To do this, you would need to request enough memory to hold the uuid generated by the C function, and then convert this memory into a Builder Data value.

The Builder code to do this might look something like this:

foreign handler __uuid_generate(in rUUID as Pointer) ​ ​ ​ ​ ​ ​ ​ handler uuid_generate() returns Data ​ ​ ​ variable tUUIDMemory as Pointer ​ ​ ​ put unsafe.MemoryAllocate(16) into tUUIDMemory ​ ​ ​ __uuid_generate(tUUIDMemory) return unsafe.DataCreateAndRelease(tUUIDMemory, 16) ​ end handler ​ ​ libsqlite ­ sqlite3_prepare

Suppose you want to map the C function:

int sqlite3_prepare( ​ void *db, const char *zSql, ​ ​ ​ int nByte, ​ void **ppStmt, const char **pzTail); ​ ​ ​ to the Builder handler:

handler sqlite3_prepare( ​ in db as Pointer, ​ ​ ​ in zSql as String, ​ ​ ​ out ppStmt as Pointer, ​ ​ ​ out pzTail as Integer) returns Integer ​ ​ ​ ​ ​

This is quite a bit more complex than the previous example, but the glue code might look something like this:

foreign handler __sqlite3_prepare( \ ​ ​ ​ in db as Pointer, \ ​ ​ ​ in zSql as Pointer, \ ​ ​ ​ in nByte as CInt, \ ​ ​ ​ out ppStmt as Pointer, \ ​ ​ ​ out pzTail as Pointer) returns CInt ​ ​ ​ ​ ​ handler sqlite3_prepare( \ ​ in db as Pointer, \ ​ ​ ​ in zSql as String, \ ​ ​ ​ out ppStmt as Pointer, \ ​ ​ ​ out pzTail as Integer) returns Integer ​ ​ ​ ​ ​ variable tSqlAsUTF8Ptr as Pointer ​ ​ ​ variable tSqlAsUTF8Length as CInt ​ ​ ​ variable tStmtReturn as Pointer ​ ​ ​ variable tTailReturn as Pointer ​ ​ ​ variable tReturnCode as CInt ​ ​ ​ unsafe.StringToUTF8(zSql, tSqlAsUTF8Ptr, tSqlAsUTF8Length) put __sqlite3_prepare(db, tSqlAsUTF8Ptr, tSqlAsUTF8Length, tStmtReturn, tTailReturn) \ ​ into tReturnCode ​ If tReturnCode is not 0 then ​ ​ put tStmtReturn into ppStmt ​ ​ ​ put unsafe.UTF8StringCountCharactersBetween(tSqlAsUTF8Ptr, tTailReturn) \ ​ ​ ​ into pzTail ​ end if return tReturnCode ​ end handler ​ ​ Windows API ­ GetCurrentDirectory

Suppose you want to map the Win32 API function:

DWORD GetWindowsDirectoryW(DWORD nBufferLength, LPWSTR lpBuffer);

To the Builder handler:

handler GetCurrentDirectoryW(out rDir as String) returns Boolean ​ ​ ​ ​ ​ ​ ​

This might look something like:

foreign handler __GetCurrentDirectoryW(in nBufferLength as CUInt32, in lpBuffer as Pointer) \ ​ ​ ​ ​ ​ ​ ​ ​ ​ returns CUInt32 ​ handler GetCurrentDirectory(out rDir as String) returns Boolean ​ ​ ​ ​ ​ ​ ​ variable tResult as CUInt32 ​ ​ ​ put __GetCurrentDirectoryW(0, nothing) into tResult ​ ​ ​ ​ ​ If tResult is not 0 then ​ ​ ​ ​ return false End if variable tBuffer as Pointer ​ ​ ​ Put unsafe.MemoryAllocate(tResult * 2) into tBuffer put __GetCurrentDirectoryW(tResult, tBuffer) into tResult ​ ​ ​ if tResult is 0 then unsafe.MemoryDeallocate(tBuffer) return false ​ End if put unsafe.StringCreateAndRelease(tBuffer, tResult ­ 1) into rDir ​ ​ ​ return true end handler

Automatic Generation

You have seen number of possible examples of how the glue code might work. However, these examples haven’t shown how to produce similar glue code for the vast numbers of libraries which exist on each platform supported by LiveCode. Writing glue code by hand for all of these things would be gigantic task. It is also highly error prone and very fiddly.

On the other hand, the glue code has a very predictable structure that is repeated over and over again for each of the functions in a library. It is possible to describe the mappings between the C functions and Builder handlers in a data file and then automatically generate the glue code.

For example, the mapping for the above three examples could be described by annotating the C function signatures in various ways:

handler uuid_generate() returns Data ​ ​ ​ wrap void libuuid_generate(unsigned char uuid[16]) ​ out uuid as ByteString ​ ​ ​ return uuid ​

handler GetWindowsDirectory(out rDir as String) returns Boolean ​ ​ ​ ​ ​ ​ ​ wrap uint32_t GetWindowsDirectoryW(uint32_t nBufferLength, wchar_t *lpBuffer) ​ call­to­measure­z(nBufferLength, lpBuffer) out lpBuffer[nBufferLength ­ 1] as UTF16StringZ into rDir ​ ​ ​ ​ ​ return result is not 0 ​ ​ ​ ​ ​

handler sqlite3_prepare(in pDatabase as Pointer, \ ​ ​ ​ ​ ​ in pSql as String, \ ​ ​ ​ out rStmt as Pointer, \ ​ ​ ​ out rOffset as Integer) returns Integer ​ ​ ​ ​ ​ wrap int sqlite3_prepare(void *db, \ ​ const char *zSql, \ int nByte, \ void **pStmt, \ const char **pzTail) in pDatabase into db ​ ​ ​ in pSql as UTF8StringZ into zSql[nByte] ​ ​ ​ ​ ​ out pStmt as Pointer into rStmt when result is not 0 ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ out chars­between(zSql, pzTail) into rOffset when result is not 0 ​ ​ ​ ​ ​ ​ ​ ​ ​ return result ​

Note that this example is just for exploring the idea, and the final form will probably be different!

The approach we are intending to take is to write a tool that helps to generate all the necessary glue code. It would take a very high­level description of a library, with all the details of how to call the foreign functions within it from Builder, and and use it to generate LiveCode Builder code which wraps the low­level foreign functions with high­level handlers.

This automatically generated module would use a set of ‘unsafe’ features which can manipulate things at the pointer and byte level to bridge between the high­level types of Builder and the low­level types of C. These features are considered unsafe because using them incorrectly can cause crashes.

Bridging to Objective­C and Java

What if you want to bind to a foreign function that’s not written in C (or doesn’t have a C­style signature)? In order to leverage the full power of all the platforms LiveCode runs on, you may need to call methods and manipulate Java and Objective­C objects.

As it turns out, though, this can be done by building on the unsafe module and glue code generation tool as both Objective­C and Java have C APIs for manipulating their objects ­ i.e. we can augment the glue generation tool to generate Builder handlers which wrap calls to Java and Objective­C objects.

Objective­C ­ setIdleTimerDisabled

As an example of an useful Objective­C API on iOS ­ we’ll take a look at setIdleTimerDisabled. This API is a method of the UIApplication object and it allows you to toggle whether the device will go into sleep mode after a period of no activity. This signature for the method is straight­forward:

@interface UIApplication ­ (void)setIdleTimerDisabled:(BOOL)state @end

Now, Builder as it stands doesn’t have objects. However, that does not mean that we cannot still interact with object­based languages! You can view any (instance) method call on an object as a function that takes the object as its first argument, followed by the method’s arguments. With this in mind, we end up with the following Builder handler:

handler UIApplicationSetIdleTimerDisabled(in pObject as Pointer, in pState as Boolean) \ ​ ​ ​ ​ ​ ​ ​ ​ ​ returns nothing ​ ​

The glue code to bridge between Builder and Objective­C would be something like the following:

handler UIApplicationSetIdleTimerDisabled(in pObject as Pointer, in pState as Boolean) \ ​ returns nothing variable tBooleanAsCBool as CBool ​ ​ ​ put pState into tBooleanAsCBool ​ ​ ​ objc_msgSend(pObject, sel_registerName(“setIdleTimerDisabled:”), tBooleanAsCBool) end handler ​ ​

Note: Here we assume that the objc_msgSend and sel_registerName foreign functions are already wrapped ​ and available.

Java ­ wake locks

Android’s ‘wake locks’ similarly prevent the device going into sleep mode. On Android a wake lock is an object which you create and then ‘acquire’ and ‘release’. For example, the acquire method has the signature:

class PowerManager.WakeLock ​ { void acquire(); ​ };

You could map this acquire method to a Builder handler:

handler PowerManagerWakeLockAcquire(in pObject as Pointer) returns nothing ​ ​ ​ ​ ​ ​ ​ ​

The glue code to bridge the method between Builder and Java would be something like the following:

handler PowerManagerWakeLockAcquire(in pObject as Pointer) returns nothing ​ ​ ​ ​ ​ ​ ​ ​ Variable tClass as Pointer ​ ​ ​ Variable tMethod as Pointer ​ ​ ​ put JNIEnvFindClass(unsafe.GetJNIEnv(), “PowerManager.WakeLock”) into tClass ​ ​ ​ put JNIEnvGetMethodID(unsafe.GetJNIEnv(), tClass, “acquire”, “()V”) into tMethod ​ ​ ​ JNIEnvCallVoidMethod(unsafe.GetJNIEnv(), pObject, tMethod) end handler ​ ​

Note: Similarly to the Objective­C example, we are assuming here that the relevant JNI (Java Native ​ Interface) foreign functions are already wrapped and available. Potential Uses

There are lots of ways to use this easier­to­use foreign function facility that we are proposing. It will unlock the ability to call any system API with ease, and that will in turn make it possible to use many of the vast number of third­party libraries which exist.

Here are a few examples of things which could be made available to Builder by implementing this feature:

● Monitor battery status on iOS. The UIDevice object on iOS provides three properties for enquiring about battery status ­ batteryLevel, batteryState and batteryMonitoringEnabled. ● Monitor battery status on Android. There is a standard class android.os.BatteryManager which provides this facility. ● Play from a user’s iTunes library on iOS. The iOS MPMusicPlayerController class provides a helpful interface for playing music from within your app. ● Import and export BSON. Wrap libbson to be able to parse and emit BSON. ● Access extended file details. Wrap getxattr, removexattr, listxattr on UNIX systems to be able to access extended file attributes. ● Text­to­Speech on Android via the android.speech.tts.TextToSpeech class.

Stretch Goals ­ Worked Examples

In addition to the core work involved in producing the unsafe module and the glue generation tool, we’d also like to produce fully functional, useful, worked examples.

Stretch 1 ­ Datastore Library

The sqlite library has become the de­facto standard for usage as a local SQL datastore for all kinds of applications and as such really deserves its own easy­to­use library for LiveCode without needing to go through revDB.

This example would split into two pieces.

The first piece would be the specification file used to generate the Builder wrapper around the sqlite library. This part could be used by anyone wanting to access sqlite functions directly from Builder, but its first purpose would be to allow us to write a new set of functions for LiveCode Script allowing direct access to sqlite databases using libsqlite directly (rather than going through revDB).

The second part would be the Builder library providing the high­level functions usable from LiveCode Script. The library would allow you to: ● open and close sqlite database files ● execute SQL statements in the context of an open sqlite database ● perform an SQL query in one go, returning the results as an array or a string ● start an SQL query and then allow the results to be stepped through one­by­one, meaning that all the data doesn’t have to be processed at once Stretch 2 ­ Audio Recording

The engine provides easy­to­use sound recording syntax in the form of the record sound command and a ​ ​ collection of global properties. However, with the demise of QuickTime, the current functionality needs to be replaced on Mac and Windows, and we would like to extend the support to all platforms.

During the Cocoa port, we already abstracted the engine code internally to provide the sound recording on Mac via a modular component (the current implementation is based on QTKit) and we propose to implement this component for all platforms using the appropriate system APIs.

This will involve writing specification files to generate the Builder wrappers for each of the necessary system libraries on each platform. Again, these low­level modules would be reusable by anyone needing to directly access audio recording related functionality.

The second part would be implementing the engine’s ‘Sound Recorder’ interface for each platform.

Stretch 3 ­ Mobile Native Field Widget

LiveCode currently allows you to create ‘native’ fields on iOS and Android. However, they are much harder to use than we’d like as they require you to use function calls to create and manipulate the objects rather than being able to drag one out and use it as if it were a normal LiveCode control.

Therefore for the third example we’d propose to produce a native field widget which would work on iOS and Android. Rather than having to create and delete it manually, you would be able to drag the widget onto your cards and it would ‘just work’ when run on Android or iOS ­ just like the new browser widget does.

Again, this will split into several pieces ­ the specification files for the relevant system libraries which are needed, as well as the actual widget implementation.