Pyslet Documentation Release 0.7.20180305 Steve Lay Mar 05, 2018 Contents 1 What’s New? 1 2 Compatibility 11 3 IMS Global Learning Consortium Specifications 21 4 The Open Data Protocol (OData) 119 5 Hypertext Transfer Protocol (RFC2616) 231 6 Other Supporting Standards 279 7 Welcome to Pyslet 461 Python Module Index 465 i ii CHAPTER 1 What’s New? To improve PEP-8 compliance a number of name changes are being made to methods and class attributes with each release. There is a module, pyslet.pep8, which contains a compatibility class for remapping missing class attribute names to their new forms and generating deprecation warnings, run your code with “python -Wd” to force these warnings to appear. As Pyslet makes the transition to Python 3 some of the old names may go away completely. The warning messages explain any changes you need to make. Although backwards compatible, using the new names is slightly faster as they don’t go through the extra deprecation wrapper. It is still possible that some previously documented names could now fail (module level functions, function arguments, etc.) but I’ve tried to include wrappers or aliases so please raise an issue on Github if you discover a bug caused by the renaming. I’ll restore any missing old-style names to improve backwards compatibility on request. 1.1 Version Numbering Pyslet version numbers use the check-in date as their last component so you can always tell if one build is newer than another. At the moment there is only one actively maintained branch of the code: version ‘0”. Changes are reported against the versions released to PyPi. The main version number increases with each PyPi release. Starting with version 0.7 changes are documented in order of build date to make it easier to see what is changing in the master branch on GitHub. Not sure which version you are using? Try: from pyslet.info import version print version 1.2 Version 0.8 TBC 1 Pyslet Documentation, Release 0.7.20180305 1.3 Version 0.7.20170805 1.3.1 Summary of new features Pyslet now supports Python 3, all tests are passing in Python 3. Travis now builds and tests Python 2.7 and Python 3.5, I’ve dropped 2.6 from the continuous integration testing because the latest Ubuntu images have dropped Python2.6 but you can still run tox on your own environments as it includes 2.6 in tox.ini. Various bug fixes in OData and HTTP code. Warning: for future compatibility with Python 3 you should ensure that you use the bytes type (and the ‘b’ prefix on any string constants) when initialising OData entity properties of type Edm.Binary. Failure to do so will raise an error in Python 3. 1.3.2 Tracked issues The following issues are resolved (or substantially resolved) in this release. #3 PEP-8 Compliance The pep8-regression.py script now checks all source files using flake8; all reported errors have been resolved Added a new metaclass-based solution to enable method renaming while maintaining support for derived classes that override using the old names. Crazy I know, but it works. #12 Bug in odata2/sqlds.py Bug when using numeric or named parameters in DB API. Added support for pyformat in DB APIs as part of enabling support for PyMySQL. #23 Framework for WSGI-based LTI Applications (beta quality) Re-engineered Session support in the wsgi module to reduce database load, replacing the Session table completely with signed cookies. If you have used the wsgi.SessionApp class directly this will be a breaking change but these classes will remain experimental until this item is closed out. The database schema required to support LTI has changed slightly as a result. Changed from Django templates to use Jinja2 (this requires almost no changes to the actual sample code templates and makes the intention of the samples much clearer). Thanks to Christopher Lee for recommending this change. Possible breaking change to wsgi module to refactor authority setting to “canonical_root”, modified WSGIContext object to accept an optional canonical_root argument and removed the optional authority argument from get_app_root and get_url. The authority setting was previously a misnomer and the wsgi sammples were not working properly with localhost. Changed wsgi module to use the OSFilePath wrapper for file paths for better compatibility with Posix file systems that use binary strings for file paths. This module was causing test failures due to some use of os.path module with mixed string types. #29 https connections fail on POST after remote server hangup The currently implemented solution is to allow an open ssl socket to be idle in the ‘blocked’ state for a maximum of 2s before sending a new request. After that time we tear down the socket and build a new one. This may now be a bit aggressive given the newer SSL behaviour (which differentiates issues in the underlying socket with different SSL exceptions). #30 Provide http connection clean-up thread 2 Chapter 1. What’s New? Pyslet Documentation, Release 0.7.20180305 The implementation is not as intelligent as I’d like it to be. The protocol version that a server last used is stored on the connection object and is lost when we clean up idle connections. Although it is likely that a new connection will speak the same protocol as the previous one there is little harm in going in to protocol detection mode again (we declare ourselves HTTP/1.1) other than the problem of using transfer encodings on an initial POST. In particular, we only drop out of keep-alive mode when the server has actually responded with an HTTP/1.0 response. #38 Make Pyslet run under Python 3 See above for details. #43 Fixes for Python running on Windows This issue came back again, both unicode file name problems and further problems due to timing in unittests. Fixed this time by mocking and monkey-patching the time.time function in the QTI tests. #47 Improve CharClass-derived doc strings Fixed - no functional changes. #49 Typo in pyslet/odata2/csdl.py Fixed OData serialisation of LongDescription element - thanks to @thomaseitler #51 Fix processing of dates in JSON format OData communication by the #server We now accept ISO string formatted dates for both DateTime and DateTimeOffset. Note that providing a timezone other than Z (+00:00) when setting a DateTime will cause the time to be zone-shifted to UTC before the value is set. Thanks to @ianwj5int. #53 Use datetime.date to create datetime object You can now set DateTimeValue using a standard python datetime.date, the value is extended to be 00:00:00 on that date. Thanks to @nmichaud #54 Fix use of super to remove self Fixed Atom Date handling bug, thanks to @nmichaud #55 Replace print_exception with logging (this includes the traceback) Thanks to @ianwj5int for reporting. #56 Garbage received when server delays response This was caused by a bug when handling 401 responses in HTTP client The issue affected any response that was received as a result of a resend (after a redirect or 401 response). The stream used to receive the data in the follow-up request was not being reset correctly and this resulted in a chunk of 0x00 bytes being written before the actual content. This bug was discovered following changes in the 20160209 build when StringIO was replaced with BytesIO for Python 3 compatibility. StringIO.truncate moves the stream pointer, BytesIO.truncate does not. As a result all resends where the 3xx or 401 response had a non-zero length body were being affected. Previously the bug only affected the rarer use case of resends of streamed downloads to real files, i.e., requests created by passing an open file in the res_body argument of ClientRequest. With thanks to @karurosu for reporting. #58 OData default values (PUT/PATCH/MERGE) Warning: if you use Pyslet for an OData server please check that PUTs are still working as required. Changed the SQL data stores to use DEFAULT values from the metadata file as part of the CREATE TABLE queries. Modified update_entity in memds, and SQL storage layers to use MERGE semantics by default, added option to enable replace (PUT) semantics using column defaults. This differs from the previous (incorrect behaviour) where unselected properties were set to NULL. 1.3. Version 0.7.20170805 3 Pyslet Documentation, Release 0.7.20180305 Updated OData server to support MERGE and ensured that PUT now uses the correct semantics (set to default instead of NULL) for values missing from the incoming request. Improved error handling to reduce log noise in SQL layer. #60 authentication example in docs Added a first cut at a documentation page for HTTP auth. #61 Add support for NTLM Experimental support for NTLM authentication now available using the python-ntlm3 module from pip/GitHub which must be installed before you can use NTLM. The module is in pyslet.ntlmauth and it can be used in a similar way to Basic auth (see set_ntlm_credentials for details.) Improved handling of error responses in all HTTP requests (includes a Python 3 bug fix) to enable the connection to be kept open more easily during pipelined requests that are terminated early by a final response from the server. This allows a large POST that generates a 401 response to abort sending of chunked bodies and retry without opening a new connection - vital for NTLM which is connection based. Added automated resend after 417 Expectation failed responses as per latest HTTP guidance. (Even for POST re- quests!) #64 Add a LICENSE file Added to distribution #65 syntax error in sqlds.SQLCollectionBase.sql_expression_substring Also added an override for SQLite given the lack of support for the standard substring syntax.