WIP: complete manual inspection of all significant changes

docs/library/array.rst

@@ -16,14 +16,14 @@ Classes
 .. class:: array.array(typecode, [iterable])
 
     Create array with elements of given type. Initial contents of the
-    array are given by *iterable*. If it is not provided, an empty
+    array are given by an `iterable`. If it is not provided, an empty
     array is created.
 
     .. method:: append(val)
 
-        Append new element *val* to the end of array, growing it.
+        Append new element ``val`` to the end of array, growing it.
 
     .. method:: extend(iterable)
 
-        Append new elements as contained in *iterable* to the end of
+        Append new elements as contained in `iterable` to the end of
         array, growing it.

docs/library/ujson.rst

@@ -16,20 +16,20 @@ Functions
 
 .. function:: dump(obj, stream)
 
-   Serialise *obj* to a JSON string, writing it to the given *stream*.
+   Serialise ``obj`` to a JSON string, writing it to the given *stream*.
 
 .. function:: dumps(obj)
 
-   Return *obj* represented as a JSON string.
+   Return ``obj`` represented as a JSON string.
 
 .. function:: load(stream)
 
-   Parse the given *stream*, interpreting it as a JSON string and
+   Parse the given ``stream``, interpreting it as a JSON string and
    deserialising the data to a Python object.  The resulting object is
    returned.
 
    Parsing continues until end-of-file is encountered.
-   A :exc:`ValueError` is raised if the data in *stream* is not correctly formed.
+   A :exc:`ValueError` is raised if the data in ``stream`` is not correctly formed.
 
 .. function:: loads(str)
 

docs/library/ure.rst

@@ -69,7 +69,6 @@ Functions
 .. data:: DEBUG
 
    Flag value, display debug information about compiled expression.
-   (Availability depends on `MicroPython port`.)
 
 
 .. _regex:

docs/reference/packages.rst

@@ -1,312 +0,0 @@
-Distribution packages, package management, and deploying applications
-=====================================================================
-
-Just as the "big" Python, MicroPython supports creation of "third party"
-packages, distributing them, and easily installing them in each user's
-environment. This chapter discusses how these actions are achieved.
-Some familiarity with Python packaging is recommended.
-
-Overview
---------
-
-Steps below represent a high-level workflow when creating and consuming
-packages:
-
-1. Python modules and packages are turned into distribution package
-   archives, and published at the Python Package Index (PyPI).
-2. `upip` package manager can be used to install a distribution package
-   on a `MicroPython port` with networking capabilities (for example,
-   on the Unix port).
-3. For ports without networking capabilities, an "installation image"
-   can be prepared on the Unix port, and transferred to a device by
-   suitable means.
-4. For low-memory ports, the installation image can be frozen as the
-   bytecode into MicroPython executable, thus minimizing the memory
-   storage overheads.
-
-The sections below describe this process in details.
-
-Distribution packages
----------------------
-
-Python modules and packages can be packaged into archives suitable for
-transfer between systems, storing at the well-known location (PyPI),
-and downloading on demand for deployment. These archives are known as
-*distribution packages* (to differentiate them from Python packages
-(means to organize Python source code)).
-
-The MicroPython distribution package format is a well-known tar.gz
-format, with some adaptations however. The Gzip compressor, used as
-an external wrapper for TAR archives, by default uses 32KB dictionary
-size, which means that to uncompress a compressed stream, 32KB of
-contguous memory needs to be allocated. This requirement may be not
-satisfiable on low-memory devices, which may have total memory available
-less than that amount, and even if not, a contiguous block like that
-may be hard to allocate due to memory fragmentation. To accommodate
-these constraints, MicroPython distribution packages use Gzip compression
-with the dictionary size of 4K, which should be a suitable compromise
-with still achieving some compression while being able to uncompressed
-even by the smallest devices.
-
-Besides the small compression dictionary size, MicroPython distribution
-packages also have other optimizations, like removing any files from
-the archive which aren't used by the installation process. In particular,
-`upip` package manager doesn't execute ``setup.py`` during installation
-(see below), and thus that file is not included in the archive.
-
-At the same time, these optimizations make MicroPython distribution
-packages not compatible with `CPython`'s package manager, ``pip``.
-This isn't considered a big problem, because:
-
-1. Packages can be installed with `upip`, and then can be used with
-   CPython (if they are compatible with it).
-2. In the other direction, majority of CPython packages would be
-   incompatible with MicroPython by various reasons, first of all,
-   the reliance on features not implemented by MicroPython.
-
-Summing up, the MicroPython distribution package archives are highly
-optimized for MicroPython's target environments, which are highly
-resource constrained devices.
-
-
-``upip`` package manager
-------------------------
-
-MicroPython distribution packages are intended to be installed using
-the `upip` package manager. `upip` is a Python application which is
-usually distributed (as frozen bytecode) with network-enabled
-`MicroPython ports <MicroPython port>`. At the very least,
-`upip` is available in the `MicroPython Unix port`.
-
-On any `MicroPython port` providing `upip`, it can be accessed as
-following::
-
-    import upip
-    upip.help()
-    upip.install(package_or_package_list, [path])
-
-Where *package_or_package_list* is the name of a distribution
-package to install, or a list of such names to install multiple
-packages. Optional *path* parameter specifies filesystem
-location to install under and defaults to the standard library
-location (see below).
-
-An example of installing a specific package and then using it::
-
-    >>> import upip
-    >>> upip.install("micropython-pystone_lowmem")
-    [...]
-    >>> import pystone_lowmem
-    >>> pystone_lowmem.main()
-
-Note that the name of Python package and the name of distribution
-package for it in general don't have to match, and oftentimes they
-don't. This is because PyPI provides a central package repository
-for all different Python implementations and versions, and thus
-distribution package names may need to be namespaced for a particular
-implementation. For example, all packages from `micropython-lib`
-follow this naming convention: for a Python module or package named
-``foo``, the distribution package name is ``micropython-foo``.
-
-For the ports which run MicroPython executable from the OS command
-prompts (like the Unix port), `upip` can be (and indeed, usually is)
-run from the command line instead of MicroPython's own REPL. The
-commands which corresponds to the example above are::
-
-    micropython -m upip -h
-    micropython -m upip install [-p <path>] <packages>...
-    micropython -m upip install micropython-pystone_lowmem
-
-[TODO: Describe installation path.]
-
-
-Cross-installing packages
--------------------------
-
-For `MicroPython ports <MicroPython port>` without native networking
-capabilities, the recommend process is "cross-installing" them into a
-"directory image" using the `MicroPython Unix port`, and then
-transferring this image to a device by suitable means.
-
-Installing to a directory image involves using ``-p`` switch to `upip`::
-
-    micropython -m upip install -p install_dir micropython-pystone_lowmem
-
-After this command, the package content (and contents of every depenency
-packages) will be available in the ``install_dir/`` subdirectory. You
-would need to transfer contents of this directory (without the
-``install_dir/`` prefix) to the device, at the suitable location, where
-it can be found by the Python ``import`` statement (see discussion of
-the `upip` installation path above).
-
-
-Cross-installing packages with freezing
----------------------------------------
-
-For the low-memory `MicroPython ports <MicroPython port>`, the process
-described in the previous section does not provide the most efficient
-resource usage,because the packages are installed in the source form,
-so need to be compiled to the bytecome on each import. This compilation
-requires RAM, and the resulting bytecode is also stored in RAM, reducing
-its amount available for storing application data. Moreover, the process
-above requires presence of the filesystem on a device, and the most
-resource-constrained devices may not even have it.
-
-The bytecode freezing is a process which resolves all the issues
-mentioned above:
-
-* The source code is pre-compiled into bytecode and store as such.
-* The bytecode is stored in ROM, not RAM.
-* Filesystem is not required for frozen packages.
-
-Using frozen bytecode requires building the executable (firmware)
-for a given `MicroPython port` from the C source code. Consequently,
-the process is:
-
-1. Follow the instructions for a particular port on setting up a
-   toolchain and building the port. For example, for ESP8266 port,
-   study instructions in ``ports/esp8266/README.md`` and follow them.
-   Make sure you can build the port and deploy the resulting
-   executable/firmware successfully before proceeding to the next steps.
-2. Build `MicroPython Unix port` and make sure it is in your PATH and
-   you can execute ``micropython``.
-3. Change to port's directory (e.g. ``ports/esp8266/`` for ESP8266).
-4. Run ``make clean-frozen``. This step cleans up any previous
-   modules which were installed for freezing (consequently, you need
-   to skip this step to add additional modules, instead of starting
-   from scratch).
-5. Run ``micropython -m upip install -p modules <packages>...`` to
-   install packages you want to freeze.
-6. Run ``make clean``.
-7. Run ``make``.
-
-After this, you should have the executable/firmware with modules as
-the bytecode inside, which you can deploy the usual way.
-
-Few notes:
-
-1. Step 5 in the sequence above assumes that the distribution package
-   is available from PyPI. If that is not the case, you would need
-   to copy Python source files manually to ``modules/`` subdirectory
-   of the port port directory. (Note that upip does not support
-   installing from e.g. version control repositories).
-2. The firmware for baremetal devices usually has size restrictions,
-   so adding too many frozen modules may overflow it. Usually, you
-   would get a linking error if this happens. However, in some cases,
-   an image may be produced, which is not runnable on a device. Such
-   cases are in general bugs, and should be reported and further
-   investigated. If you face such a situation, as an initial step,
-   you may want to decrease the amount of frozen modules included.
-
-
-Creating distribution packages
-------------------------------
-
-Distribution packages for MicroPython are created in the same manner
-as for CPython or any other Python implementation, see references at
-the end of chapter. Setuptools (instead of distutils) should be used,
-because distutils do not support dependencies and other features. "Source
-distribution" (``sdist``) format is used for packaging. The post-processing
-discussed above, (and pre-processing discussed in the following section)
-is achieved by using custom ``sdist`` command for setuptools. Thus, packaging
-steps remain the same as for the standard setuptools, the user just
-needs to override ``sdist`` command implementation by passing the
-appropriate argument to ``setup()`` call::
-
-    from setuptools import setup
-    import sdist_upip
-
-    setup(
-        ...,
-        cmdclass={'sdist': sdist_upip.sdist}
-    )
-
-The sdist_upip.py module as referenced above can be found in
-`micropython-lib`:
-https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py
-
-
-Application resources
----------------------
-
-A complete application, besides the source code, oftentimes also consists
-of data files, e.g. web page templates, game images, etc. It's clear how
-to deal with those when application is installed manually - you just put
-those data files in the filesystem at some location and use the normal
-file access functions.
-
-The situation is different when deploying applications from packages - this
-is more advanced, streamlined and flexible way, but also requires more
-advanced approach to accessing data files. This approach is treating
-the data files as "resources", and abstracting away access to them.
-
-Python supports resource access using its "setuptools" library, using
-``pkg_resources`` module. MicroPython, following its usual approach,
-implements subset of the functionality of that module, specifically
-``pkg_resources.resource_stream(package, resource)`` function.
-The idea is that an application calls this function, passing a
-resource identifier, which is a relative path to data file within
-the specified package (usually top-level application package). It
-returns a stream object which can be used to access resource contents.
-Thus, the ``resource_stream()`` emulates interface of the standard
-`open()` function.
-
-Implementation-wise, ``resource_stream()`` uses file operations
-underlyingly, if distribution package is install in the filesystem.
-However, it also supports functioning without the underlying filesystem,
-e.g. if the package is frozen as the bytecode. This however requires
-an extra intermediate step when packaging application - creation of
-"Python resource module".
-
-The idea of this module is to convert binary data to a Python bytes
-object, and put it into the dictionary, indexed by the resource name.
-This conversion is done automatically using overridden ``sdist`` command
-described in the previous section.
-
-Let's trace the complete process using the following example. Suppose
-your application has the following structure::
-
-    my_app/
-        __main__.py
-        utils.py
-        data/
-            page.html
-            image.png
-
-``__main__.py`` and ``utils.py`` should access resources using the
-following calls::
-
-    import pkg_resources
-
-    pkg_resources.resource_stream(__name__, "data/page.html")
-    pkg_resources.resource_stream(__name__, "data/image.png")
-
-You can develop and debug using the `MicroPython Unix port` as usual.
-When time comes to make a distribution package out of it, just use
-overridden "sdist" command from sdist_upip.py module as described in
-the previous section.
-
-This will create a Python resource module named ``R.py``, based on the
-files declared in ``MANIFEST`` or ``MANIFEST.in`` files (any non-``.py``
-file will be considered a resource and added to ``R.py``) - before
-proceeding with the normal packaging steps.
-
-Prepared like this, your application will work both when deployed to
-filesystem and as frozen bytecode.
-
-If you would like to debug ``R.py`` creation, you can run::
-
-    python3 setup.py sdist --manifest-only
-
-Alternatively, you can use tools/mpy_bin2res.py script from the
-MicroPython distribution, in which can you will need to pass paths
-to all resource files::
-
-    mpy_bin2res.py data/page.html data/image.png
-
-References
-----------
-
-* Python Packaging User Guide: https://packaging.python.org/
-* Setuptools documentation: https://setuptools.readthedocs.io/
-* Distutils documentation: https://docs.python.org/3/library/distutils.html

extmod/lwip-include/lwipopts.h

@@ -23,7 +23,7 @@ typedef uint32_t sys_prot_t;
 #define LWIP_NETCONN 0
 #define LWIP_SOCKET 0
 
-#ifdef MICROPY_PY_LWIP_SLIP
+#if MICROPY_PY_LWIP_SLIP
 #define LWIP_HAVE_SLIPIF 1
 #endif
 

extmod/modlwip.c

@@ -69,12 +69,12 @@
 #define ip_reset_option(pcb, opt) ((pcb)->so_options &= ~(opt))
 #endif
 
-#ifdef MICROPY_PY_LWIP_SLIP
+#if MICROPY_PY_LWIP_SLIP
 #include "netif/slipif.h"
 #include "lwip/sio.h"
 #endif
 
-#ifdef MICROPY_PY_LWIP_SLIP
+#if MICROPY_PY_LWIP_SLIP
 /******************************************************************************/
 // Slip object for modlwip. Requires a serial driver for the port that supports
 // the lwip serial callback functions.
@@ -1419,7 +1419,7 @@ STATIC mp_obj_t lwip_print_pcbs() {
 }
 MP_DEFINE_CONST_FUN_OBJ_0(lwip_print_pcbs_obj, lwip_print_pcbs);
 
-#ifdef MICROPY_PY_LWIP
+#if MICROPY_PY_LWIP
 
 STATIC const mp_rom_map_elem_t mp_module_lwip_globals_table[] = {
     { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_lwip) },
@@ -1429,7 +1429,7 @@ STATIC const mp_rom_map_elem_t mp_module_lwip_globals_table[] = {
     { MP_ROM_QSTR(MP_QSTR_print_pcbs), MP_ROM_PTR(&lwip_print_pcbs_obj) },
     // objects
     { MP_ROM_QSTR(MP_QSTR_socket), MP_ROM_PTR(&lwip_socket_type) },
-#ifdef MICROPY_PY_LWIP_SLIP
+#if MICROPY_PY_LWIP_SLIP
     { MP_ROM_QSTR(MP_QSTR_slip), MP_ROM_PTR(&lwip_slip_type) },
 #endif
     // class constants

extmod/moduhashlib.c

@@ -29,6 +29,14 @@
 
 #include "py/runtime.h"
 
+static void check_not_unicode(const mp_obj_t arg) {
+#if MICROPY_CPYTHON_COMPAT
+    if (MP_OBJ_IS_STR(arg)) {
+        mp_raise_TypeError("a bytes-like object is required");
+    }
+#endif
+}
+
 #if MICROPY_PY_UHASHLIB
 
 #if MICROPY_PY_UHASHLIB_SHA256
@@ -47,20 +55,12 @@
 #include "lib/axtls/crypto/crypto.h"
 #endif
 
-static void check_not_unicode(const mp_obj_t arg) {
-#if MICROPY_CPYTHON_COMPAT
-    if (MP_OBJ_IS_STR(arg)) {
-        mp_raise_TypeError("a bytes-like object is required");
-    }
-#endif
-
 #if MICROPY_SSL_MBEDTLS
 #include "mbedtls/sha1.h"
 #endif
 
 #endif
 
-}
 
 typedef struct _mp_obj_hash_t {
     mp_obj_base_t base;

extmod/vfs_fat.h

@@ -63,11 +63,11 @@ typedef struct _pyb_file_obj_t {
 
 // These should be general types (mpy TOOD says so). In micropython, these are defined in
 // mpconfigport.h.
-#define mp_type_fileio mp_type_vfs_fat_fileio
+////////////#define mp_type_fileio mp_type_vfs_fat_fileio
-#define mp_type_textio mp_type_vfs_fat_textio
+////////////#define mp_type_textio mp_type_vfs_fat_textio
 
-extern const mp_obj_type_t mp_type_fileio;
+////////////extern const mp_obj_type_t mp_type_fileio;
-extern const mp_obj_type_t mp_type_textio;
+////////////extern const mp_obj_type_t mp_type_textio;
 
 extern const byte fresult_to_errno_table[20];
 extern const mp_obj_type_t mp_fat_vfs_type;

extmod/vfs_fat_diskio.c

@@ -36,6 +36,8 @@
 #include "py/mphal.h"
 
 #include "py/runtime.h"
+#include "py/binary.h"
+#include "py/objarray.h"
 #include "lib/oofatfs/ff.h"
 #include "lib/oofatfs/diskio.h"
 #include "extmod/vfs_fat.h"
@@ -51,62 +53,6 @@ STATIC fs_user_mount_t *disk_get_device(void *bdev) {
     return (fs_user_mount_t*)bdev;
 }
 
-/*-----------------------------------------------------------------------*/
-/* Initialize a Drive                                                    */
-/*-----------------------------------------------------------------------*/
-
-STATIC
-DSTATUS disk_initialize (
-    bdev_t pdrv              /* Physical drive number (0..) */
-)
-{
-    fs_user_mount_t *vfs = disk_get_device(pdrv);
-    if (vfs == NULL) {
-        return STA_NOINIT;
-    }
-
-    if (vfs->flags & FSUSER_HAVE_IOCTL) {
-        // new protocol with ioctl; call ioctl(INIT, 0)
-        vfs->u.ioctl[2] = MP_OBJ_NEW_SMALL_INT(BP_IOCTL_INIT);
-        vfs->u.ioctl[3] = MP_OBJ_NEW_SMALL_INT(0); // unused
-        mp_obj_t ret = mp_call_method_n_kw(2, 0, vfs->u.ioctl);
-        if (ret != mp_const_none && MP_OBJ_SMALL_INT_VALUE(ret) != 0) {
-            // error initialising
-            return STA_NOINIT;
-        }
-    }
-
-    if (vfs->writeblocks[0] == MP_OBJ_NULL) {
-        return STA_PROTECT;
-    } else {
-        return 0;
-    }
-}
-
-/*-----------------------------------------------------------------------*/
-/* Get Disk Status                                                       */
-/*-----------------------------------------------------------------------*/
-
-STATIC
-DSTATUS disk_status (
-    bdev_t pdrv      /* Physical drive nmuber (0..) */
-)
-{
-    fs_user_mount_t *vfs = disk_get_device(pdrv);
-    if (vfs == NULL) {
-        return STA_NOINIT;
-    }
-
-    // This is used to determine the writeability of the disk from MicroPython.
-    // So, if its USB writable we make it read-only from MicroPython.
-    if (vfs->writeblocks[0] == MP_OBJ_NULL ||
-        (vfs->flags & FSUSER_USB_WRITABLE) != 0) {
-        return STA_PROTECT;
-    } else {
-        return 0;
-    }
-}
-
 /*-----------------------------------------------------------------------*/
 /* Read Sector(s)                                                        */
 /*-----------------------------------------------------------------------*/
@@ -208,27 +154,56 @@ DRESULT disk_ioctl (
         return RES_PARERR;
     }
 
+    // First part: call the relevant method of the underlying block device
+    mp_obj_t ret = mp_const_none;
     if (vfs->flags & FSUSER_HAVE_IOCTL) {
         // new protocol with ioctl
+        static const uint8_t op_map[8] = {
+            [CTRL_SYNC] = BP_IOCTL_SYNC,
+            [GET_SECTOR_COUNT] = BP_IOCTL_SEC_COUNT,
+            [GET_SECTOR_SIZE] = BP_IOCTL_SEC_SIZE,
+            [IOCTL_INIT] = BP_IOCTL_INIT,
+        };
+        uint8_t bp_op = op_map[cmd & 7];
+        if (bp_op != 0) {
+            vfs->u.ioctl[2] = MP_OBJ_NEW_SMALL_INT(bp_op);
+            vfs->u.ioctl[3] = MP_OBJ_NEW_SMALL_INT(0); // unused
+            ret = mp_call_method_n_kw(2, 0, vfs->u.ioctl);
+        }
+    } else {
+        // old protocol with sync and count
+        switch (cmd) {
+            case CTRL_SYNC:
+                if (vfs->u.old.sync[0] != MP_OBJ_NULL) {
+                    mp_call_method_n_kw(0, 0, vfs->u.old.sync);
+                }
+                break;
+
+            case GET_SECTOR_COUNT:
+                ret = mp_call_method_n_kw(0, 0, vfs->u.old.count);
+                break;
+
+            case GET_SECTOR_SIZE:
+                // old protocol has fixed sector size of 512 bytes
+                break;
+
+            case IOCTL_INIT:
+                // old protocol doesn't have init
+                break;
+        }
+    }
+
+    // Second part: convert the result for return
     switch (cmd) {
         case CTRL_SYNC:
-                vfs->u.ioctl[2] = MP_OBJ_NEW_SMALL_INT(BP_IOCTL_SYNC);
-                vfs->u.ioctl[3] = MP_OBJ_NEW_SMALL_INT(0); // unused
-                mp_call_method_n_kw(2, 0, vfs->u.ioctl);
             return RES_OK;
 
         case GET_SECTOR_COUNT: {
-                vfs->u.ioctl[2] = MP_OBJ_NEW_SMALL_INT(BP_IOCTL_SEC_COUNT);
-                vfs->u.ioctl[3] = MP_OBJ_NEW_SMALL_INT(0); // unused
-                mp_obj_t ret = mp_call_method_n_kw(2, 0, vfs->u.ioctl);
             *((DWORD*)buff) = mp_obj_get_int(ret);
             return RES_OK;
         }
 
         case GET_SECTOR_SIZE: {
-                vfs->u.ioctl[2] = MP_OBJ_NEW_SMALL_INT(BP_IOCTL_SEC_SIZE);
-                vfs->u.ioctl[3] = MP_OBJ_NEW_SMALL_INT(0); // unused
-                mp_obj_t ret = mp_call_method_n_kw(2, 0, vfs->u.ioctl);
             if (ret == mp_const_none) {
                 // Default sector size
                 *((WORD*)buff) = 512;
@@ -247,55 +222,23 @@ DRESULT disk_ioctl (
             return RES_OK;
 
         case IOCTL_INIT:
-                *((DSTATUS*)buff) = disk_initialize(pdrv);
+        case IOCTL_STATUS: {
-                return RES_OK;
+            DSTATUS stat;
-
+            if (ret != mp_const_none && MP_OBJ_SMALL_INT_VALUE(ret) != 0) {
-            case IOCTL_STATUS:
+                // error initialising
-                *((DSTATUS*)buff) = disk_status(pdrv);
+                stat = STA_NOINIT;
-                return RES_OK;
+            } else if (vfs->writeblocks[0] == MP_OBJ_NULL) {
-
+                stat = STA_PROTECT;
-            default:
-                return RES_PARERR;
-        }
             } else {
-        // old protocol with sync and count
+                stat = 0;
-        switch (cmd) {
-            case CTRL_SYNC:
-                if (vfs->u.old.sync[0] != MP_OBJ_NULL) {
-                    mp_call_method_n_kw(0, 0, vfs->u.old.sync);
             }
-                return RES_OK;
+            *((DSTATUS*)buff) = stat;
-
-            case GET_SECTOR_COUNT: {
-                mp_obj_t ret = mp_call_method_n_kw(0, 0, vfs->u.old.count);
-                *((DWORD*)buff) = mp_obj_get_int(ret);
             return RES_OK;
         }
 
-            case GET_SECTOR_SIZE:
-                *((WORD*)buff) = 512; // old protocol had fixed sector size
-                #if _MAX_SS != _MIN_SS
-                // need to store ssize because we use it in disk_read/disk_write
-                vfs->fatfs.ssize = 512;
-                #endif
-                return RES_OK;
-
-            case GET_BLOCK_SIZE:
-                *((DWORD*)buff) = 1; // erase block size in units of sector size
-                return RES_OK;
-
-            case IOCTL_INIT:
-                *((DSTATUS*)buff) = disk_initialize(pdrv);
-                return RES_OK;
-
-            case IOCTL_STATUS:
-                *((DSTATUS*)buff) = disk_status(pdrv);
-                return RES_OK;
-
         default:
             return RES_PARERR;
     }
-    }
 }
 
 #endif // MICROPY_VFS && MICROPY_VFS_FAT

lib/axtls

@@ -1 +1 @@
-Subproject commit dac9176cac58cc5e49669a9a4d404a6f6dd7cc10
+Subproject commit 43a6e6bd3bbc03dc501e16b89fba0ef042ed3ea0

lib/mp-readline/readline.h

@@ -26,8 +26,6 @@
 #ifndef MICROPY_INCLUDED_LIB_MP_READLINE_READLINE_H
 #define MICROPY_INCLUDED_LIB_MP_READLINE_READLINE_H
 
-#include "py/misc.h"
-
 #define CHAR_CTRL_A (1)
 #define CHAR_CTRL_B (2)
 #define CHAR_CTRL_C (3)

ports/esp8266/Makefile

@@ -77,7 +77,7 @@ SRC_C = \
 	esp_init_data.c \
 	gccollect.c \
 	lexerstr32.c \
-	espuart.c \
+	uart.c \
 	esppwm.c \
 	espneopixel.c \
 	intr.c \
@@ -193,7 +193,10 @@ LIB_SRC_C += \
 	lib/oofatfs/option/unicode.c
 endif
 
+DRIVERS_SRC_C = $(addprefix drivers/,\
 	bus/softspi.c \
+	)
+
 SRC_S = \
 	gchelper.s \
 
@@ -206,6 +209,7 @@ OBJ += $(addprefix $(BUILD)/, $(SRC_SHARED_MODULE_EXPANDED:.c=.o))
 OBJ += $(addprefix $(BUILD)/, $(STM_SRC_C:.c=.o))
 OBJ += $(addprefix $(BUILD)/, $(EXTMOD_SRC_C:.c=.o))
 OBJ += $(addprefix $(BUILD)/, $(LIB_SRC_C:.c=.o))
+OBJ += $(addprefix $(BUILD)/, $(DRIVERS_SRC_C:.c=.o))
 
 # List of sources for qstr extraction
 SRC_QSTR += $(SRC_C) $(SRC_COMMON_HAL_EXPANDED) $(SRC_SHARED_MODULE_EXPANDED) $(STM_SRC_C) $(EXTMOD_SRC_C) $(DRIVERS_SRC_C)

ports/esp8266/common-hal/busio/UART.c

@@ -29,7 +29,7 @@
 #include "shared-bindings/busio/UART.h"
 
 #include "ets_sys.h"
-#include "espuart.h"
+#include "uart.h"
 
 #include "py/nlr.h"
 

ports/esp8266/common-hal/os/__init__.c

@@ -27,9 +27,11 @@
 
 #include <string.h>
 
+#include "esp_mphal.h"
 #include "etshal.h"
 #include "py/objtuple.h"
 #include "py/objstr.h"
+#include "extmod/misc.h"
 #include "genhdr/mpversion.h"
 #include "user_interface.h"
 
@@ -75,19 +77,6 @@ bool common_hal_os_urandom(uint8_t* buffer, uint32_t length) {
             i++;
             new_random >>= 8;
         }
-// We wrap the mp_uos_dupterm function to detect if a UART is attached or not
-mp_obj_t os_dupterm(size_t n_args, const mp_obj_t *args) {
-    mp_obj_t prev_obj = mp_uos_dupterm_obj.fun.var(n_args, args);
-    if (mp_obj_get_type(args[0]) == &pyb_uart_type) {
-        ++uart_attached_to_dupterm;
-    }
-    if (mp_obj_get_type(prev_obj) == &pyb_uart_type) {
-        --uart_attached_to_dupterm;
-    }
-    return prev_obj;
-}
-STATIC MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(os_dupterm_obj, 1, 2, os_dupterm);
-
     }
     return true;
 }

ports/esp8266/esp_mphal.c

@@ -27,7 +27,7 @@
 #include <stdio.h>
 #include "ets_sys.h"
 #include "etshal.h"
-#include "espuart.h"
+#include "uart.h"
 #include "esp_mphal.h"
 #include "user_interface.h"
 #include "ets_alt_task.h"

ports/esp8266/espneopixel.c

@@ -16,7 +16,7 @@
 
 #define NEO_KHZ400 (1)
 
-void /*ICACHE_RAM_ATTR*/ esp_neopixel_write(uint8_t pin, uint8_t *pixels, uint32_t numBytes) {
+void /*ICACHE_RAM_ATTR*/ esp_neopixel_write(uint8_t pin, uint8_t *pixels, uint32_t numBytes, bool is800KHz) {
 
   uint8_t *p, *end, pix, mask;
   uint32_t t, time0, time1, period, c, startTime, pinMask;
@@ -30,10 +30,19 @@ void /*ICACHE_RAM_ATTR*/ esp_neopixel_write(uint8_t pin, uint8_t *pixels, uint32
 
   uint32_t fcpu = system_get_cpu_freq() * 1000000;
 
-
+#ifdef NEO_KHZ400
+  if(is800KHz) {
+#endif
     time0  = fcpu / 2857143; // 0.35us
     time1  = fcpu / 1250000; // 0.8us
     period = fcpu /  800000; // 1.25us per bit
+#ifdef NEO_KHZ400
+  } else { // 400 KHz bitstream
+    time0  = fcpu / 2000000; // 0.5uS
+    time1  = fcpu /  833333; // 1.2us
+    period = fcpu /  400000; // 2.5us per bit
+  }
+#endif
 
   uint32_t irq_state = mp_hal_quiet_timing_enter();
   for(t = time0;; t = time0) {

ports/esp8266/espneopixel.h

@@ -1,6 +1,6 @@
 #ifndef MICROPY_INCLUDED_ESP8266_ESPNEOPIXEL_H
 #define MICROPY_INCLUDED_ESP8266_ESPNEOPIXEL_H
 
-void esp_neopixel_write(uint8_t pin, uint8_t *pixels, uint32_t numBytes);
+void esp_neopixel_write(uint8_t pin, uint8_t *pixels, uint32_t numBytes, bool is800KHz);
 
 #endif // MICROPY_INCLUDED_ESP8266_ESPNEOPIXEL_H

ports/esp8266/etshal.h

@@ -6,12 +6,14 @@
 // see http://esp8266-re.foogod.com/wiki/Random_Number_Generator
 #define WDEV_HWRNG ((volatile uint32_t*)0x3ff20e44)
 
+void ets_delay_us(uint16_t us);
 void ets_intr_lock(void);
 void ets_intr_unlock(void);
 void ets_isr_mask(uint32_t mask);
 void ets_isr_unmask(uint32_t mask);
 void ets_isr_attach(int irq_no, void (*handler)(void *), void *arg);
 void ets_install_putc1();
+void uart_div_modify(uint8_t uart, uint32_t divisor);
 void ets_set_idle_cb(void (*handler)(void *), void *arg);
 
 void ets_timer_arm_new(os_timer_t *tim, uint32_t millis, bool repeat, bool is_milli_timer);
@@ -30,6 +32,12 @@ void MD5Init(MD5_CTX *context);
 void MD5Update(MD5_CTX *context, const void *data, unsigned int len);
 void MD5Final(unsigned char digest[16], MD5_CTX *context);
 
+// These prototypes are for recent SDKs with "malloc tracking"
+void *pvPortMalloc(size_t sz, const char *fname, unsigned line);
+void *pvPortZalloc(size_t sz, const char *fname, unsigned line);
+void *pvPortRealloc(void *p, unsigned sz, const char *fname, unsigned line);
+void vPortFree(void *p, const char *fname, unsigned line);
+
 uint32_t SPIRead(uint32_t offset, void *buf, uint32_t len);
 uint32_t SPIWrite(uint32_t offset, const void *buf, uint32_t len);
 uint32_t SPIEraseSector(int sector);

ports/esp8266/machine_uart.c

@@ -29,7 +29,7 @@
 #include <string.h>
 
 #include "ets_sys.h"
-#include "espuart.h"
+#include "uart.h"
 
 #include "py/runtime.h"
 #include "py/stream.h"

ports/esp8266/main.c

@@ -111,25 +111,6 @@ STATIC void mp_reset(void) {
     readline_init0();
     dupterm_task_init();
     pwmout_reset();
-
-    // Check if there are any dupterm objects registered and if not then
-    // activate UART(0), or else there will never be any chance to get a REPL
-    size_t idx;
-    for (idx = 0; idx < MICROPY_PY_OS_DUPTERM; ++idx) {
-        if (MP_STATE_VM(dupterm_objs[idx]) != MP_OBJ_NULL) {
-            break;
-        }
-    }
-    if (idx == MICROPY_PY_OS_DUPTERM) {
-        mp_obj_t args[2];
-        args[0] = MP_OBJ_NEW_SMALL_INT(0);
-        args[1] = MP_OBJ_NEW_SMALL_INT(115200);
-        args[0] = pyb_uart_type.make_new(&pyb_uart_type, 2, 0, args);
-        args[1] = MP_OBJ_NEW_SMALL_INT(1);
-        extern mp_obj_t os_dupterm(size_t n_args, const mp_obj_t *args);
-        os_dupterm(2, args);
-        mp_hal_stdout_tx_str("Activated UART(0) for REPL\r\n");
-    }
 }
 
 bool soft_reset(void) {

ports/esp8266/modesp.c

@@ -30,7 +30,7 @@
 #include "py/runtime.h"
 #include "py/mperrno.h"
 #include "py/mphal.h"
-#include "espuart.h"
+#include "uart.h"
 #include "user_interface.h"
 #include "mem.h"
 #include "modmachine.h"

ports/esp8266/modnetwork.c

@@ -426,9 +426,9 @@ STATIC mp_obj_t esp_config(size_t n_args, const mp_obj_t *args, mp_map_t *kwargs
         }
         case MP_QSTR_essid:
             if (self->if_id == STATION_IF) {
-                val = mp_obj_new_str((char*)cfg.sta.ssid, strlen((char*)cfg.sta.ssid), false);
+                val = mp_obj_new_str((char*)cfg.sta.ssid, strlen((char*)cfg.sta.ssid));
             } else {
-                val = mp_obj_new_str((char*)cfg.ap.ssid, cfg.ap.ssid_len, false);
+                val = mp_obj_new_str((char*)cfg.ap.ssid, cfg.ap.ssid_len);
             }
             break;
         case MP_QSTR_hidden:

ports/esp8266/modules/inisetup.py

@@ -48,8 +48,6 @@ def setup():
 # This file is executed on every boot (including wake-boot from deepsleep)
 #import esp
 #esp.osdebug(None)
-import uos, machine
-uos.dupterm(machine.UART(0, 115200), 1)
 import gc
 #import webrepl
 #webrepl.start()

ports/esp8266/modules/webrepl_setup.py

@@ -34,6 +34,12 @@ def exists(fname):
     except OSError:
         return False
 
+def copy_stream(s_in, s_out):
+    buf = bytearray(64)
+    while 1:
+        sz = s_in.readinto(buf)
+        s_out.write(buf, sz)
+
 
 def get_daemon_status():
     with open(RC) as f:
@@ -44,6 +50,10 @@ def get_daemon_status():
                 return True
         return None
 
+def add_daemon():
+    with open(RC) as old_f, open(RC + ".tmp", "w") as new_f:
+        new_f.write("import webrepl\nwebrepl.start()\n")
+        copy_stream(old_f, new_f)
 
 def change_daemon(action):
     LINES = ("import webrepl", "webrepl.start()")

ports/esp8266/modutime.c

@@ -75,7 +75,7 @@ STATIC mp_obj_t time_localtime(size_t n_args, const mp_obj_t *args) {
     };
     return mp_obj_new_tuple(8, tuple);
 }
-MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(utime_localtime_obj, 0, 1, time_localtime);
+MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(time_localtime_obj, 0, 1, time_localtime);
 
 /// \function mktime()
 /// This is inverse function of localtime. It's argument is a full 8-tuple
@@ -95,7 +95,7 @@ STATIC mp_obj_t time_mktime(mp_obj_t tuple) {
             mp_obj_get_int(elem[1]), mp_obj_get_int(elem[2]), mp_obj_get_int(elem[3]),
             mp_obj_get_int(elem[4]), mp_obj_get_int(elem[5])));
 }
-MP_DEFINE_CONST_FUN_OBJ_1(utime_mktime_obj, time_mktime);
+MP_DEFINE_CONST_FUN_OBJ_1(time_mktime_obj, time_mktime);
 
 /// \function time()
 /// Returns the number of seconds, as an integer, since 1/1/2000.
@@ -103,13 +103,13 @@ STATIC mp_obj_t time_time(void) {
     // get date and time
     return mp_obj_new_int(pyb_rtc_get_us_since_2000() / 1000 / 1000);
 }
-MP_DEFINE_CONST_FUN_OBJ_0(utime_time_obj, time_time);
+MP_DEFINE_CONST_FUN_OBJ_0(time_time_obj, time_time);
 
 STATIC const mp_rom_map_elem_t time_module_globals_table[] = {
     { MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_utime) },
 
-    { MP_ROM_QSTR(MP_QSTR_localtime), MP_ROM_PTR(&utime_localtime_obj) },
+    { MP_ROM_QSTR(MP_QSTR_localtime), MP_ROM_PTR(&time_localtime_obj) },
-    { MP_ROM_QSTR(MP_QSTR_mktime), MP_ROM_PTR(&utime_mktime_obj) },
+    { MP_ROM_QSTR(MP_QSTR_mktime), MP_ROM_PTR(&time_mktime_obj) },
     { MP_ROM_QSTR(MP_QSTR_sleep), MP_ROM_PTR(&mp_utime_sleep_obj) },
     { MP_ROM_QSTR(MP_QSTR_sleep_ms), MP_ROM_PTR(&mp_utime_sleep_ms_obj) },
     { MP_ROM_QSTR(MP_QSTR_sleep_us), MP_ROM_PTR(&mp_utime_sleep_us_obj) },
@@ -118,7 +118,7 @@ STATIC const mp_rom_map_elem_t time_module_globals_table[] = {
     { MP_ROM_QSTR(MP_QSTR_ticks_cpu), MP_ROM_PTR(&mp_utime_ticks_cpu_obj) },
     { MP_ROM_QSTR(MP_QSTR_ticks_add), MP_ROM_PTR(&mp_utime_ticks_add_obj) },
     { MP_ROM_QSTR(MP_QSTR_ticks_diff), MP_ROM_PTR(&mp_utime_ticks_diff_obj) },
-    { MP_ROM_QSTR(MP_QSTR_time), MP_ROM_PTR(&utime_time_obj) },
+    { MP_ROM_QSTR(MP_QSTR_time), MP_ROM_PTR(&time_time_obj) },
 };
 
 STATIC MP_DEFINE_CONST_DICT(time_module_globals, time_module_globals_table);

ports/esp8266/mpconfigport.h

@@ -91,7 +91,7 @@
 #define MICROPY_PY_FRAMEBUF         (1)
 #define MICROPY_PY_MICROPYTHON_MEM_INFO (1)
 #define MICROPY_PY_OS_DUPTERM       (2)
-#define MICROPY_CPYTHON_COMPAT      (0)
+#define MICROPY_CPYTHON_COMPAT      (1)
 #define MICROPY_LONGINT_IMPL        (MICROPY_LONGINT_IMPL_MPZ)
 #define MICROPY_FLOAT_IMPL          (MICROPY_FLOAT_IMPL_FLOAT)
 #define MICROPY_FLOAT_HIGH_QUALITY_HASH (1)

ports/esp8266/uart.c

@@ -11,7 +11,7 @@
 *******************************************************************************/
 #include "ets_sys.h"
 #include "osapi.h"
-#include "espuart.h"
+#include "uart.h"
 #include "osapi.h"
 #include "uart_register.h"
 #include "etshal.h"

ports/nrf/mpconfigport.h

@@ -67,10 +67,6 @@
 #define MICROPY_VFS                              (1)
 #define MICROPY_VFS_FAT                          (MICROPY_VFS)
 
-// TODO these should be generic, not bound to fatfs
-#define mp_type_fileio fatfs_type_fileio
-#define mp_type_textio fatfs_type_textio
-
 // use vfs's functions for import stat and builtin open
 #if MICROPY_VFS
 #define mp_import_stat mp_vfs_import_stat

py/asmx86.h

@@ -112,7 +112,7 @@ void asm_x86_mov_r32_to_local(asm_x86_t* as, int src_r32, int dest_local_num);
 void asm_x86_mov_local_addr_to_r32(asm_x86_t* as, int local_num, int dest_r32);
 void asm_x86_call_ind(asm_x86_t* as, void* ptr, mp_uint_t n_args, int temp_r32);
 
-#ifdef GENERIC_ASM_API
+#if defined(GENERIC_ASM_API) && GENERIC_ASM_API
 
 // The following macros provide a (mostly) arch-independent API to
 // generate native code, and are used by the native emitter.

py/moduerrno.c

@@ -110,6 +110,7 @@ qstr mp_errno_to_str(mp_obj_t errno_val) {
             case EEXIST: return MP_QSTR_File_space_exists;
             case ENODEV: return MP_QSTR_Unsupported_space_operation;
             case EINVAL: return MP_QSTR_Invalid_space_argument;
+            case EROFS:  return MP_QSTR_Read_hyphen_only_space_filesystem;
         }
     }
 

shared-module/os/__init__.c

@@ -112,9 +112,8 @@ mp_obj_t common_hal_os_listdir(const char* path) {
     mp_obj_t dir_list = mp_obj_new_list(0, NULL);
     mp_obj_t next;
     while ((next = mp_iternext(iter_obj)) != MP_OBJ_STOP_ITERATION) {
-        mp_obj_t *items;
+        // next[0] is the filename.
-        mp_obj_get_array_fixed_n(next, 3, &items);
+        mp_obj_list_append(dir_list, mp_obj_subscr(next, MP_OBJ_NEW_SMALL_INT(0), MP_OBJ_SENTINEL));
-        mp_obj_list_append(dir_list, items[0]);
     }
     return dir_list;
 }