Commit Graph

35 Commits

Author SHA1 Message Date
Diego Elio Pettenò 34b4993d63 Add license to some obvious files. 2020-07-06 19:16:25 +01:00
Jeff Epler 473e9c5ffb f-strings: Make optional, defaulting to !CIRCUITPY_MINIMAL_BUILD
This should reclaim *most* code space added to handle f-strings.
However, there may be some small code growth as parse_string_literal
takes a new parameter (which will always be 0, so hopefully the optimizer
eliminates it)
2020-03-09 09:03:25 -05:00
Josh Klar 3a7a5ba686 py: Implement partial PEP-498 (f-string) support
This implements (most of) the PEP-498 spec for f-strings, with two
exceptions:

- raw f-strings (`fr` or `rf` prefixes) raise `NotImplementedError`
- one special corner case does not function as specified in the PEP
(more on that in a moment)

This is implemented in the core as a syntax translation, brute-forcing
all f-strings to run through `String.format`. For example, the statement
`x='world'; print(f'hello {x}')` gets translated *at a syntax level*
(injected into the lexer) to `x='world'; print('hello {}'.format(x))`.
While this may lead to weird column results in tracebacks, it seemed
like the fastest, most efficient, and *likely* most RAM-friendly option,
despite being implemented under the hood with a completely separate
`vstr_t`.

Since [string concatenation of adjacent literals is implemented in the
lexer](534b7c368d),
two side effects emerge:

- All strings with at least one f-string portion are concatenated into a
single literal which *must* be run through `String.format()` wholesale,
and:
- Concatenation of a raw string with interpolation characters with an
f-string will cause `IndexError`/`KeyError`, which is both different
from CPython *and* different from the corner case mentioned in the PEP
(which gave an example of the following:)

```python
x = 10
y = 'hi'
assert ('a' 'b' f'{x}' '{c}' f'str<{y:^4}>' 'd' 'e') == 'ab10{c}str< hi >de'
```

The above-linked commit detailed a pretty solid case for leaving string
concatenation in the lexer rather than putting it in the parser, and
undoing that decision would likely be disproportionately costly on
resources for the sake of a probably-low-impact corner case. An
alternative to become complaint with this corner case of the PEP would
be to revert to string concatenation in the parser *only when an
f-string is part of concatenation*, though I've done no investigation on
the difficulty or costs of doing this.

A decent set of tests is included. I've manually tested this on the
`unix` port on Linux and on a Feather M4 Express (`atmel-samd`) and
things seem sane.
2020-03-09 08:16:07 -05:00
Alexander Steffen 55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Alexander Steffen 299bc62586 all: Unify header guard usage.
The code conventions suggest using header guards, but do not define how
those should look like and instead point to existing files. However, not
all existing files follow the same scheme, sometimes omitting header guards
altogether, sometimes using non-standard names, making it easy to
accidentally pick a "wrong" example.

This commit ensures that all header files of the MicroPython project (that
were not simply copied from somewhere else) follow the same pattern, that
was already present in the majority of files, especially in the py folder.

The rules are as follows.

Naming convention:
* start with the words MICROPY_INCLUDED
* contain the full path to the file
* replace special characters with _

In addition, there are no empty lines before #ifndef, between #ifndef and
one empty line before #endif. #endif is followed by a comment containing
the name of the guard macro.

py/grammar.h cannot use header guards by design, since it has to be
included multiple times in a single C file. Several other files also do not
need header guards as they are only used internally and guaranteed to be
included only once:
* MICROPY_MPHALPORT_H
* mpconfigboard.h
* mpconfigport.h
* mpthreadport.h
* pin_defs_*.h
* qstrdefs*.h
2017-07-18 11:57:39 +10:00
Damien George 5124a94067 py/lexer: Convert mp_uint_t to size_t where appropriate. 2017-02-17 12:44:24 +11:00
Damien George 773278ec30 py/lexer: Simplify handling of line-continuation error.
Previous to this patch there was an explicit check for errors with line
continuation (where backslash was not immediately followed by a newline).

But this check is not necessary: if there is an error then the remaining
logic of the tokeniser will reject the backslash and correctly produce a
syntax error.
2017-02-17 11:30:14 +11:00
Damien George ae43679792 py/lexer: Use strcmp to make keyword searching more efficient.
Since the table of keywords is sorted, we can use strcmp to do the search
and stop part way through the search if the comparison is less-than.

Because all tokens that are names are subject to this search, this
optimisation will improve the overall speed of the lexer when processing
a script.

The change also decreases code size by a little bit because we now use
strcmp instead of the custom str_strn_equal function.
2017-02-17 11:10:35 +11:00
Damien George c305ae3243 py/lexer: Permanently disable the mp_lexer_show_token function.
The lexer is very mature and this debug function is no longer used.  If
it's really needed one can uncomment it and recompile.
2016-12-22 10:49:54 +11:00
Damien George 5bdf1650de py/lexer: Make lexer use an mp_reader as its source. 2016-11-16 18:35:01 +11:00
pohmelie 81ebba7e02 py: add async/await/async for/async with syntax
They are sugar for marking function as generator, "yield from"
and pep492 python "semantically equivalents" respectively.

@dpgeorge was the original author of this patch, but @pohmelie made
changes to implement `async for` and `async with`.
2016-04-13 15:26:38 +01:00
Damien George 031278f661 unix: Allow to cat a script into stdin from the command line.
See issue #1306.
2015-06-04 23:42:45 +01:00
Damien George 2e2e404ff7 py: Allow to compile with extra warnings (sign-compare, unused-param). 2015-03-19 00:25:33 +00:00
Damien George 7d414a1b52 py: Parse big-int/float/imag constants directly in parser.
Previous to this patch, a big-int, float or imag constant was interned
(made into a qstr) and then parsed at runtime to create an object each
time it was needed.  This is wasteful in RAM and not efficient.  Now,
these constants are parsed straight away in the parser and turned into
objects.  This allows constants with large numbers of digits (so
addresses issue #1103) and takes us a step closer to #722.
2015-02-08 01:57:40 +00:00
Damien George b4b10fd350 py: Put all global state together in state structures.
This patch consolidates all global variables in py/ core into one place,
in a global structure.  Root pointers are all located together to make
GC tracing easier and more efficient.
2015-01-07 20:33:00 +00:00
Damien George 51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Paul Sokolovsky 8ab6f90674 py: Move to guarded includes for compile.h and related headers. 2014-12-27 16:12:17 +02:00
Damien George a4c52c5a3d py: Optimise lexer by exposing lexer type.
mp_lexer_t type is exposed, mp_token_t type is removed, and simple lexer
functions (like checking current token kind) are now inlined.

This saves 784 bytes ROM on 32-bit unix, 348 bytes on stmhal, and 460
bytes on bare-arm.  It also saves a tiny bit of RAM since mp_lexer_t
is a bit smaller.  Also will run a bit more efficiently.
2014-12-05 19:35:18 +00:00
Damien George 94fbe9711a py: Change lexer stream API to return bytes not chars.
Lexer is now 8-bit clean inside strings.
2014-07-30 11:46:05 +01:00
Damien George 54eb4e723e lexer: Convert type (u)int to mp_(u)int_t. 2014-07-03 13:47:47 +01:00
Paul Sokolovsky d3439d0c60 py: Instead of having "debug on" var, have "optimization level" var.
This allows to have multiple "optimization" levels (CPython has two
(-OO removes docstrings), we can have more).
2014-06-03 12:32:59 +03:00
Damien George 04b9147e15 Add license header to (almost) all files.
Blanket wide to all .c and .h files.  Some files originating from ST are
difficult to deal with (license wise) so it was left out of those.

Also merged modpyb.h, modos.h, modstm.h and modtime.h in stmhal/.
2014-05-03 23:27:38 +01:00
Damien George e09ffa1400 Search paths properly on import and execute __init__.py if it exists. 2014-02-05 23:57:48 +00:00
Damien George b829b5caec Implement mp_parse_node_free; print properly repr(string). 2014-01-25 13:51:19 +00:00
Damien George 08335004cf Add source file name and line number to error messages.
Byte code has a map from byte-code offset to source-code line number,
used to give better error messages.
2014-01-18 23:24:36 +00:00
Damien George 9528cd66d7 Convert parse errors to exceptions.
Parser no longer prints an error, but instead returns an exception ID
and message.
2014-01-15 21:23:31 +00:00
Damien George 69a818d418 py: Improve memory management for parser; add lexer error for bad line cont. 2014-01-12 13:55:24 +00:00
Damien George 9193f89296 Move lexerstr to main py directory (everyone uses it). 2014-01-08 15:28:26 +00:00
Damien George e9906ac3d7 Add ellipsis object. 2014-01-04 18:44:46 +00:00
Damien George 66028ab6dc Basic implementation of import.
import works for simple cases.  Still work to do on finding the right
script, and setting globals/locals correctly when running an imported
function.
2014-01-03 14:03:48 +00:00
Damien d99b05282d Change object representation from 1 big union to individual structs.
A big change.  Micro Python objects are allocated as individual structs
with the first element being a pointer to the type information (which
is itself an object).  This scheme follows CPython.  Much more flexible,
not necessarily slower, uses same heap memory, and can allocate objects
statically.

Also change name prefix, from py_ to mp_ (mp for Micro Python).
2013-12-21 18:17:45 +00:00
Damien fa2162bc77 Integrate new lexer stream with stm framework. 2013-10-20 17:42:00 +01:00
Damien a5185f4bc8 Abstract out back-end stream functionality from lexer. 2013-10-20 14:41:27 +01:00
Damien 91d387de7d Improve indent/dedent error checking and reporting. 2013-10-09 15:09:52 +01:00
Damien 429d71943d Initial commit. 2013-10-04 19:53:11 +01:00