Date: 2009-10-20 22:02:00
python enhancement proposal

The other day while writing some code in Python, I had an idea. The bit of code I was writing was shaped something like this:

    for a in results:
        if a.something:

It occurred to me that if I were to write this as a list comprehension, then the "if" would combine with the "for" on the same line:

    [do_action(a) for a in results if a.something]

What if I could write this instead:

    for a in results if a.something:

This looks like a nice compact syntax for a reasonably common construct, which happens to also be nicely aligned with the list comprehension syntax. I thought I'd have a go at modifying the Python language to accept this, and perhaps even submit a PEP (Python Enhancement Proposal) for the core language.

I read about the PEP process, found a related PEP to enhance the generator syntax (from 2009), which led me to a long thread discussing that PEP, from which I found a discussion about an "if-syntax for regular for-loops" (from 2008), and finally Guido's opinion on the matter (from 2006), which states that he doesn't like it and that it was proposed and rejected before. And as it turns out, after all that I agree with Guido.

Lessons learned (none the hard way!):

[info]edm : for-if

It seems to me that what you want grammatically is:

for a in results where a.something:

since "if" doesn't really work for readability there (even if it does reduce the number of tokens that the tokenizer needs to know about).

I'm pretty sure I've used a language with a construct like that, but I can't remember for sure what it is -- possibly PICK BASIC, or one of its derivatives.

FWIW, the Perl equivalent would be:

foreach my $a in (grep { $_->something() } @results) {

which loses a certain something in readability, but is none the less quite handy at times. (Typically when I use a construct like that I'll do the grep outside the foreach construct, with something like:

my @filtered_results = grep { $_->something() } @results;

I keep meaning to look at python more closely as an alternative to perl. But each time I consider it, I find python too "low level" for the type of programming I'm doing for reasons like this. (Which is ironic as the first serious programming I did was Z80 assembler.)


[info]ghewgill : Re: for-if

I agree that "where" reads slightly better than "if", but Python already uses "if" for that purpose in the list generator context. Perhaps the language you're thinking of is SQL? :)

The Python equivalent of that Perl would be (using generator syntax):

for a in (x for x in a if x.something):

which is reasonably convenient but awkwardly introduces a new identifier x (which is limited in scope to the generator expression itself). I felt that this would read better if I could use a.something instead of x.something which motivated the above idea. Perl actually has the same issue by using $_ instead of $a in the test.

Python is an excellent language without all the extra unreadability introduced by Perl. What sort of programming are you doing where Python would be too low level but Perl isn't?

[info]edm : Re: for-if
The language I'm thinking of could be SQL, but I'm still convinced I've used another language with a "where" clause. Another possibility is Progress's 4GL. Both the Progress 4GL and PICK BASIC are database manipulation languages like SQL, but both are intended for application programming (which SQL mostly isn't -- but I've seen large applications written in Oracle PLSQL plus a template front end; debugging was... difficult).

The introduction of another identifier in the "list selector" (for want of a better term) seems pretty similar to extra variables used inside an anonymous function or closure. Perl consistently uses $_ for this "data being anonymously manipulated" case, which probably helps make it more idiomatic. I agree it'd be cleaner to do the SQL-like thing and have the "where" (or "if") directly reference the thing you're manipulating but it probably complicates language implementation a bit.

Most of my recent programming is data manipulation, especially database data manipulation and text file transformation. I've grown very used to using map {} and grep {}, loop constructions that can test their exit condition, and regular expressions. (I fear this means that I'm a secret LISP programmer.) The python equivalent idiomatic constructs seem less convenient. But that could just be lack of time with the language. (I didn't remember the python generator syntax until you pointed it out, for instance.)

[info]ghewgill : Re: for-if
Without knowing precisely what you're doing, I would suggest that your impressions about Python stem from a lack of familiarity. Python has direct equivalents for map and grep (map and filter), just as many loop constructions as Perl, and full regular expressions. Generators and list comprehensions are really powerful tools that mostly supersede the need for map and filter, while Perl actually lacks such a thing. For example, to find the square of every odd number in a list:

Perl: map {$_*$_} grep {$_ & 1} @a;
Python: [x*x for x in a if x & 1]

Perl's regular expression manipulation support is tightly integrated into the language while Python has the re module, so Python regular expression manipulation is a wee bit more verbose. While the regular expressions themselves are largely the same, the stuff around them is what I'm referring to, such as instead of just $1. I have not found this to be a difficulty, especially since you can have more than one Match object sitting around if you need to. Python prefers to be explicit while Perl does a lot of things implicitly.

After many years with Perl, I was reluctant to get into Python due to unfamiliarity with the language. But I started initially because Python had better SOAP support than Perl for some RPC development I was doing, and I grew to love the language. Learning the "Pythonic" idioms took some time, just as learning the same things in Perl takes time.
[info]edm : Re: for-if
It sounds like you're right about lack of familiarity being the major issue (combined with much of my experience with Python being the Python of 5-6 years ago). Thanks for the suggestions and comparisons. As you say, learning the idioms of the and getting accustomed to thinking about problems in that way is the biggest step.

Greg Hewgill <>