Mumble mumble something about conflating languages with implementations.

It will only get more difficult to maintain your app.

[1] https://python3statement.org/ - note many libraries weren't even waiting until 2020. It is a lot of work to maintain code with python 2 cruft. Not all packages are listed there, for example Django is Python 3 only, starting from 2.0 (currently at 3.0)

Unless some python 3 fanatic goes out of his way to write a python 2 library deleting virus the existing code wont disappear. Also some of these pledges only limit feature releases, afaik numpy planned to still provide a long term support version with bugfixes for python 2. It also helps that python already comes with a lot of build in bells and whistles so third party libraries aren't always necessary either.

You might be lucky and someone else might do that for you, but it will be harder and harder with time. Already according to JetBrains survey in 2019 (I believe) about 80% of people surveyed they already use python 3.

As for numpy I just checked[1] and the only wheels they are providing for the latest version are 3.5+ the package also says that it is 3 only.

[1] https://pypi.org/project/numpy/#files

> And, well, if you’re already making a backward-incompatible version, here’s this checklist of other breaking changes you might as well bring along for the ride.

Sorry, that doesn’t track. Treating quoted strings as UTF-8 by default instead of ASCII-or-arbitrary-bytes would have been a small migration that would not have taken over a decade to complete.

    # declare that the code itself is written in utf8
    use utf8;

    my $ā = 'ā';

    use feature 'unicode_strings';

    use v5.12;

Really mostly what you have to do is declare the encodings of the file handles.

    # change it for all files
    use open ':encoding(UTF-8)';

    open my $fh, '<', 'example.txt';
    binmode $fh, ':utf8';

---

Basically Perl just defaults to the old original ways. If you need a new feature which would break old code, you just declare that you need it.

Because of that, most code that was written for an earlier form of Perl still works on the latest version.

When python was written in 1989 Unicode didn't exist yet.

As for your second argument, many people bring out Go, that had such amazing idea of using everything as UTF-8 and it works great. They don't realize that Go is pretty much doing the same thing that Python does (ignoring how the string is represented internally, since that shouldn't really be programmer's concern).

Go clearly distinguishes between string (string type) and bytes ([]byte type) to use string as bytes you have to cast it to []byte and to convert bytes to string you need to cast them to string.

That's the equivalent of doing variable.encode() to get bytes and you do variable.decode() to get a string.

What python 3 inroduced is two types str and bytes, and blocked any implicit casting between them. That's exactly same thing Go does.

The only difference is implementation detail, Go stores strings as utf-8 and casting doesn't require any work, they are just for having compiler catch errors it also ignores environment variables and always uses utf-8. Python has an internal[1] representation and does do conversion. It respects LANG and other variables and uses that for stdin/out/err. Initially when those variables were undefined it assumed us-ascii which created some issues, but I believe now that was fixed and utf-8 is the default.

[1] Python 3 actually tries to be smart and uses UCS1 (Latin 1), UCS2 or UCS4 depending what characters are contained. If an UTTF-8 conversion was requested it will also cache that representation (as a C-string) so it won't do the conversion next time.

That was me in a parallel thread. Java and JavaScript internally use UTF-16 encoding. I also mentioned C, which treats strings as byte arrays, and C++, which supports C strings as well as introducing a string class that is still just byte arrays.

> As for your second argument, many people bring out Go, that had such amazing idea of using everything as UTF-8 and it works great.

Has Go ever broken backwards compatibility? Let me clarify my second argument: if you are going to break backwards compatibility, you should do so in a minimal way that eases the pain of migration. The Python maintainers decided that breaking backwards compatibility meant throwing in the kitchen sink, succumbing to second system effect, and essentially forking the language for over a decade. The migration from Ruby 1.8 to 1.9 was less painful, though in fairness I suppose the migration from Perl 5 to Perl 6 was even more painful.

That is because you can easily use Perl5 modules in Raku.

    use v6;

    use Scalar::Util:from<Perl5> <looks_like_number>;

    say ?looks_like_number( '5.0' ); # True

Then you can migrate one module at a time.

You can even subclass Perl classes using this technology.

Basically you can use the old codebase to fill in the parts of the new codebase that you haven't transferred over yet.

---

By that same token you can transition from Python to Raku in much the same way. The module that handles that for Python isn't as featurefull as the one for Perl yet.

    use v6;

    {
        # load the interface module
        use Inline::Python;

        use base64:from<Python>;

        my $b64 = base64::b64encode('ABCD');

        say $b64;
        # Buf:0x<51 55 4A 44 52 41 3D 3D>

        say $b64.decode;
        # QUJDRA==
    }

    {
        # Raku wrapper around a native library
        use Base64::Native;

        my $b64 = base64-encode('ABCD');

        say $b64;
        # Buf[uint8]:0x<51 55 4A 44 52 41 3D 3D>

        say $b64.decode;
        # QUJDRA==
    }

    { 
        use MIME::Base64:from<Perl5>;

        my $b64 = encode_base64('ABCD');

        say $b64;
        # QUJDRA==
    }

    {
        use Inline::Ruby;
        use base64:from<Ruby>;

        # workaround for apparent missing feature in Inline::Ruby
        my \Base64 = EVAL ｢Base64｣, :lang<Ruby>;

        my $b64 = Base64.encode64('ABCD');

        say $b64;
        # «QUJDRA==
        # »:rb

        say ~$b64;
        # QUJDRA==
    }

So if I had to I could transition from any of those other languages above to Raku at my leisure.

Not like Python2 to Python3 where it has to mostly be all or nothing.

C and C++ doesn't really have Unicode support, and most C and C++ applications don't support unicode. There are libraries that you need to use to get this kind of support.

> Has Go ever broken backwards compatibility? Let me clarify my second argument: if you are going to break backwards compatibility, you should do so in a minimal way that eases the pain of migration. The Python maintainers decided that breaking backwards compatibility meant throwing in the kitchen sink, succumbing to second system effect, and essentially forking the language for over a decade. The migration from Ruby 1.8 to 1.9 was less painful, though in fairness I suppose the migration from Perl 5 to Perl 6 was even more painful.

Go is only 10 years old Python is 31. And in fact it had some breaking changes for example in 1.4, 1.12. Those are easy to fix since they would show up during compilation. Python is a dynamic language and unless you use something like mypy you don't have that luxury.

Going back to python, what was broken in Python 2 is that str type could represent both text and bytes, and the difficulty was that most Python 2 applications are broken (yes they worked fine with ascii text but broke in interesting ways whenever unicode was used. You might say, so what, why should I care if I don't use Unicode. The problem was that mixing these two types and implicit casting that python 2 did made it extremely hard to write correct code even when you know what you're doing. With python 3 is no effort.

There is a good write up by one of Python developers why python 3 was necessary[1].

[1] https://snarky.ca/why-python-3-exists/

You know, it’s astounding to me that you managed to quote my entire point and still didn’t even bother to acknowledge it, let alone respond to it.

If they had to break backwards compatibility to fix string encoding, that’s fine and I get it. That doesn’t explain or justify breaking backwards compatibility in a dozen additional ways that have nothing to do with string encoding.

Are you going to address that point or just go on another irrelevant tangent?

That being sad, you can integrate Perl code in Raku (using the Inline::Perl5 module), and vice-versa.

Now with Python 3 it's the other way around -- Windows is fine but Linux has issues. However, the problems for linux are less severe: often you can get away with assuming that everything is UTF-8. And you can still work with bytes if you absolutely need to.

No, they're not. Windows can't magically send your program Unicode. It sends your program strings of bytes, which your program interprets as Unicode with the UTF-16 encoding. The actual raw data your program is being sent by Windows is still strings of bytes.

> you can still work with bytes if you absolutely need to

In your own code, yes, you can, but you can't tell the Standard Library to treat sys.std{in|out|err} as bytes, or fix their encodings (at least, not until Python 3.7, when you can do the latter), when it incorrectly detects the encoding of whatever Unicode the system is sending/receiving to/from them.

> AFAIK neither sys.argv nor sys.environ had a unicode-supporting alternative in Python 2)

That's because none was needed. You got strings of bytes and you could decode them to whatever you wanted, if you knew the encoding and wanted to work with them as Unicode. That's exactly what a language/library should do when it can't rely on a particular encoding or on detecting the encoding--work with the lowest common denominator, which is strings of bytes.

Actually you can, you should use sys.std{in,out,err}.buffer, which will be binary[1]

> or fix their encodings (at least, not until Python 3.7, when you can do the latter), when it incorrectly detects the encoding of whatever Unicode the system is sending/receiving to/from them.

I'm assuming you're talking about scenario where LANG/LC_* was not defined, then Python assumed us-ascii encoding. I think in 3.7 they changed default to UTF-8.

[1] https://docs.python.org/3/library/sys.html#sys.stdin

That's fine for your own code, as I said. It doesn't help at all for code in standard library modules that uses the standard streams, which is what I was referring to.

> I think in 3.7 they changed default to UTF-8

Yes, they did, which is certainly a saner default in today's world than ASCII, but it still doesn't cover all use cases. It would have been better to not have a default at all and make application programs explicitly do encoding/decoding wherever it made the most sense for the application.

I'm not aware what code you're talking about. All functions I can think of expect to provide streams explicitly.

> Yes, they did, which is certainly a saner default in today's world than ASCII, but it still doesn't cover all use cases. It would have been better to not have a default at all and make application programs explicitly do encoding/decoding wherever it made the most sense for the application.

I disagree, it would be far more confusing when stdin/stdout/stderr were sometimes text sometimes binary. If you meant that they should always be binary that's also unoptimal. In most use cases an user works with text.

All the places in the standard library that explicitly write output or error messages to sys.stdout or sys.stderr. (There are far fewer places that explicitly take input from sys.stdin, so there's that, I suppose.)

> it would be far more confusing when stdin/stdout/stderr were sometimes text sometimes binary

I am not suggesting that. They should always be binary, i.e., streams of bytes. That's the lowest common denominator for all uses cases, so that's what a language runtime and a library should be doing.

> If you meant that they should always be binary that's also unoptimal. In most use cases an user works with text.

Users who work with text can easily wrap binary streams in a TextIOWrapper (or an appropriate alternative) if the basic streams are always binary.

Users who work with binary but can't control library code that insists on treating things as text are SOL if the basic streams are text, with buffer attributes that let user code use the binary version but only in code the user explicitly controls.

(IIRC it used to do that in 3.0, but they backtracked very quickly - and 3.0 was effectively treated as a beta by the community at large, anyway.)

So your claim is that "Python 2" is a language spec, not an implementation? And that there will be future releases of this language spec in the future? I doubt it.

I agree it's likely that there will be people wasting their time maintaining an interpreter fork, but that will not be Python-the-language (a trademarked term BTW), it will be a fork of the implementation.

I suppose one could argue that, the CPython implementation is the language specification. (And I seem to recall hearing that notion somewhere years ago.) In which case, it would not be possible to freeze development of the language without freezing the implementation as well. There are various reasons I wholeheartedly disagree with such a characterization, but I guess there's some self-consistency there at least.

Development of CPython 2 has ended, bugs and all. It's past its end of life, this is well known and has been known for a long time. Any remaining bugs are the problem of the users, not the responsibility of the former developers.

Sure people will fork it and do stuff with those forks, but those will no longer be new versions of CPython, they will be new versions of some-fork-of-CPython.

PyPy maintains a Python 2 implementation and will continue to do so.

Cython, IronPython, Brython, Stackless Python, Nuitka, etc.

Audit all strings coming in and going out for encoding issues. Update all dependencies to their python 3 equivalent. Replace dependencies that hadn’t been updated (typically older django dependencies). Use python-future to bulk update incompatibilities. Changes to metaclasses were annoying. Force all uses of pickle to use protocol version 2. I documented some more during the migration on Twitter https://twitter.com/jarshwah/status/1209381850822496256?s=21

We began getting the code base into a compatible position about 1.5 years earlier. A final push of 3-4 weeks of work got it over the line, with many bug fixes after the deployment.

Other older larger systems will have similar problems at a larger scale.

This isn’t a condemnation by the way. Python 3 is better. The only reason we held out so long was because of the business justification. Once we couldn’t wait any longer it got prioritised.

"It's a dirty job, but someone's got to do it".

One industry example is https://vfxplatform.com/ - they just (this year) moved to Python3, but with some delays, from the site:

    The move to Python 3 was delayed from CY2019 to CY2020 due to:

    No supported combination of Qt 5.6, Python 3 and PySide 2 so Qt first needed to be upgraded.
    Upgrade of both Qt and Python in the same year was too large a commitment for software vendors and large studios.
    Python 3 in CY2020 is a firm commitment, it will be a required upgrade as Python 2 will no longer be supported beyond 2020. Software vendors are strongly encouraged to provide a tech preview release in 2019 to help studios with testing during their Python migration efforts.

I heard that some places still using Turbo Pascal for DOS and have to stick with 32 bit machines because 64 bit can't run 16 bit DOS code.

Too many software development projects are treated as one-off events where people commission them and assume they will work forever without updates. Software requires maintenance, and people who commission software development projects without planning on how they are going to be maintained in the future are taking on risk. Any risk involved in updating that abandoned code in future is a consequence of that decision.

But if you actively changing the code, the maintenance will get more and more expensive. With packages dropping python 2 support if you discover a bug in one of your dependencies and fix is in package that no longer work on python 2 you'll need to backport the fix (and maintain your fork) or migrate your code.

The string-handling changes, while necessary, are also a bear to deal with. Since python is dynamically typed, you need to work to find all the places where you need to add a ".decode()" or ".encode()". If you don't have excellent test coverage already, you're going to miss some, and it'll be a game of whack-a-mole until you get them all... assuming you have actually gotten them all.

Dicts were by definition unordered until Python 3.7 [0], so you were relying on undefined behaviour. If you need an ordered dictionary and support Python 3.6 or below, you should use OrderedDict [1].

[0] https://mail.python.org/pipermail/python-dev/2017-December/1...

[1] https://docs.python.org/3/library/collections.html#collectio...

Enumeration order in dict keys was never guaranteed (until 2019), even on 2. So basically that code relied on undocumented cpython behaviour that was strongly advised against, i.e. it was broken already. 3 simply made the brokenness more visible.

Although there are ways, you can still incrementally adapt code base to work on both pythons. Also pylint with py3k option, mypy can help. There's also six packages, but many people seem to had good luck with futurize.

There's also something that I tried a while ago and it surprisingly worked (although it might not work that well on larger codebase?), basically you can use Cython (not to be confused with CPython) to compile Python 2 code and then include it in python 3, this would enable migration file by file.

I'm not looking forward to the scramble when it does upgrade, as we're using a ton of community modules that may or may not be abandoned.

effort += 1 + 1

Ported to Python3 for you.

  effort = (effort := effort + 1) + 1

You're not wrong, but as a practical matter, that pretty much describes our entire industry.

You're right though, we're fighting our way out of technical debt and switching to python 3 was absolutely necessary. It's forced us to sort out a lot of sketchy string / bytes handling. We do, mercifully, have ~90% test coverage.

One issue with the test suite was that it made heavy use of a thing called django_any that isn't supported in python 3, so decided to replace it with Factory Boy. We have about 500 django models that needed new factories. Factory Boy works quite differently and it was a lot of work to make the factories behave similarly to the old ones where possible, and update most of our ~4000 tests for the new behaviour.

So that was one issue. It was tempting to just patch django_any, but we decided to tackle the technical debt instead.

Most codebases basically.

Canonical will not provide long term support for Python 2 as part of Ubuntu 20.04 LTS. In Ubuntu 20.04, Python 2 is a "universe" package [2] that does not receive updates by Canonical. This means that the you will only get Python 2 security update guarantees with Ubuntu is on 18.04 LTS until April 2023.

Debian is making an active effort [3] to remove Python 2 and packages that depend on it for its next release. It'll likely support Python 2 as part of Debian Buster until 2024.

Note that if you're reading this to delay your move to Python 3 by another few years, you're doing it wrong. This list shows even all slow enterprise-y distros have a deadline for Python 2, not that you can stretch your stuff for a couple of more years :)

[1]: https://access.redhat.com/solutions/4455511

[2]: https://packages.ubuntu.com/focal/python2

[3]: https://wiki.debian.org/Python/2Removal

Otherwise even if your python has security patches for next 4 years, it won't do you any good when you find a bug in one of your dependencies and bugfix is in a version that's python 3 only

> During DebConf19 we¹ have tried to figure out how to manage Python 2 and PyPy module removal from Debian and below is our proposal. [0]

Debian are in the midst of a large project [1] to remove Python 2 as quickly as they possibly can. Whilst some bugfixes may happen, Debian are already telling you in no uncertain terms:

> port upstream package to python3

> remove any Python 2 use

[0] https://lists.debian.org/debian-python/2019/07/msg00080.html

[1] https://wiki.debian.org/Python/2Removal

If redhat decided to add new features to python 2.7, I'm sure the PSF would make a stink

You probably confusing it with Tauton (a Python 2.7 with backported Python 3 features) that tried to place itself as Python 2.8. By backporting these changes they created essentially 3rd version of Python that was incompatible with other 2.

Maybe they could call the new 2.7 interpreter IceSnake.

> As such, stating accurately that software ... is compatible with the Python programming language, or that it contains the Python programming language, is always allowed.

Looks like Pillow has wheel for 3.8 now since April 2nd, so might work now (no compilation is needed). I don't know other packages so can't check them. Psycopg2 would probably be another one with this issue (also fixed on April 6th).

[1] https://pypi.org/project/Pillow/#files look for cp38 wheel files.

The far bigger worry would be unexpected encoding issues corrupting the data when going from 2->3.

Knowing researchers (at least in academia), they're the last people I would expect to change.