I think the only way MS knows how to make money is lock someone into an ecosystem. I think their next attempt should be a closed source lean checker that has will have proprietary code for NL -> lean formalization checker.
I don't think that's true. Many big companies contribute to FOSS in ways that aren't just to attempt to lock people in. Often it's because they use the software and don't want to maintain patches forever - everyone wins if things actually get fixed. Or even because they just want to be known as a good place to work.
That describes how hands off MS was with Postgres committers ime a few years ago. Useful to house them to for helpful office hours too. We'll see how that holds up post HorizonDB
They definitely have other intentions, but it's often the same as people contributing in their free time: they use the software, want to make it better for their purposes, but don't want to be the ones to maintain it.
Altruism happens usually among members of a tightly knit group, like family members or soldiers of a given unit. Teachers sacrificinng their life to save children. FOSS-developers, are they really producing software out of altruism? Maybe some of them are but it is hard for us to know whether that is so or not.
If a FAANG releases Kafka, REACT, Kubernetes, or any number of projects that they found helpful, and released into open source - there will always be a claim that they do it with some second intention, such as "be seen to be a positive contributor to the community"
But the truth is - so do I - as a humble contributor to a document, test, bug fix, or feature add to a FOSS project - where's the beef?
If only, those FAANGs with hyperscaler and AI business always have a way to plunge those improvements into customer acquisitions for their cloud products, not available on the other vendors.
Yes, Microsoft fired the Faster CPython team and Facebook/Instagram tries to buy its way into Python core in order to get their buggy free threading in.
This should be called Python Type Server Protocol, no? Seems weird to mine the generic protocol namespace for a protocol that only supports Python. Especially coming from the company behind LSP, which actually is generic.
This is the company that named its web framework "MVC". At least this time they're not eternally polluting search results for an industry standard term.
As someone with minimal knowledge of LSP specifics, I'm a bit confused about why this exists. Is it the case that LSP alone does not provide adequate functionality for Python type queries?
> The "type server protocol" is a desperate attempt from microsoft to prevent pylance from becoming obsolete by outsourcing its type checking to other language servers like basedpyright, ty or pyrefly -- while continuing to keep certain language server features exclusive to pylance.
> This protocol benefits nobody except microsoft. It adds needless complexity to any language server that chooses to support it. Basedpyright does not support it and strongly advises other language servers to avoid supporting it as well.
It seems to be a more structured representation of the type. Normally in LSP you'd have a plain string or markdown that you render in the IDE, but nothing that you can analyse structurally.
It's a shame it's not just an LSP extension though, or a feature in a newer LSP version. LSP is maintained by Microsoft engineers.
I thought workspace/symbol returns an unstructured payload wrapped w/ some metadata.
Since the format is unspecified I imagine many language servers would just return a string representation of the type, which in Python is a PITA to parse because not every type has a closed-form representation (overloaded callable types, for example, have to be declared as a protocol).
It’s really good that folks over at Astral decided to implement it. It was difficult for me to understand what does this protocol provide that couldn’t be implemented with a new LSP capability, the linked document doesn’t really explain that, but what did become clear to me is that seemingly Microsoft are trying to invent new features for their proprietary LSP server that only works in VS Code so people will have harder times switching to other editors. This way, no matter what Microsoft comes up with, will be matched with very high quality open source tools.
So basically adding machine readable type-at-point queries as an lsp extension so Pylance and Pywright can be separated?
The 'result types'format is super python specific, like not having a for all binder. I don't think this specific schema will be used for any non-python languages.
Language server protocol is the interface of a software piece that sits between an IDE and a language toolchain so you don't need to change code in the IDE to add support of new languages (you add support on the LSP server). Then server provides things such as completions/intelli-sense, navigation to definitions and some correction checking. From the anouncement TSP extends LSP for better python type awarness.
I just use the Pyrefly extension in VSCode/Zed, run Pyrefly in the CI, and everything works well. The fragmentation around pyright/pylance/basedpyright/cursor-pyright etc. makes it pretty much unusable, you can't get consistent results across IDEs.
In LSP, UTF-16 is not just for transport. Source locations are also referred to by their UTF-16 offset. That means applications are required to store a persistent mapping from UTF-16 offsets to whatever they use internally, keep that mapping updated as the document is edited, and map back and forth constantly.
Oh well. It was primarily designed to be a good protocol for VSCode, not for any other users. That's just the price you pay when using a corporate standard.
Oh you're right, except since LSP 3.17 position encoding can be negotiated with clients/servers that support it. I initially got confused because my work with LSP started with 3.17 and I always used UTF-8 encoding.
When I wrote the initial version of the neovim LSP client, I got intimately familiar with LSP, and how much I hate it. Microsoft protocols have a habit of being extremely specialized to their application despite being recommended for broader use. I will never, ever, ever adapt another microsoft language protocol.
It's such a shame python got such mindshare. It's a disaster of a language and ecosystem that people keep bolting and hacking things that much better thought out languages have had for decades.
At this point, Python is less of a programming language by itself but more of an interop protocol between various C/C++/Fortran/Rust binaries. But again, it's just part of the cycle, Python was a breath of fresh air 10 years ago when academia is still dominated by Matlab, but now it is mostly a mess of a language that's good for throwaway scripts but not very convenient for large software projects compared to statically typed languages.
still trying to figure out a way to explain it, but i thought i needed a programming language, and it turns out what i need is an interop protocol. i want to be able to "translate" English text to a more or less dense rendering (or a completely different tone/pedagogical approach) without adding or losing meaning.
I remember falling in love with Python when I first learned it at version 1.5, liked it even more with 2.0, but when 3.0 came around, that's when I noticed the ecosystem falling apart. Not for the obvious incompatibility schism: I do think most of 3.0's changes were good ones, and it finally is the standard default version now, but in many other ways it feels like the ecosystem has still never really recovered. Virtualenv is still the same grotesque hack, where rather than doing what other languages like Perl and Ruby do and making some wrapper around setting PYTHONPATH, it relies on python's behavior of locating the library path relative to the install, thus hardlinking a whole python installation in and making you "activate" it first. That nothing has managed to replace it in 20 years speaks volumes.
I'm hoping one day people look at Python the same way they look at Perl and PHP.
There has been so much programming language innovation in the past decade that I'm surprised someone hasn't built a better Python. I'd like to see something like Rhai or Roto take off which has a natural upgrade path to Rust.
Maybe it's because interpreted languages just aren't in vogue anymore and probably won't be for a long time if humans aren't even coding anymore.
Roughly the time when Guido stopped fighting the hordes. The hordes won, wrested the language away from its creator and now they own the pieces. 2 to 3 was the worst possible way a language has ever been upgraded bar none. Hilariously, even today Python 2 persists in many places because of this.
> Guido wrested himself by accepting the stupid := operator against the huge opposition.
Yes, that's the attitude I was talking about.
See, when you are a maintainer of a popular language with a community at some point the bickering becomes the job. And when you are a creative person managing a horde of egos who all participate to put their stamp on something the only reasonable thing to do is to leave them to their own devices, which he did. I really don't blame him. Guido is one of the nicest people and to see how the Python community did not have a single second's consideration for how their incessant infighting must have made him feel is one of the worst stories of the FOSS movement.
I think the idea is that you start out believing people will cooperate, and that the best idea will naturally surface, and then you find out humans don’t actually work like that.
And a couple of psychopath core devs gleefully celebrated hours after his resignation, marginalizing the event and tried to grab power directly. The two snakes are an appropriate symbol for Python.
It has found its way in many Linux distros as a replacement for bash scripts when the implementation becomes just too convoluted. It's a boring syntax (though with few semantic quirks), intuitive to understand and hack, so the exact opposite to shell scripts. It get replaced by an implementation in C or Rust when the perfs start to falter.
Another possible reason is that it's taught in college as an introductory language, and also that it became popular in AI circles, and thus models were mostly trained on it, perpetuating its usage. They got much better with other languages though, so maybe the situation is changing (improving maybe?).
It has its merits: the syntax forces students to format their code correctly, it is very close to pseudo-code (is that a good thing though, in the end?).
Pretty much every other mainstream language bar JS. I'd say also Ruby because I also do not like a lot of its choices, but there's at least a kind of heady elegance to it in a REPL, though I would never want to work on a production system written in Ruby.
Frankly, I just really prefer statically and strongly typed compiled languages.
I did my last company's first Demo and MVP in Python after running through the Koans at my co-founder's behest and ultimately we decided that the production system would not be done in Python. There are some cool things about it - specifically I do like that annotations are real AOP seams, but the numerous footguns and lack of types and a compiler just make it a very annoying ecosystem to work in for me.
Those are closer languages but my broader point is that these application-level languages are more or less interchangeable from a professional standpoint. A Java or C# pro should probably not spend time becoming a Python pro because they could have spent that time producing equivalent value in the languages they are already experts in.
I don't agree at all that it's a coke/pepsi situation. That analogy is for when two mostly-interchangeable choices develop tribal followings based largely on personal preference. Coke and Pepsi are both caramel carbonated soft drinks that are terrible for your health differing only slightly in flavour. Otherwise they're materially identical.
C# is strongly/statically typed, compiled, and quite fast (after the vm has started), whereas python is weakly/dynamically typed, interpreted, and generally slow as fuck. Those are two fundamentally different languages, how could it possibly be a coke/pepsi situation?
I think python is fantastic for certain use cases: gluing other tools together (like LAPACK or other ancient libraries), system automation, prototyping, exploitation (I think pretty much all of us were working on stealmybitcoin in python), and exploratory data analysis.
The problems arise when people try to write, say, gigantic GUI applications in python, or really anything with CPU-bound workloads. Part of being an engineer is using the right tool for the right job. Python is the wrong tool for lots of jobs. That shouldn't be a controversial statement.
It's strongly/dynamically typed, not weakly typed. Why do people keep getting this wrong? Has any version of Python been weakly typed? I know Python 2 and 3 aren't, perhaps the first version or ABC, its predecessor language.
It's kinda funny to say people are getting it "wrong" when strong/weak typing isn't really defined anywhere or universally agreed upon.
In fact, even "dynamic typing" can be considered to be an oxymoron, since types are formally defined to be a static property of a language. A more correct term would be "uni-typed".
I'm not saying that the meanings you're using for static/dynamic and weak/strong are bad, just that chastising someone for getting them "wrong l" doesn't make sense for terms that are not widely agreed upon.
You're trying too hard to play the role of the "aktually" guy. Assuming good intentions and taking a pass at replying would have been a better contribution to the discussion.
Numpy is over 2 decades old. Pytorch is a decade old. Python devs know what "right tool for the job" means and focus on solving problems instead of being language purists.
This has nothing to do with purism. I use python all the time, where it makes sense.
What I don't do is cultivate a "favorite language" and make it my whole identity, and then take criticism of that language as some sort of personal affront. Python objectively has some serious flaws. That shouldn't make anyone emotional, and I don't know why people get their backs up like this.
Who is cultivating a "favorite language"? I missed that part. Agreed that this should not be an emotional issue, that is unprofessional.
Problem with any popular language is an outsider sees one or two statistically-likely codebases that are total turds, and then gets one-shot by the Goomba fallacy.
And the thing that maddens me the most is that it has a critical mass where it’s basically the default starting language and most people never really move from there. And so we’re stuck with this grotesque abomination that somehow keeps on rolling. The amount of money that was poured into python to make it barely bearable is probably the GDP of a small nation.
20 years working on Python. Two things have driven its success.
First, it's easy to learn and reasonably powerful. You can make a lot of useful things really quickly, which was true even 15 years ago. It's only gotten easier since, and the ecosystem has continued to grow.
Second, ML latched onto it. That gave Python a lot of credibility for people coming through CS programs and into industry, especially as AI use cases were picking up momentum. Think Tensorflow starting 10-ish years ago, then Pytorch, and so on.
[opinion time] Unfortunately, ML has contributed to disaster. The quality of software engineering in ML applications lags badly behind that in other segments, and there's where most Python development is happening now. The feedback loop of training Bad Python into LLMs is only making it worse. The skill floor was dropping, and now it's practically zero.
It also wasn't helped much by language design decisions. Gradual typing was a huge mistake. It would have been much better to go the Typescript route and define a superset of the language that compiles to native Python. The grafted-in typing features that rely on runtime behavior and never got an actual spec for a type system haven't helped.
It's incredibly easy for new engineers to pick up. On the spectrum between "features easy for beginners but nightmarish for experienced engineers" and "features complex for beginners but great for experienced engineers" it always makes the tradeoff for the former.
Go is awesome and has great ecosystem. I think if you want to just deliver Go is one of the best language out there. Of course as long as the project makes sense(I.e no client side/GUI stuff, there it gets messy)
It can do everything, it's a nightmare to write correctly and is so full of warts that really only a mother could love it.
Go is "elegant" and properly designed, which leads to fundamental unassailable limitations; like its GUI story or writing a backend for other languages (cgo is so slow and -build-mode=c-shared is a hog).
No operator overloading makes it difficult for numerical/scientific work - Python is stronger here.
"just get stuff done" is your crappy dump truck which is welded together out of spare bits and is barely hanging on, this is how I feel about C++.
Go is a lot more elegant and well designed, but it requires "smooth roads" (An OS and Runtime) and goes fast... it's more like a Dodge Hellcat..
EDIT: if we consider speed a function of development time rather than runtime, of course at runtime C++ can be as fast as it gets.
The dump truck that is C++ definitely goes just as fast if not faster than that Hellcat, though.
To me it seems more like C++ is the metal shop's old school 3-axis mill: effective and dangerous. Can do almost anything, including easily maim you unless you're continually vigilant.
Go seems like a CNC machine: just as effective, much less dangerous because it relies on digital safeguards and embedded controllers. You need to go to extreme lengths to make it do something it wasn't designed to do.
Python seems like lego. Shit simple to build things with, lets you start building immediately, fantastic for prototypes, but finished product is still just press-fit plastic bricks
It's no wonder then that Google used to be all in on Python in the 2000s as a language for juniors and then invented another language with similar users in mind. With AI becoming what it is they probably did not need to do the latter.
Python owes a lot to that initialish Google push where for a few years you couldn't get a job at google without knowing python or you could easily get a job if you knew python. Maybe both.
It's definitely a better language than Matlab, and Numpy made it actually fast for that purpose. AI research really took advantage of that even before LLMs hit the scene.
I'm not the OP, but it seemed to me that major teaching institutions settled on Python as a pedagogic language after previously having rallied around Java for same, and various other languages before that.
They provide binaries for almost every open source library for every popular platform. If you need something, whatever it is, python was and is probably the easiest way to install and use it.
Marketing in academia, decent C-API that led to use in scientific Python.
The language is a disaster since it deviated from its original purpose. The community is ruled by insane, mediocre narcissists who drove away all reasonable people.
It was the first language that actually felt ready-to-use, available for free-as-in-speech. Batteries included, robust online development community, sensible featureset, main libs that were designed bottom-up instead of top-down. I remember learning Java 1.2 or 1.4 after learning Python and it felt like a huge regression. Swing was a joke compared to TkInter (which was weak compared to proper GUI editors like VB but at least made nice scalable native-feeling GUIs and had a sensible API) and the standard collections were miserable without generics.
But Python peaked like 20 years ago. The later features were too ugly, and it was too slow to fill the obvious gaps in the language.
I've been thinking recently, there's hardly any reason not to use a native language like Rust anymore. The classic reasoning for languages like Python is that it prioritizes developer time and resources over compute resources, but now developer time is rapidly trending to zero and everything is compute dominated.
Not having to build still saves wall clock time. Also Python is a concise language (fewer tokens in context) with a monster ecosystem that avoids a lot of code having to written by agents (fewer tokens for the current task and fewer for future tasks by keeping the code base smaller).
Most of that monster ecosystem is exactly that: monstrous code in one-off throwaway libraries written by people who don't care about software, as it's not their job.
If you set a bar, any bar, there's really not much offered by Python that you cannot find in a more reasonable language. Especially if you're gluing together C++ and Rust libraries for data processing or ML workloads. Python is still disproportionately popular there but I'm also convinced its days are counted. C++ will be tremendously harder to get rid of, in comparison, if that's even possible.
I work in python. It's only real and big issue is the slow interpreter. Everything else is resolvable one way or another. The only other big issue it has is the envy of people who imagine that if not for python, their beloved language would be as used as python is now. Most of their beloved languages are older than python or when they were created, they had much bigger ticket backing than python and those languages are still not where python is. The reason? Python is made to solve the issue one has now, not the issue one might have in the future. Local optimum at its best and worst. And honestly, most people have the current issue to solve and rarely reach the imaginary issue to be. If python didn't exist, it would have been invented.
The issue is that it just becomes a “bad at everything” tarpit.
- performance is awful.
- gradual typing is sort of ok, but relies too much on runtime behaviour.
- async support is a bit all over the place.
- uv has made the packaging experience sort of liveable, but collectively it’s still very limited: feature support, conditional dependencies, rudimentary tree shaking, cargo-style workspaces, etc.
In an era where your options of tech stack are freer than ever, unless you explicitly need PyTorch, I’m not really sure why you’d choose Python over literally everything else.
As a Ruby dev I'm kind of primed to hate on Python a little bit, but I have to agree. I've written some Python over the years, I've tried to like it, but it just doesn't feel good to me. There are things that are really great about Rust, Elixir, C#, Swift, Java, Common Lisp... Sometimes it's the standard library, sometimes the package manager, sometimes a feature (borrow checker, pattern matching), sometimes a principle (everything is an object, the code is just data). In Python, everything is kind of meh. Like they came up with the list comprehension syntax and that was it for cool things.
List comprehensions? Really? I remember when they were pretty and I was learning them alongside other languages with higher order functions and Python's approach seemed to be the ugliest of its cohort. They were neither as logical as simple Lispy "take a function as a parameter" things like Ruby, nor were they the kind of clean and legible stuff we expect from Python.
I'm not saying I personally like them, but they're not as bland as the rest of the language. FWIW, I watched 3 non-dev people around me learn Python in uni (biologist, lawyer, graphics designer) and they all liked the list comprehension syntax.
Venvs, activate, deactivate, shitty scripts hacking on $PATH. How did this ever ship? Then there was poetry, `uv`, and maybe something in else in $CURRENT_YEAR.
Try to set it up on a new macOS, go on. Install a package from pip, and there's a bunch of garbage that won't let you do it.
It's horrible. Python packaging engineer is now an FTE, maybe that's what they wanted.
Python had no corporate backing, no marketing, it became popular because people actually like to use it. Sorry you've only been using "better thought out languages" instead.
The syntax and meaningful spaces are annoying. The dynamic typing is abused to hell in everything including the standard library. It has wacky paradigms like "exceptions instead of conditionals" or having "else" clause for loops.
It lacks decent packaging and by design cannot handle multiple versions of dependencies or perform metadata-only/declarative resolution. To resolve the dependencies of a Python package you need to install it. Many Python packages resolve their dependencies imperatively depending on the current system state. When you need binary dependencies like numpy it gets even worse since libpython doesn't have a stable ABI.
It is one of the most underdesigned languages but it holds a dear place due to being easier to translate from pseudocode to actual implementation. This resulted many academics to pick it up and write advanced math libraries in it.
Trisodium phosphate was at one time extensively used in formulations for a variety of consumer-grade soaps and detergents, and the most common use for trisodium phosphate has been in cleaning agents.
i primarily work in python and from a language-design standpoint, sure, there are generally better options out there. but all the people convincing themselves that a language with this much of a user base and this large of an ecosystem will fade away because of ai or some other reason are deluding themselves. i'm sure the death of python will be imminent for the next several decades, just as it has been for java
languages with dynamic types will be obsolete thanks to AI, you heard it here first. It is just introduces more errors, with AI you want formal checking, sure static types are not that, but it is a small piece of it. If 10 percent of errors are types, with a typed language keeping everything the same you have 10 percent less errors, simple as that.
100%. This is a solution to a problem you wouldn't have if you were just using a statically typed language. AI will naturally drive us to more rigid languages.
DetachHead (maintainer of basedpyright) has criticized the TSP before:
- https://docs.basedpyright.com/v1.40.2/type-server/
- https://github.com/microsoft/pylance-release/discussions/718...
Ty (the type checker maintained by astral) however will apparently support the TSP: https://github.com/astral-sh/ruff/discussions/23432
I think the only way MS knows how to make money is lock someone into an ecosystem. I think their next attempt should be a closed source lean checker that has will have proprietary code for NL -> lean formalization checker.
Applies to all big tech contributing to FOSS, they don't do it out of charity.
Definitely not true. Historically, it's been one of the ways they attract (and scout) top talent.
So they do it to attract talent then, and not out of charity?
Free talent, and definitely true.
I don't think that's true. Many big companies contribute to FOSS in ways that aren't just to attempt to lock people in. Often it's because they use the software and don't want to maintain patches forever - everyone wins if things actually get fixed. Or even because they just want to be known as a good place to work.
That describes how hands off MS was with Postgres committers ime a few years ago. Useful to house them to for helpful office hours too. We'll see how that holds up post HorizonDB
So… not out of charity then?
You're mixing motivation and end goal. Just because their motivations aren't purely charitable doesn't mean the end goal must be nefarious.
No, mostly not, but that does not mean they engage with locking customers in.
Of course not? I don't think anyone claimed it was out of charity.
Not purely, but "charity" and "only doing it to lock people in" are not the only options.
Name one big tech company in the same monetary level, that doesn't contribute to FOSS without second intentions.
They definitely have other intentions, but it's often the same as people contributing in their free time: they use the software, want to make it better for their purposes, but don't want to be the ones to maintain it.
Don't we all have "second intentions"?
Altruism happens usually among members of a tightly knit group, like family members or soldiers of a given unit. Teachers sacrificinng their life to save children. FOSS-developers, are they really producing software out of altruism? Maybe some of them are but it is hard for us to know whether that is so or not.
I don't think any (longer term) FOSS developer develops software out of altruism.
They mostly start the software because they themselves want to have it.
When they start working mostly for other people's needs, I think it's a mix of altruism, pride and probably enjoying the work.
If a FAANG releases Kafka, REACT, Kubernetes, or any number of projects that they found helpful, and released into open source - there will always be a claim that they do it with some second intention, such as "be seen to be a positive contributor to the community"
But the truth is - so do I - as a humble contributor to a document, test, bug fix, or feature add to a FOSS project - where's the beef?
If only, those FAANGs with hyperscaler and AI business always have a way to plunge those improvements into customer acquisitions for their cloud products, not available on the other vendors.
No. Microsoft and facebook are the try-hardest by far.
Yes, Microsoft fired the Faster CPython team and Facebook/Instagram tries to buy its way into Python core in order to get their buggy free threading in.
Totally altruistic!
NVIDIA, AWS, Google, Intel, AMD,...
This should be called Python Type Server Protocol, no? Seems weird to mine the generic protocol namespace for a protocol that only supports Python. Especially coming from the company behind LSP, which actually is generic.
Type Server Subsystem for Python
Copilot+ Type Server Subsystem for Python Series S
Copilot+ Type Server Subsystem for Python Series S .NET
…Server 2027.
...Subscription Edition
...365
.PYTHON
For Teams
… Home Edition
SP3
MAX PRO
Language Server Protocol Server Subsystem for Type Server Protocol Servers for Python
... for Workgroups
Python Server Subsytem Typing - PSST
It is namespaced under the pylance project. I also don't see any announcement here, just a markdown file.
Perhaps the post title could use a tweak
This was the announcement https://devblogs.microsoft.com/python/announcing-tsp/
This is the company that named its web framework "MVC". At least this time they're not eternally polluting search results for an industry standard term.
I've usually heard it called "ASP.NET MVC" except when the former part was clear from context.
MVC is still a general way of structuring code. Would it be the only way a MVC framework could or should be implemented for ASP.NET?
That's an argument against any descriptive name at all.
100% - it would make more sense if they also supported elixir or other dynamic languages
I do wonder if it generalizes outside of python similar to how LSPs do.
So now I need a TSP AND an LSP eating loads of memory?
As someone with minimal knowledge of LSP specifics, I'm a bit confused about why this exists. Is it the case that LSP alone does not provide adequate functionality for Python type queries?
> why this exists
https://docs.basedpyright.com/v1.40.2/type-server/
> The "type server protocol" is a desperate attempt from microsoft to prevent pylance from becoming obsolete by outsourcing its type checking to other language servers like basedpyright, ty or pyrefly -- while continuing to keep certain language server features exclusive to pylance.
> This protocol benefits nobody except microsoft. It adds needless complexity to any language server that chooses to support it. Basedpyright does not support it and strongly advises other language servers to avoid supporting it as well.
> For more info see here https://github.com/microsoft/pylance-release/discussions/718... and here https://github.com/astral-sh/ty/issues/1190#issuecomment-337...
Are language servers forced to support it?
"Good" to see the Embrace, Extend, Extinguish playbook is still being used.
https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...
It seems to be a more structured representation of the type. Normally in LSP you'd have a plain string or markdown that you render in the IDE, but nothing that you can analyse structurally.
It's a shame it's not just an LSP extension though, or a feature in a newer LSP version. LSP is maintained by Microsoft engineers.
workspace/symbol will do it (and the output may need some massaging), but it's optional.
I thought workspace/symbol returns an unstructured payload wrapped w/ some metadata.
Since the format is unspecified I imagine many language servers would just return a string representation of the type, which in Python is a PITA to parse because not every type has a closed-form representation (overloaded callable types, for example, have to be declared as a protocol).
Fair enough, I had no idea it was unstructured.
It’s really good that folks over at Astral decided to implement it. It was difficult for me to understand what does this protocol provide that couldn’t be implemented with a new LSP capability, the linked document doesn’t really explain that, but what did become clear to me is that seemingly Microsoft are trying to invent new features for their proprietary LSP server that only works in VS Code so people will have harder times switching to other editors. This way, no matter what Microsoft comes up with, will be matched with very high quality open source tools.
Indeed, this seemed not to happen to the same degree: https://ghuntley.com/fracture/
vscode forks were shown to be viable.
So basically adding machine readable type-at-point queries as an lsp extension so Pylance and Pywright can be separated?
The 'result types'format is super python specific, like not having a for all binder. I don't think this specific schema will be used for any non-python languages.
Can someone explain what this is to someone like me that doesn't know python very well, but programs a lot in c#?
Language server protocol is the interface of a software piece that sits between an IDE and a language toolchain so you don't need to change code in the IDE to add support of new languages (you add support on the LSP server). Then server provides things such as completions/intelli-sense, navigation to definitions and some correction checking. From the anouncement TSP extends LSP for better python type awarness.
Thanks
I just use the Pyrefly extension in VSCode/Zed, run Pyrefly in the CI, and everything works well. The fragmentation around pyright/pylance/basedpyright/cursor-pyright etc. makes it pretty much unusable, you can't get consistent results across IDEs.
It kept with UTF-16. New protocol is a chance to correct that poor decision.
It's just an .md file and a few hundred lines of json schema defining some extension messages, not a bottom-up protocol revision.
What's wrong with UTF-16 for transport?
In LSP, UTF-16 is not just for transport. Source locations are also referred to by their UTF-16 offset. That means applications are required to store a persistent mapping from UTF-16 offsets to whatever they use internally, keep that mapping updated as the document is edited, and map back and forth constantly.
Oh well. It was primarily designed to be a good protocol for VSCode, not for any other users. That's just the price you pay when using a corporate standard.
No? In LSP source locations are line+character: https://microsoft.github.io/language-server-protocol/specifi...
https://microsoft.github.io/language-server-protocol/specifi...
The column is measured in UTF-16 code units
Oh you're right, except since LSP 3.17 position encoding can be negotiated with clients/servers that support it. I initially got confused because my work with LSP started with 3.17 and I always used UTF-8 encoding.
https://microsoft.github.io/language-server-protocol/specifi...
When I wrote the initial version of the neovim LSP client, I got intimately familiar with LSP, and how much I hate it. Microsoft protocols have a habit of being extremely specialized to their application despite being recommended for broader use. I will never, ever, ever adapt another microsoft language protocol.
Is there any application for that, except python?
Right now it's specific to Python but I think there are other languages w/o an official type checker that might benefit.
PHP, Erlang, Ruby, etc.
The primary reason i guess is standardizing the type checking, so you can bring your own (3rd party).
Reading the main README, I'm not sure why I'd reach for this over ant other typed JSON RPC, tRPC or similar. What are the benefits I'm missing here?
It's such a shame python got such mindshare. It's a disaster of a language and ecosystem that people keep bolting and hacking things that much better thought out languages have had for decades.
At this point, Python is less of a programming language by itself but more of an interop protocol between various C/C++/Fortran/Rust binaries. But again, it's just part of the cycle, Python was a breath of fresh air 10 years ago when academia is still dominated by Matlab, but now it is mostly a mess of a language that's good for throwaway scripts but not very convenient for large software projects compared to statically typed languages.
thank you for putting this this way. it helped me crack a big unrelated problem
Out of curiosity, may I ask what the big unrelated problem is?
still trying to figure out a way to explain it, but i thought i needed a programming language, and it turns out what i need is an interop protocol. i want to be able to "translate" English text to a more or less dense rendering (or a completely different tone/pedagogical approach) without adding or losing meaning.
Interesting. Look up "semantic compression", and you can also try out one of my experiments to see if it works for you.
https://github.com/yuechen-li-dev/GenerativeCompressionProto...
wow i will definitely try this out. this might be even MORE helpful!
"without adding or losing meaning" sounds impossible for spoken language.
it does, and it obviously is, but i guess that is what makes it fun!
I remember falling in love with Python when I first learned it at version 1.5, liked it even more with 2.0, but when 3.0 came around, that's when I noticed the ecosystem falling apart. Not for the obvious incompatibility schism: I do think most of 3.0's changes were good ones, and it finally is the standard default version now, but in many other ways it feels like the ecosystem has still never really recovered. Virtualenv is still the same grotesque hack, where rather than doing what other languages like Perl and Ruby do and making some wrapper around setting PYTHONPATH, it relies on python's behavior of locating the library path relative to the install, thus hardlinking a whole python installation in and making you "activate" it first. That nothing has managed to replace it in 20 years speaks volumes.
I'm hoping one day people look at Python the same way they look at Perl and PHP.
There has been so much programming language innovation in the past decade that I'm surprised someone hasn't built a better Python. I'd like to see something like Rhai or Roto take off which has a natural upgrade path to Rust.
Maybe it's because interpreted languages just aren't in vogue anymore and probably won't be for a long time if humans aren't even coding anymore.
> I'm surprised someone hasn't built a better Python.
I believe Mojo is an attempt at that. Python is probably too entrenched by my estimation.
Roughly the time when Guido stopped fighting the hordes. The hordes won, wrested the language away from its creator and now they own the pieces. 2 to 3 was the worst possible way a language has ever been upgraded bar none. Hilariously, even today Python 2 persists in many places because of this.
> 2 to 3 was the worst possible way a language has ever been upgraded bar none
Perl 6?
Which gave up and became its own language, its own level of confusion.
Essentially killing both languages.
> Formerly named Perl 6, it was renamed in October 2019.
I can't imagine how anyone can complain about this. The community revolted. The team listened. What else do we want?
Guido wrested himself by accepting the stupid := operator against the huge opposition.
> Guido wrested himself by accepting the stupid := operator against the huge opposition.
Yes, that's the attitude I was talking about.
See, when you are a maintainer of a popular language with a community at some point the bickering becomes the job. And when you are a creative person managing a horde of egos who all participate to put their stamp on something the only reasonable thing to do is to leave them to their own devices, which he did. I really don't blame him. Guido is one of the nicest people and to see how the Python community did not have a single second's consideration for how their incessant infighting must have made him feel is one of the worst stories of the FOSS movement.
Guido was celebrated because he had taste.
If you don't want bickering create a closed project, make it open but don't accept contributions, etc.
You can't have it both ways "I want a big diverse community, but they should all agree with me"
I think the idea is that you start out believing people will cooperate, and that the best idea will naturally surface, and then you find out humans don’t actually work like that.
It's not about agreement. It's about basic decency.
And a couple of psychopath core devs gleefully celebrated hours after his resignation, marginalizing the event and tried to grab power directly. The two snakes are an appropriate symbol for Python.
Exactly. No good deed goes unpunished.
It has found its way in many Linux distros as a replacement for bash scripts when the implementation becomes just too convoluted. It's a boring syntax (though with few semantic quirks), intuitive to understand and hack, so the exact opposite to shell scripts. It get replaced by an implementation in C or Rust when the perfs start to falter. Another possible reason is that it's taught in college as an introductory language, and also that it became popular in AI circles, and thus models were mostly trained on it, perpetuating its usage. They got much better with other languages though, so maybe the situation is changing (improving maybe?). It has its merits: the syntax forces students to format their code correctly, it is very close to pseudo-code (is that a good thing though, in the end?).
> thus hardlinking a whole python installation in and making you "activate" it first
You don't have to activate a venv to use it; `$VENV_PATH/bin/python3` just works.
(It also softlinks/symlinks the binaries.)
I work in python daily, it's not necessarily my favourite language, but I'd say it's mostly fine.
What languages are you thinking of specifically when you mention better thought out ones?
Pretty much every other mainstream language bar JS. I'd say also Ruby because I also do not like a lot of its choices, but there's at least a kind of heady elegance to it in a REPL, though I would never want to work on a production system written in Ruby.
Frankly, I just really prefer statically and strongly typed compiled languages.
I did my last company's first Demo and MVP in Python after running through the Koans at my co-founder's behest and ultimately we decided that the production system would not be done in Python. There are some cool things about it - specifically I do like that annotations are real AOP seams, but the numerous footguns and lack of types and a compiler just make it a very annoying ecosystem to work in for me.
They are a heavy C# user. Coke/Pepsi situation.
Java would be Coke to C#'s Pepsi
Those are closer languages but my broader point is that these application-level languages are more or less interchangeable from a professional standpoint. A Java or C# pro should probably not spend time becoming a Python pro because they could have spent that time producing equivalent value in the languages they are already experts in.
I don't agree at all that it's a coke/pepsi situation. That analogy is for when two mostly-interchangeable choices develop tribal followings based largely on personal preference. Coke and Pepsi are both caramel carbonated soft drinks that are terrible for your health differing only slightly in flavour. Otherwise they're materially identical.
C# is strongly/statically typed, compiled, and quite fast (after the vm has started), whereas python is weakly/dynamically typed, interpreted, and generally slow as fuck. Those are two fundamentally different languages, how could it possibly be a coke/pepsi situation?
I think python is fantastic for certain use cases: gluing other tools together (like LAPACK or other ancient libraries), system automation, prototyping, exploitation (I think pretty much all of us were working on stealmybitcoin in python), and exploratory data analysis.
The problems arise when people try to write, say, gigantic GUI applications in python, or really anything with CPU-bound workloads. Part of being an engineer is using the right tool for the right job. Python is the wrong tool for lots of jobs. That shouldn't be a controversial statement.
> whereas python is weakly/dynamically typed
It's strongly/dynamically typed, not weakly typed. Why do people keep getting this wrong? Has any version of Python been weakly typed? I know Python 2 and 3 aren't, perhaps the first version or ABC, its predecessor language.
> Why do people keep getting this wrong?
It's kinda funny to say people are getting it "wrong" when strong/weak typing isn't really defined anywhere or universally agreed upon.
In fact, even "dynamic typing" can be considered to be an oxymoron, since types are formally defined to be a static property of a language. A more correct term would be "uni-typed".
I'm not saying that the meanings you're using for static/dynamic and weak/strong are bad, just that chastising someone for getting them "wrong l" doesn't make sense for terms that are not widely agreed upon.
You're trying too hard to play the role of the "aktually" guy. Assuming good intentions and taking a pass at replying would have been a better contribution to the discussion.
Numpy is over 2 decades old. Pytorch is a decade old. Python devs know what "right tool for the job" means and focus on solving problems instead of being language purists.
This has nothing to do with purism. I use python all the time, where it makes sense.
What I don't do is cultivate a "favorite language" and make it my whole identity, and then take criticism of that language as some sort of personal affront. Python objectively has some serious flaws. That shouldn't make anyone emotional, and I don't know why people get their backs up like this.
Who is cultivating a "favorite language"? I missed that part. Agreed that this should not be an emotional issue, that is unprofessional.
Problem with any popular language is an outsider sees one or two statistically-likely codebases that are total turds, and then gets one-shot by the Goomba fallacy.
> Who is cultivating a "favorite language"?
I don’t know, but deflecting legitimate criticism of a language by calling it “language purism” seems pretty emotional to me.
And the thing that maddens me the most is that it has a critical mass where it’s basically the default starting language and most people never really move from there. And so we’re stuck with this grotesque abomination that somehow keeps on rolling. The amount of money that was poured into python to make it barely bearable is probably the GDP of a small nation.
I’m not a fan either, but it doesn’t bother me at all. Who cares? Let the python people do their thing.
Fascinating take.
What do you attribute Python's success to, in spite of being a disaster of a language and ecosystem?
20 years working on Python. Two things have driven its success.
First, it's easy to learn and reasonably powerful. You can make a lot of useful things really quickly, which was true even 15 years ago. It's only gotten easier since, and the ecosystem has continued to grow.
Second, ML latched onto it. That gave Python a lot of credibility for people coming through CS programs and into industry, especially as AI use cases were picking up momentum. Think Tensorflow starting 10-ish years ago, then Pytorch, and so on.
[opinion time] Unfortunately, ML has contributed to disaster. The quality of software engineering in ML applications lags badly behind that in other segments, and there's where most Python development is happening now. The feedback loop of training Bad Python into LLMs is only making it worse. The skill floor was dropping, and now it's practically zero.
It also wasn't helped much by language design decisions. Gradual typing was a huge mistake. It would have been much better to go the Typescript route and define a superset of the language that compiles to native Python. The grafted-in typing features that rely on runtime behavior and never got an actual spec for a type system haven't helped.
It's incredibly easy for new engineers to pick up. On the spectrum between "features easy for beginners but nightmarish for experienced engineers" and "features complex for beginners but great for experienced engineers" it always makes the tradeoff for the former.
Yes, just like Go, it's a junior developer onboarding language, that somehow managed to overstay its welcome.
Go is awesome and has great ecosystem. I think if you want to just deliver Go is one of the best language out there. Of course as long as the project makes sense(I.e no client side/GUI stuff, there it gets messy)
Some people live to shit on Go since you can't really do "crazy cool shit" as someone I knew put it.
Apparently you can't be a senior dev unless you "just here to deliver"
C++ is the "just here to deliver" language.
It can do everything, it's a nightmare to write correctly and is so full of warts that really only a mother could love it.
Go is "elegant" and properly designed, which leads to fundamental unassailable limitations; like its GUI story or writing a backend for other languages (cgo is so slow and -build-mode=c-shared is a hog).
No operator overloading makes it difficult for numerical/scientific work - Python is stronger here.
"just get stuff done" is your crappy dump truck which is welded together out of spare bits and is barely hanging on, this is how I feel about C++.
Go is a lot more elegant and well designed, but it requires "smooth roads" (An OS and Runtime) and goes fast... it's more like a Dodge Hellcat..
EDIT: if we consider speed a function of development time rather than runtime, of course at runtime C++ can be as fast as it gets.
The dump truck that is C++ definitely goes just as fast if not faster than that Hellcat, though.
To me it seems more like C++ is the metal shop's old school 3-axis mill: effective and dangerous. Can do almost anything, including easily maim you unless you're continually vigilant.
Go seems like a CNC machine: just as effective, much less dangerous because it relies on digital safeguards and embedded controllers. You need to go to extreme lengths to make it do something it wasn't designed to do.
Python seems like lego. Shit simple to build things with, lets you start building immediately, fantastic for prototypes, but finished product is still just press-fit plastic bricks
Well, if we consider speed of development rather than speed of the runtime, the analogy holds better.
If Go is a junior developer onboarding language, then I proclaim myself a perpetual onboarding junior.
And that's ok.
It's no wonder then that Google used to be all in on Python in the 2000s as a language for juniors and then invented another language with similar users in mind. With AI becoming what it is they probably did not need to do the latter.
Python owes a lot to that initialish Google push where for a few years you couldn't get a job at google without knowing python or you could easily get a job if you knew python. Maybe both.
Probably the batteries included Standard Library.
Cross platform interpreter.
NumPy and SciPy really set the stage.
It's definitely a better language than Matlab, and Numpy made it actually fast for that purpose. AI research really took advantage of that even before LLMs hit the scene.
BASIC rebooted. Right at the right time too, BASIC was never going to thrive in the days of the internet.
I'll give Python one thing though, it is by far the most elegant way to glue together a large number of otherwise incompatible C libraries.
I'm not the OP, but it seemed to me that major teaching institutions settled on Python as a pedagogic language after previously having rallied around Java for same, and various other languages before that.
They provide binaries for almost every open source library for every popular platform. If you need something, whatever it is, python was and is probably the easiest way to install and use it.
https://en.wikipedia.org/wiki/Worse_is_better
We've known for a very long time that shitty things can become very successful in this field and why.
well, for one, my boss can use it and kinda understand it, so it must be dead simple
Definitely ease of use...
Marketing in academia, decent C-API that led to use in scientific Python.
The language is a disaster since it deviated from its original purpose. The community is ruled by insane, mediocre narcissists who drove away all reasonable people.
It was the first language that actually felt ready-to-use, available for free-as-in-speech. Batteries included, robust online development community, sensible featureset, main libs that were designed bottom-up instead of top-down. I remember learning Java 1.2 or 1.4 after learning Python and it felt like a huge regression. Swing was a joke compared to TkInter (which was weak compared to proper GUI editors like VB but at least made nice scalable native-feeling GUIs and had a sensible API) and the standard collections were miserable without generics.
But Python peaked like 20 years ago. The later features were too ugly, and it was too slow to fill the obvious gaps in the language.
It smells like an easy to learn language, it’s interpreted and other competitors in that category were highly specialized or got stuck with DHH.
In other words it’s easy to sell. Combined with the widespread anti-intellectual attitude in the software industry that’s enough.
Edit: it does help it’s genuinely a good scripting language.
- Math + visualization libraries (which ironically have to break out of Python to do their job)
- Reasonable ergonomics for describing data flows, as long as you never run the code, which you often don't care about if you're a mathematician
- Adoption by people who optimize pumping citations, not reproducible results -- non-working code is a feature when you're trying to BS
I've been thinking recently, there's hardly any reason not to use a native language like Rust anymore. The classic reasoning for languages like Python is that it prioritizes developer time and resources over compute resources, but now developer time is rapidly trending to zero and everything is compute dominated.
Not having to build still saves wall clock time. Also Python is a concise language (fewer tokens in context) with a monster ecosystem that avoids a lot of code having to written by agents (fewer tokens for the current task and fewer for future tasks by keeping the code base smaller).
Most of that monster ecosystem is exactly that: monstrous code in one-off throwaway libraries written by people who don't care about software, as it's not their job.
If you set a bar, any bar, there's really not much offered by Python that you cannot find in a more reasonable language. Especially if you're gluing together C++ and Rust libraries for data processing or ML workloads. Python is still disproportionately popular there but I'm also convinced its days are counted. C++ will be tremendously harder to get rid of, in comparison, if that's even possible.
Rust has Rhai and Roto too if you need better than Python level of developer considerations.
Unpopular take, but:
I work in python. It's only real and big issue is the slow interpreter. Everything else is resolvable one way or another. The only other big issue it has is the envy of people who imagine that if not for python, their beloved language would be as used as python is now. Most of their beloved languages are older than python or when they were created, they had much bigger ticket backing than python and those languages are still not where python is. The reason? Python is made to solve the issue one has now, not the issue one might have in the future. Local optimum at its best and worst. And honestly, most people have the current issue to solve and rarely reach the imaginary issue to be. If python didn't exist, it would have been invented.
The issue is that it just becomes a “bad at everything” tarpit.
- performance is awful.
- gradual typing is sort of ok, but relies too much on runtime behaviour.
- async support is a bit all over the place.
- uv has made the packaging experience sort of liveable, but collectively it’s still very limited: feature support, conditional dependencies, rudimentary tree shaking, cargo-style workspaces, etc.
In an era where your options of tech stack are freer than ever, unless you explicitly need PyTorch, I’m not really sure why you’d choose Python over literally everything else.
What would you choose for a backend nowadays then?
No other language came even close to a combination of easiness and expressivity of Python.
OTOH I don't want to live in a universe where Haskell wins
I will take Python over typescript every second of every day. TypeScript is just so unreadable compared to Python.
As a Ruby dev I'm kind of primed to hate on Python a little bit, but I have to agree. I've written some Python over the years, I've tried to like it, but it just doesn't feel good to me. There are things that are really great about Rust, Elixir, C#, Swift, Java, Common Lisp... Sometimes it's the standard library, sometimes the package manager, sometimes a feature (borrow checker, pattern matching), sometimes a principle (everything is an object, the code is just data). In Python, everything is kind of meh. Like they came up with the list comprehension syntax and that was it for cool things.
List comprehensions? Really? I remember when they were pretty and I was learning them alongside other languages with higher order functions and Python's approach seemed to be the ugliest of its cohort. They were neither as logical as simple Lispy "take a function as a parameter" things like Ruby, nor were they the kind of clean and legible stuff we expect from Python.
I'm not saying I personally like them, but they're not as bland as the rest of the language. FWIW, I watched 3 non-dev people around me learn Python in uni (biologist, lawyer, graphics designer) and they all liked the list comprehension syntax.
The packaging is a nightmare thats for sure.
Venvs, activate, deactivate, shitty scripts hacking on $PATH. How did this ever ship? Then there was poetry, `uv`, and maybe something in else in $CURRENT_YEAR.
Try to set it up on a new macOS, go on. Install a package from pip, and there's a bunch of garbage that won't let you do it.
It's horrible. Python packaging engineer is now an FTE, maybe that's what they wanted.
The fact that they didn't solve Venv when they did the breaking change in Python 3 is absolutely bonkers.
Python had no corporate backing, no marketing, it became popular because people actually like to use it. Sorry you've only been using "better thought out languages" instead.
Python should have done runtime type checking instead of compile-time only String and List annotations and so on
Worse is better. Avoid using it yourself and tell others; that's the best you can do.
Python is pretty decent. What are you talking about?
The syntax and meaningful spaces are annoying. The dynamic typing is abused to hell in everything including the standard library. It has wacky paradigms like "exceptions instead of conditionals" or having "else" clause for loops.
It lacks decent packaging and by design cannot handle multiple versions of dependencies or perform metadata-only/declarative resolution. To resolve the dependencies of a Python package you need to install it. Many Python packages resolve their dependencies imperatively depending on the current system state. When you need binary dependencies like numpy it gets even worse since libpython doesn't have a stable ABI.
It is one of the most underdesigned languages but it holds a dear place due to being easier to translate from pseudocode to actual implementation. This resulted many academics to pick it up and write advanced math libraries in it.
I blame Perl 6
it could be worse like …PHP
> It's a disaster of a language and ecosystem
"unhinged" take
Interesting, LSP, DAP, MCP, and TSP now...
TSP : TriSodium Phosphate. Very good for cleaning up a mess.
From Wiki:
Trisodium phosphate was at one time extensively used in formulations for a variety of consumer-grade soaps and detergents, and the most common use for trisodium phosphate has been in cleaning agents.
Ref: https://en.wikipedia.org/wiki/Trisodium_phosphate
Is it Python specific only?
i primarily work in python and from a language-design standpoint, sure, there are generally better options out there. but all the people convincing themselves that a language with this much of a user base and this large of an ecosystem will fade away because of ai or some other reason are deluding themselves. i'm sure the death of python will be imminent for the next several decades, just as it has been for java
Blog posts:
Pylance + Pyrefly: bring your preferred type checker to Visual Studio Code
https://devblogs.microsoft.com/python/announcing-tsp/
Pyrefly + Pylance Integration in VS Code
https://pyrefly.org/blog/pyrefly-pylance/
Type-shed, type-server. When can this Python madness stop? How about using CORBA to make it even more enterprisey?
Why don't you spend your resources on F# instead of this nonsense?
> Why don't you spend your resources on F# instead of this nonsense?
Absolutely!
F# is criminally under-invested in.
Exactly what i came to write as well. I'd expect Microsoft to invest in F# so that python can slowly fade away the way php did.
Run server on your PC to serialize and deserialize JSON payloads back and forth just to get basic IDE functionality bro.
The cottage industry around fixing gaps in Python is sometimes awe-inspiring and most times just downright awful.
Maybe just make a Python 4 with better types built in at this point...
languages with dynamic types will be obsolete thanks to AI, you heard it here first. It is just introduces more errors, with AI you want formal checking, sure static types are not that, but it is a small piece of it. If 10 percent of errors are types, with a typed language keeping everything the same you have 10 percent less errors, simple as that.
100%. This is a solution to a problem you wouldn't have if you were just using a statically typed language. AI will naturally drive us to more rigid languages.