Older HN users may recall when busy discussions had comments split across several pages. This is because the Arc [1] language that HN runs on was originally hosted on top of Racket [2] and the implementation was too slow to handle giant discussions at HN scale. Around September 2024 Dang et al finished porting Arc to SBCL, and performance increased so much that even the largest discussions no longer need splitting. The server is unresponsive/restarting a lot less frequently since these changes, too, despite continued growth in traffic and comments:
"Btw, we rolled this out over 3 weeks ago and I think you're the first person to ask about it on HN. There was one earlier question by email. I think that qualifies as a splash-free dive."
It was running BC. I had high hopes for switching to CS because I'd heard the same thing you had, but when I tried it, HN slowed to a crawl. This stuff is so unpredictable.
I was the person who emailed him about it earlier.
2024-09-05, me:
On another topic, I just noticed that the 700+ comments on https://news.ycombinator.com/item?id=41445413 all render on a single page. Hurray! Is the pagination approach obsolete now? I know that you've commented several times about wanting to optimize the code so pagination wasn't necessary. I don't know if that's finished or if pagination will have to go on the next time there's a big breaking story.
Dan G:
Yes: the performance improvements I've been working on for years finally got deployed, so pagination is turned off for now.
(In case you're curious, the change is that Arc now runs over SBCL instead of Racket.)
...
Btw you're the only person I know of who's noticed this and pointed it out so far!
I have very mixed feelings about how much I know about this site.
It's different, yes. The HN implementation is called clarc. PG suggested we spell it "clerk" as a joke on the British pronunciation of the latter, but I chickened out.
I talked to one of the Anarki devs (or at least someone who uses it) about possibly open-sourcing a version of clarc which would run the original open-sourced version of HN, but it's a bit hard because the implementation would require some careful surgery to factor out the relevant parts.
Yes because we just add the things we need, at whatever layer it makes the most sense to add them.
This type of application stack that includes the language you're writing in and even, when convenient, its implementation, is really satisfying to work with. There is much less need for workarounds, arbitrary choices, and various indirections (e.g. what used to be called dependency injection, for example). All the plumbing is an order of magnitude simpler and it allows us to keep the codebase much smaller than it would otherwise be. I also spend basically zero time bitching about having to deal with software dependencies, making me realize how much of my former life as a programnmer was taken up with just that.
I think of this as sort of the unikernel form of application dev and of course it's a fine fit for a Lisp, since "write the language you want to write your program in as you write your program in it" is the natural style there. The tradeoff is that there's a lot of vertical coupling between the layers. If you want to factor out one layer for general consumption, e.g. to open source the language implementation so other people can build cool things with it, there's a fair bit of work to do.
Also, since the language implementation exists to run a specific application, we just don't bother supporting we don't need for HN. That too comes back to bite you when you want to open-source it!
HN has had 15+ years of work since the original news application was open-sourced; that's a lot of things-we-added at-some-point. Most of those are at the application level but some ended up in Arc and some in the Arc implementation it was convenient to put them there. This is especially handy when you have limited time to work on the code.
Replying while @dang is editing - so might be talking past current parent:
I suppose I shouldn't be surprised that Arc/clarc would be modified as news is modified (Arc sort of being built around news in the first place).
I just wouldn't expect there to be hn specific sauce in clarc that would make sense to excise if opening up clarc; AFAIK it's been stated that there's some secret sauce wrt fighting spam, detecting voting rings and so on...
Then again, thinking more about it, it sounds reasonable that some of that might land on the Arc/clarc side, not in news.
(sorry for editing on the fly - I can explain why I do that but I know it can be annoying when someone is trying to reply! and yes, all that sounds right.)
> too many focus on SBCL + Emacs and then complain about Lisp tooling.
well yea, lispworks and allegro are expensive commercial projects. I wish sbcl, the defacto best open source option had better tooling. emacs is great and all for the true believers but I'm an unwashed vscode user. For plenty of reasons I can't justify it in my startup but I'd love to spend more time working with common lisp for personal projects but my time is limited so I prefer clojure or rust.
LispWorks and Allegro are both interesting, but I've found their IDE offerings to be very limited. I haven't used either since I was playing around with CL during Covid, but from what I recall, even the basic IDE experience of writing code was severely lacking: poor autocomplete, poor syntax highlighting, clunky interfaces. In most discussions I see about them, they're only recommended for their compilers, not for their IDE offerings.
I think LispWorks is fine (also look at these plugins https://github.com/apr3vau/lw-plugins - terminal integration, code folding, side tree, markdown highlighting, Nerd Fonts, fuzzy-matching, enhanced directory mode, expand region, pair editing, SVG rendering…) but I had this feeling with the newer web-based Allegro IDE (the poor syntax highlighting surprised me, did I do sthg wrong?).
It was and remained an esoteric mystery to me ever since I saw Nichimen's work (with it); Pricing was just out of this world to even consider at the time.
I use emacs regularly (in fact I have it running right now) and I think the complaints against it are perfectly valid. Emacs is awesome in lots of ways, but it also really, really sucks in lots of other ways.
But putting emacs aside, the SBCL tooling seems reasonable to me. The real reason I rarely reach for lisp these days is not the tooling, but because the Common Lisp library ecosystem is a wasteland of partial implementations and abandoned code bases.
It's also been my experience that LLMs are better at writing more mainstream languages, especially "newbie-proof" languages like Go.
In any case, I don't see why one would reach for Allegro or Lispworks over SBCL unless one really enjoys writing lisp by hand and needs specific features they offer that SBCL doesn't. I would imagine those features are vanishingly few.
>SBCL derives most of its code from CMU CL, created at Carnegie Mellon University. Radical changes have been made to some parts of the system (particularly bootstrapping) but many fundamentals (like the mapping of Lisp abstractions onto the underlying hardware, the basic architecture of the compiler, and much of the runtime support code) are only slightly changed. Enough changes have been made to the interface and architecture that calling the new system CMU Common Lisp would cause confusion - the world does not need multiple incompatible systems named CMU CL. But it's appropriate to acknowledge the descent from the CMU hackers (and post-CMU CMU CL hackers) who did most of the heavy lifting to make the system work. So the system is named Steel Bank after the industries where Andrew Carnegie and Andrew Mellon, respectively, made the big bucks.
In addition to the official reference to CMU, there is a second origin for the name.
SBCL - Sanely Bootstrappable Common Lisp
You see, when SBCL was forked from CMU, a major effort was done so that it could be compiled using any reasonably complete Common Lisp implementation, unlike CMU CL. Because CMU CL essentially could only be compiled by itself, preferably in the same version, which meant compiling and especially cross-compiling was complex process that involved bringing the internal state of CMUCL process to "new version".
SBCL redid the logic heavily into being able to host the core SBCL compiler parts in any mostly-complete (does not have to be complete!) ANSI CL implementation, then uses that to compile the complete form.
Meaning you can grab SBCL source tarball, plus one of GNU clisp, ECL, Clozure CL, even GNU Common Lisp at one point, or any of the commercial implementations, including of course CMUCL, and C compiler (for the thin runtime support) and build a complete and unproblematic SBCL release with few commands
Can we get a "(1999)" date on this, please? Only half joking becuase I see Common Lisp and, sure, I upvote ... but honestly, what's the purpose of this HN submission without context?
SBCL is obviously fantastic but let's contrast with another popular implementation: Embeddable Common Lisp. https://ecl.common-lisp.dev/
Top marks for SBCL performance but ECL can be a better fit for embedding into mobile applications, running on lighter weight hardware, and in the browser.
We upgraded to 2.6.1 about a week ago and switched to using the new(ish) parallel(ish) garbage collector. I still can't tell what the impact has been.
Claude Code (which is a wizard at analyzing log files but also, I fear, an incorrigible curve-fitter) insisted that it was a real breakthrough and an excellent choice! On the other hand there was a major slowdown last night, ending in SBCL dying from heap exhaustion. I haven't had a chance to dig into that yet.
If by the cap you mean the heap size passed in as the --dynamic-space-size argument, then it definitely didn't hit the cap. It was using about 2/3 of that.
> My understanding is the new parallel GC compacts the heap rather infrequently
I'm going to caveat this by stating up front that obviously HN's source code is not public so I don't know what your hot path looks like, and that I'm not a domain expert on garbage collection, but I do write a fair amount of lisp for SBCL.
Immix-style collectors, like the new GC in SBCL, only compact on an opportunistic basis and so you get fragmentation pressure under load. In that situation, you might be well under the dynamic space size cap but if it can't find a large enough contiguous chunk of free heap it will still die.
So, fragmentation would be my prime suspect given what you described.
A well known quantum computing company's entire stack runs on SBCL, with Emacs in production... works really well, don't knock it until you've tried it. Phenomenal REPL.
Older HN users may recall when busy discussions had comments split across several pages. This is because the Arc [1] language that HN runs on was originally hosted on top of Racket [2] and the implementation was too slow to handle giant discussions at HN scale. Around September 2024 Dang et al finished porting Arc to SBCL, and performance increased so much that even the largest discussions no longer need splitting. The server is unresponsive/restarting a lot less frequently since these changes, too, despite continued growth in traffic and comments:
https://news.ycombinator.com/item?id=41679215
[1] https://paulgraham.com/arc.html
[2] https://racket-lang.org/
I love this comment:
"Btw, we rolled this out over 3 weeks ago and I think you're the first person to ask about it on HN. There was one earlier question by email. I think that qualifies as a splash-free dive."
I had no idea and I'm an HN addict!
The "splash-free dive" metaphor came from the greatly missed sctb and was the springboard for much fun conversation at HNHQ.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
I’m curious: was Arc running Racket BC or CS? I understand it got a big performance boost after switching to Chez Scheme.
It was running BC. I had high hopes for switching to CS because I'd heard the same thing you had, but when I tried it, HN slowed to a crawl. This stuff is so unpredictable.
I was the person who emailed him about it earlier.
2024-09-05, me:
On another topic, I just noticed that the 700+ comments on https://news.ycombinator.com/item?id=41445413 all render on a single page. Hurray! Is the pagination approach obsolete now? I know that you've commented several times about wanting to optimize the code so pagination wasn't necessary. I don't know if that's finished or if pagination will have to go on the next time there's a big breaking story.
Dan G:
Yes: the performance improvements I've been working on for years finally got deployed, so pagination is turned off for now.
(In case you're curious, the change is that Arc now runs over SBCL instead of Racket.)
...
Btw you're the only person I know of who's noticed this and pointed it out so far!
I have very mixed feelings about how much I know about this site.
I suppose this version of Arc for sbcl is different from what hn runs on?:
https://github.com/pauek/arc-sbcl
And there's no version of Anarki that runs on sbcl?:
https://arclanguage.github.io/
It's different, yes. The HN implementation is called clarc. PG suggested we spell it "clerk" as a joke on the British pronunciation of the latter, but I chickened out.
I talked to one of the Anarki devs (or at least someone who uses it) about possibly open-sourcing a version of clarc which would run the original open-sourced version of HN, but it's a bit hard because the implementation would require some careful surgery to factor out the relevant parts.
There's hn specific parts to the clarc implementation of Arc? (As opposed to the hn version of the "news" application)?
Yes because we just add the things we need, at whatever layer it makes the most sense to add them.
This type of application stack that includes the language you're writing in and even, when convenient, its implementation, is really satisfying to work with. There is much less need for workarounds, arbitrary choices, and various indirections (e.g. what used to be called dependency injection, for example). All the plumbing is an order of magnitude simpler and it allows us to keep the codebase much smaller than it would otherwise be. I also spend basically zero time bitching about having to deal with software dependencies, making me realize how much of my former life as a programnmer was taken up with just that.
I think of this as sort of the unikernel form of application dev and of course it's a fine fit for a Lisp, since "write the language you want to write your program in as you write your program in it" is the natural style there. The tradeoff is that there's a lot of vertical coupling between the layers. If you want to factor out one layer for general consumption, e.g. to open source the language implementation so other people can build cool things with it, there's a fair bit of work to do.
Also, since the language implementation exists to run a specific application, we just don't bother supporting we don't need for HN. That too comes back to bite you when you want to open-source it!
HN has had 15+ years of work since the original news application was open-sourced; that's a lot of things-we-added at-some-point. Most of those are at the application level but some ended up in Arc and some in the Arc implementation it was convenient to put them there. This is especially handy when you have limited time to work on the code.
Replying while @dang is editing - so might be talking past current parent:
I suppose I shouldn't be surprised that Arc/clarc would be modified as news is modified (Arc sort of being built around news in the first place).
I just wouldn't expect there to be hn specific sauce in clarc that would make sense to excise if opening up clarc; AFAIK it's been stated that there's some secret sauce wrt fighting spam, detecting voting rings and so on...
Then again, thinking more about it, it sounds reasonable that some of that might land on the Arc/clarc side, not in news.
[Ed: I think that turned out quite well]
(sorry for editing on the fly - I can explain why I do that but I know it can be annoying when someone is trying to reply! and yes, all that sounds right.)
While great option, LispWorks and Allegro Common Lisp should not be overlooked, too many focus on SBCL + Emacs and then complain about Lisp tooling.
> too many focus on SBCL + Emacs and then complain about Lisp tooling.
well yea, lispworks and allegro are expensive commercial projects. I wish sbcl, the defacto best open source option had better tooling. emacs is great and all for the true believers but I'm an unwashed vscode user. For plenty of reasons I can't justify it in my startup but I'd love to spend more time working with common lisp for personal projects but my time is limited so I prefer clojure or rust.
They have community editions.
I can't load my side projects in LispWorks' community edition, so… (because of the heap limit)
LispWorks and Allegro are both interesting, but I've found their IDE offerings to be very limited. I haven't used either since I was playing around with CL during Covid, but from what I recall, even the basic IDE experience of writing code was severely lacking: poor autocomplete, poor syntax highlighting, clunky interfaces. In most discussions I see about them, they're only recommended for their compilers, not for their IDE offerings.
I think LispWorks is fine (also look at these plugins https://github.com/apr3vau/lw-plugins - terminal integration, code folding, side tree, markdown highlighting, Nerd Fonts, fuzzy-matching, enhanced directory mode, expand region, pair editing, SVG rendering…) but I had this feeling with the newer web-based Allegro IDE (the poor syntax highlighting surprised me, did I do sthg wrong?).
It was and remained an esoteric mystery to me ever since I saw Nichimen's work (with it); Pricing was just out of this world to even consider at the time.
I use emacs regularly (in fact I have it running right now) and I think the complaints against it are perfectly valid. Emacs is awesome in lots of ways, but it also really, really sucks in lots of other ways.
But putting emacs aside, the SBCL tooling seems reasonable to me. The real reason I rarely reach for lisp these days is not the tooling, but because the Common Lisp library ecosystem is a wasteland of partial implementations and abandoned code bases.
It's also been my experience that LLMs are better at writing more mainstream languages, especially "newbie-proof" languages like Go.
In any case, I don't see why one would reach for Allegro or Lispworks over SBCL unless one really enjoys writing lisp by hand and needs specific features they offer that SBCL doesn't. I would imagine those features are vanishingly few.
I'd prefer that a LispWorks user answer, but there are quite a few interesting features, such as:
- tree shaking and small binaries (±5MB a GUI app)
- the CAPI cross-platform and native GUI toolkit
- mobile platforme runtime (iOs, Android)
- its Java interface
- its KnowledgeWorks system for "rule-based, object-oriented, logical, functional and database programming"
- more?
ps: today we maintain a list of pretty decent libraries: https://github.com/CodyReichert/awesome-cl/
How come it is named like that? It's a product of some old school consortium?
I know that ford,gm etc also made some R&D into software
From the About page:
>SBCL derives most of its code from CMU CL, created at Carnegie Mellon University. Radical changes have been made to some parts of the system (particularly bootstrapping) but many fundamentals (like the mapping of Lisp abstractions onto the underlying hardware, the basic architecture of the compiler, and much of the runtime support code) are only slightly changed. Enough changes have been made to the interface and architecture that calling the new system CMU Common Lisp would cause confusion - the world does not need multiple incompatible systems named CMU CL. But it's appropriate to acknowledge the descent from the CMU hackers (and post-CMU CMU CL hackers) who did most of the heavy lifting to make the system work. So the system is named Steel Bank after the industries where Andrew Carnegie and Andrew Mellon, respectively, made the big bucks.
In addition to the official reference to CMU, there is a second origin for the name.
You see, when SBCL was forked from CMU, a major effort was done so that it could be compiled using any reasonably complete Common Lisp implementation, unlike CMU CL. Because CMU CL essentially could only be compiled by itself, preferably in the same version, which meant compiling and especially cross-compiling was complex process that involved bringing the internal state of CMUCL process to "new version".SBCL redid the logic heavily into being able to host the core SBCL compiler parts in any mostly-complete (does not have to be complete!) ANSI CL implementation, then uses that to compile the complete form.
Meaning you can grab SBCL source tarball, plus one of GNU clisp, ECL, Clozure CL, even GNU Common Lisp at one point, or any of the commercial implementations, including of course CMUCL, and C compiler (for the thin runtime support) and build a complete and unproblematic SBCL release with few commands
Can we get a "(1999)" date on this, please? Only half joking becuase I see Common Lisp and, sure, I upvote ... but honestly, what's the purpose of this HN submission without context?
SBCL is obviously fantastic but let's contrast with another popular implementation: Embeddable Common Lisp. https://ecl.common-lisp.dev/
Top marks for SBCL performance but ECL can be a better fit for embedding into mobile applications, running on lighter weight hardware, and in the browser.
nb: there is a SBCL release at end of every month: https://www.sbcl.org/all-news.html
We upgraded to 2.6.1 about a week ago and switched to using the new(ish) parallel(ish) garbage collector. I still can't tell what the impact has been.
Claude Code (which is a wizard at analyzing log files but also, I fear, an incorrigible curve-fitter) insisted that it was a real breakthrough and an excellent choice! On the other hand there was a major slowdown last night, ending in SBCL dying from heap exhaustion. I haven't had a chance to dig into that yet.
>SBCL dying from heap exhaustion
Due to hitting the cap, or to framentation? My understanding is the new parallel GC compacts the heap rather infrequently.
If by the cap you mean the heap size passed in as the --dynamic-space-size argument, then it definitely didn't hit the cap. It was using about 2/3 of that.
> My understanding is the new parallel GC compacts the heap rather infrequently
Can you explain more?
I'm going to caveat this by stating up front that obviously HN's source code is not public so I don't know what your hot path looks like, and that I'm not a domain expert on garbage collection, but I do write a fair amount of lisp for SBCL.
Immix-style collectors, like the new GC in SBCL, only compact on an opportunistic basis and so you get fragmentation pressure under load. In that situation, you might be well under the dynamic space size cap but if it can't find a large enough contiguous chunk of free heap it will still die.
So, fragmentation would be my prime suspect given what you described.
Sorry for suddenly clinging to you for support but might we be better off using the older GC in that case?
Thanks. Your link gives more insight into "why submit now?" Appreciate it.
My favorite bit of SBCL trivia is the name: this is descended from Carnegie Mellon's build.
Steel. Bank.
I don't get it :-(
I think it's just that Andrew Carnegie made his fortune in the steel industry, and the Mellons made their fortune in banking.
Yes and
Carnegie Technical Schools was founded in 1900 based on a $1m donation from Andrew Carnegie,
Mellon Institute of Industrial Research was originally founded in 1913 by Andrew and Richard Mellon.
Carnegie Mellon was created by combining the two institutions in 1967.
That's very fun and makes so much more sense than my half guess that it was from a defunct regional mid 20th century bank I had never heard of.
What about it?
It's awesome and the lucky 10,000 deserve to be introduced to it?
A well known quantum computing company's entire stack runs on SBCL, with Emacs in production... works really well, don't knock it until you've tried it. Phenomenal REPL.
Would this be the same place that Coalton came out of? (just curious)
It's a recurrent event when someone on HN discovers some well-known piece of technology.
agreed, but "tosh" posted numerous times on SBCL over the last 7 years, so it's a valid question.
https://hn.algolia.com/?q=tosh+sbcl