I did use default settings on those tools. I'd agree it wasn't the best benchmark and certainly wouldn't hold up if the goal here was to break any records in compression. The tools and settings chosen were mostly to validate 'this ended up roughly in the same ballpark as other tools' over 'Claude burned a lot of tokens and still made a terrible compression tool'.
That is a very generous interpretation of that absolute stinker of a tweet, to the point that it's basically a mischaracterization.
Hashimoto there was entirely too busy tooting his own horn and parroting his latest favorite phrase (AI psychosis), rather than making any actually coherent points.
He pitches that people should "just be better programmers" en masse all of a sudden, instead of letting LLMs optimize their code. Not unlike the beyond delusional parallel reality that people coping about GLP-1 inhibitors are living in. And he explicitly prefers this fantasy over the very reality that he directly observed, refusing to grapple with the fact that deep expertise and care is rare, and forever will be.
It's safe to conclude then that the psychosis he's talking about is squarely within the boundaries of his head. Further evidence in the way of this is him auto-fellatiously tweeting that he reviews the code he generates (wow, what a hero!), and him indiscriminately performing prompt injection attacks against those cloning his repositories, taking great pride and obnoxious schadenfreude in this along the way.
Never aspired to be an open source maintainer, but if I ever mistakenly will, I do hope I will recall this trashfire of an unraveling for sure. He's going full speed ahead against the windmills, while ThePrimeagen and his legions of very much flesh-and-blood sycophants are fanning him on throughout his journey.
reading that tweet it came across to me that he set the agent up for failure. garbage in, garbage out! maybe a more fair comparison would have been a prompt to design & implement the renderer with a clean start. throwing it all out and starting from scratch is something these models, and people, are explicitly trained against doing
other than that, I don't think any serious person would trust a ralph-loop to accomplish something truly substantial. it's good for ensuring rote tasks get completed without the agent stopping too soon, but not much more
not the greatest benchmark, bzip2 many times beats xz, gzip is competitive with zstd
maybe it used default compression levels, and the defaults are not "matched"
fumny because author also wrote this post:
https://www.elliotcsmith.com/how-to-avoid-picking-terrible-m...
I did use default settings on those tools. I'd agree it wasn't the best benchmark and certainly wouldn't hold up if the goal here was to break any records in compression. The tools and settings chosen were mostly to validate 'this ended up roughly in the same ballpark as other tools' over 'Claude burned a lot of tokens and still made a terrible compression tool'.
[flagged]
> The choice of objective function is key (...) A phenomenon recently captured by Mitchell Hashimoto in this X thread: https://xcancel.com/mitchellh/status/2060088112257372610
That is a very generous interpretation of that absolute stinker of a tweet, to the point that it's basically a mischaracterization.
Hashimoto there was entirely too busy tooting his own horn and parroting his latest favorite phrase (AI psychosis), rather than making any actually coherent points.
He pitches that people should "just be better programmers" en masse all of a sudden, instead of letting LLMs optimize their code. Not unlike the beyond delusional parallel reality that people coping about GLP-1 inhibitors are living in. And he explicitly prefers this fantasy over the very reality that he directly observed, refusing to grapple with the fact that deep expertise and care is rare, and forever will be.
It's safe to conclude then that the psychosis he's talking about is squarely within the boundaries of his head. Further evidence in the way of this is him auto-fellatiously tweeting that he reviews the code he generates (wow, what a hero!), and him indiscriminately performing prompt injection attacks against those cloning his repositories, taking great pride and obnoxious schadenfreude in this along the way.
Never aspired to be an open source maintainer, but if I ever mistakenly will, I do hope I will recall this trashfire of an unraveling for sure. He's going full speed ahead against the windmills, while ThePrimeagen and his legions of very much flesh-and-blood sycophants are fanning him on throughout his journey.
reading that tweet it came across to me that he set the agent up for failure. garbage in, garbage out! maybe a more fair comparison would have been a prompt to design & implement the renderer with a clean start. throwing it all out and starting from scratch is something these models, and people, are explicitly trained against doing
other than that, I don't think any serious person would trust a ralph-loop to accomplish something truly substantial. it's good for ensuring rote tasks get completed without the agent stopping too soon, but not much more
I guess it depends on what your definition of truly substantial is.
With the right objective function and enough compute I think you can definitely make progress. A lot of problems can be framed as search.
I think some of these proofs coming out is a good example of this.