are you asking what moat you can establish around a product that uses customer-facing LLM inference?
IMO, it's incredible the ability to build things rapidly, solo, and extremely cheaply. So as long as that's the case, the only moat left for you is customer access, customer loyalty, and pricing (though I think that last one is arguable)
The challenge as I see it is that for new products in this space to be successful, the solo dev either needs to BE or be partner with someone who has deep intimate domain knowledge and large network for the pain point their product solves (e.g. a 15 year supply chain manager for large manufacturing company rolls into solo dev a meaningful solution to a problem in their domain) or else the solo dev needs to already have a large social reach they can leverage to access customers.
Those are the kinds of moats I see being most effective.
By moat I just mean something a solo dev can build/pursue that doesn't get absorbed by vLLM/SGLang in few months. I'm fairly new to this space so not trying to get too deep into kernels or hardware.
Something around inference tooling: benchmarking or eval/regression checks for quantized models or something else. Something that's useful as a standalone tool/project or to learn the space properly. Just curious where there's still some opportunities.
> can build/pursue that doesn't get absorbed by vLLM/SGLang in few months.
That's the rub isn't it. The real risk as exposed by the latest Navier-tokes situation is that they cannot fully guarantee that math solution producing model didn't use the work-in-progress of the researchers who were interacting with the system to then beat the researchers to the published solution. Can any of us expect that these models won't use our own work-in-progress either internally or indirectly to overcome any technical 'moat' we might imagine to exist? Sure, there is a setting for "do not train" but what guarantees can exist given the demonstrated failure to constrain models from engaging in other deceptive actions on their own or offsite systems.
It would seem that where a 'moat' might still exist in code is probably exactly near to kernels or hardware, or in a business aspect which is orthogonal to the actual code. As previously stated; the understanding of the pain point for which the code is a solution, the depth of relationships within the industry or consumers who are potential users of the code or the outputs of that code, the ability to gain exclusive access to some actionable inputs which no matter what other code is generated to address the pain point or what scale of consumer awareness might arise the outputs will remain unique, are pretty much the core moats which remain going forward.
The standalone tool/product which is a learning project doesn't appear to have readiness to access two of those three moat types, so probably not much opportunity left for learning a way to the top that is durable if you're planning on building with or competing with LLM generated codebases.
I guess another kind of moat is, a tool which makes your own internal process much more manageable/controllable/effective and gives whatever else you do some kind of capability that you could not access or get for less than or equal to the total cost (tokens and opportunity costs) of making that tool. This might not be a 'real' moat by the definition you are asking about, but if it gives you more capability than others working in the same problem space, it's still a moat and so long as you manage not to expose 'how the sausage gets made' it could be yours. But maybe I'm just leaning to much in the direction of a Dark Forest theory.
are you asking what moat you can establish around a product that uses customer-facing LLM inference?
IMO, it's incredible the ability to build things rapidly, solo, and extremely cheaply. So as long as that's the case, the only moat left for you is customer access, customer loyalty, and pricing (though I think that last one is arguable)
The challenge as I see it is that for new products in this space to be successful, the solo dev either needs to BE or be partner with someone who has deep intimate domain knowledge and large network for the pain point their product solves (e.g. a 15 year supply chain manager for large manufacturing company rolls into solo dev a meaningful solution to a problem in their domain) or else the solo dev needs to already have a large social reach they can leverage to access customers.
Those are the kinds of moats I see being most effective.
Define a "Moat in LLM Inference".
By moat I just mean something a solo dev can build/pursue that doesn't get absorbed by vLLM/SGLang in few months. I'm fairly new to this space so not trying to get too deep into kernels or hardware.
Something around inference tooling: benchmarking or eval/regression checks for quantized models or something else. Something that's useful as a standalone tool/project or to learn the space properly. Just curious where there's still some opportunities.
> can build/pursue that doesn't get absorbed by vLLM/SGLang in few months.
That's the rub isn't it. The real risk as exposed by the latest Navier-tokes situation is that they cannot fully guarantee that math solution producing model didn't use the work-in-progress of the researchers who were interacting with the system to then beat the researchers to the published solution. Can any of us expect that these models won't use our own work-in-progress either internally or indirectly to overcome any technical 'moat' we might imagine to exist? Sure, there is a setting for "do not train" but what guarantees can exist given the demonstrated failure to constrain models from engaging in other deceptive actions on their own or offsite systems.
It would seem that where a 'moat' might still exist in code is probably exactly near to kernels or hardware, or in a business aspect which is orthogonal to the actual code. As previously stated; the understanding of the pain point for which the code is a solution, the depth of relationships within the industry or consumers who are potential users of the code or the outputs of that code, the ability to gain exclusive access to some actionable inputs which no matter what other code is generated to address the pain point or what scale of consumer awareness might arise the outputs will remain unique, are pretty much the core moats which remain going forward.
The standalone tool/product which is a learning project doesn't appear to have readiness to access two of those three moat types, so probably not much opportunity left for learning a way to the top that is durable if you're planning on building with or competing with LLM generated codebases.
I guess another kind of moat is, a tool which makes your own internal process much more manageable/controllable/effective and gives whatever else you do some kind of capability that you could not access or get for less than or equal to the total cost (tokens and opportunity costs) of making that tool. This might not be a 'real' moat by the definition you are asking about, but if it gives you more capability than others working in the same problem space, it's still a moat and so long as you manage not to expose 'how the sausage gets made' it could be yours. But maybe I'm just leaning to much in the direction of a Dark Forest theory.