* "Self-hosted: Runs entirely on your infrastructure. No data leaves your network."
* "Bring Your Own LLM: Anthropic, OpenAI, Gemini, or open-weight models via vLLM."
With so many newbies wanting these kinds of services it might be worth adjusting the first bullet to say: "No data leaves your network, at least as long as you don't use any Anthropic, OpenAI, or Gemini models via the network of course"
Most organizations are going to be self hosting on aws, gcp or azure... So as long as you use their inference services as your LLM then you can keep it all within the private network
Even self-hosting on AWS, GCP, or Azure isn't local enough for certain application, such as people doing export-controlled work where any sysadmin or person with physical access to the server/data is required to be a US Person (or equivalent in other countries). This is the niche that the govcloud solutions are aimed at serving. But some people just want to build big actually-private, actually self-hosted systems and do their own physical and network security.
Are all people with physical access to the servers or network access to the hosts guaranteed to be US persons? Are all physical and network accesses logged for audits? That's the kind of thing govcloud promises that export control auditors want to see.
I felt like "Confidential Compute" tech could solve this issue once and for all but I'm not so sure after seeing some of the attacks people can do with physical access.
Another option of course is to not use cloud at all and have your own rack in a locked room with a good security system and/or armed US person guards.
Exactly, enterprise customers almost always use private model endpoints on their cloud provider for any serious deployments. Data stays within the customer's VPC, data security and privacy is guaranteed by the cloud providers.
(ParadeDB maintainer here). This is super cool. Congrats on the project, and I'm excited to see ParadeDB be used to power this kind of use case. If there's anything else you need to ship Omni, don't hesitate to reach out to me!
One-click deploy with Render, and we're directly in contact with the core team to get it added to their official docs. I hear the PR is up internally :)
How are you managing multiplayer and permissions? I see in the docs that you can add multiple users and that queries are filtered by the requesting user such that the user only sees what they have access to. The docs aren't particularly clear on how this is being accomplished.
Does each user do their own auth and the ingest runs for each user using stored user creds, perhaps deduplicating the data in the index, but storing permissions metadata for query time filtering?
Or is there a single "team" level integration credential that indexes everything in the workspace and separately builds a permissions model based on the ACLs from the source system API?
So it depends on the app - e.g., Google has domain-wide delegation where the workspace admin can provide service account creds that allow us to impersonate all users in the workspace and index all their files/email. During indexing, we determine the users/groups who have permissions file and persist that in the db. (It's not perfect, because Google Drive permission model is a bit complex, but I'm working on it.) This model is much simpler than doing per-user OAuth.
In general, the goal is to use an org-wide installation method wherever possible, and record the identify of the user we are impersonating when ingesting data in the ACL. There are some gaps in the permission-gathering step in some of the connectors, I'm still working on fixing those.
So far both projects are quite similar… the only major difference being the search index. Onyx uses vespa.ai for BM25 and vector search, I decided to go down the Postgres-only route.
Tangential, when you mentioned "Full-text (BM25) and semantic (pgvector) search", what are the significant benefits of the latter? I used to think of BM25 indexes as vectors of documents, which support search, "more like this" etc.
I also started to build something similar for us, as an PoC/alternative to Glean. I'm curious how you handle data isolation, where each user has access to just the messages in their own Slack channels, or Jira tickets from only workspaces they have access to? Managing user mapping was also super painful in AWS Q for Business.
Currently permissions are handled in the app layer - it's simply a WHERE clause filter that restricts access to only those records that the user has read permissions for in the source. But I plan to upgrade this to use RLS in Postgres eventually.
For Slack specifically, right now the connector only indexes public channels. For private channels, I'm still working on full permission inheritance - capturing all channel members, and giving them read permissions to messages indexed from that channel. It's a bit challenging because channel members can change over time, and you'll have to keep permissions updated in real-time.
I've done some RAG using postgres and the vector db extension, look into it if you're doing that type of search; it's certainly simpler than bolting another solution for it.
Yeah, Omni uses Postgres and pgvector for search. ParadeDB is essentially just Postgres with the pgsearch extension that brings in Tantivy, a full-text search engine (like Apache Lucene).
Tangeant: Why is integrating with teams SO difficult?
I started parsing its system logs to create entries in our system automatically to book my times - just not todeal with their silly REST api requirements.
Not yet, there’s a Microsoft connector implementation, but it only does Sharepoint, OneDrive, Outlook etc. and I haven’t tested it thoroughly yet. Teams required some special setup to work IIRC, so I skipped it. Will keep it on the roadmap though!
Typical RAG implementations I’ve seen take the user query and directly run it against the full-text search and embedding indexes. This produces sub-par results because the query embedding doesn’t really capture fully what the user is really looking for.
A better solution is to send the user query to the LLM, and let it construct and run queries against the index via tool calling. Nothing too ground-breaking tbh, pretty much every AI search agent does this now. But it produces much better results.
Oops, sorry! That page is still a WIP, haven't pushed it yet. The plan was to expose the main search and chat APIs so that users can build integrations with third-party messaging apps (e.g. Slack), but haven't gotten around to properly documenting all the APIs yet.
Can we please not change the meaning of chat to mean agent interface? It was painful to see crypto suddenly meaning token instead if cryptography. Plus i really dont want to “chat” with ai. its a textual interface
Fair point, although I think we have OpenAI to blame for that - for buying chat.com and pointing it to the most popular textual AI interface of them all :)
I've done small scale experiments with up to 100-500k rows, and did not notice any significant degradation in search query latency - p95 still well under 1s.
I haven't directly compared against Elasticsearch yet, but I plan to do that next and publish some numbers. There's a benchmark harness setup already: https://github.com/getomnico/omni/tree/master/benchmarks, but there's a couple issues with it right now that I need to address first before I do a large scale run (the ParadeDB index settings need some tuning).
we have a pretty intensively used postgres backed app handling thousands of users concurrently. After 6 years and thousands of paying custoners, we are only now approaching to the limits of what it can support on the horizon. TLDR: when you get there, you can hire some people to help you break things off as needed. if you're still trying to prove your business model and carve yoruself a segment of the market, just use postgres
Thanks for sharing! Big part of the reason why I decided on postgres, everything I've read about people using it in prod tells me that most organizations never really grow beyond requiring anything more than what it offers.
Most of the time just re-casting what you want in a horizontally shardable way is the "right" way to do it with any rdbms (if you scale) but at this point you can get boxes on AWS with 32TiB of ram, and most organizations don't have that much total data across their entire suite of stuff (many do, most don't.)
* "Self-hosted: Runs entirely on your infrastructure. No data leaves your network."
* "Bring Your Own LLM: Anthropic, OpenAI, Gemini, or open-weight models via vLLM."
With so many newbies wanting these kinds of services it might be worth adjusting the first bullet to say: "No data leaves your network, at least as long as you don't use any Anthropic, OpenAI, or Gemini models via the network of course"
That's a good point, it might make sense to clarify that for individuals who want to self-host. I'll make the change, thanks!
Most organizations are going to be self hosting on aws, gcp or azure... So as long as you use their inference services as your LLM then you can keep it all within the private network
Even self-hosting on AWS, GCP, or Azure isn't local enough for certain application, such as people doing export-controlled work where any sysadmin or person with physical access to the server/data is required to be a US Person (or equivalent in other countries). This is the niche that the govcloud solutions are aimed at serving. But some people just want to build big actually-private, actually self-hosted systems and do their own physical and network security.
AWS Bedrock seems to say the inference code is only scanned for CASM and no one trains on your data.
Are all people with physical access to the servers or network access to the hosts guaranteed to be US persons? Are all physical and network accesses logged for audits? That's the kind of thing govcloud promises that export control auditors want to see.
I felt like "Confidential Compute" tech could solve this issue once and for all but I'm not so sure after seeing some of the attacks people can do with physical access.
Another option of course is to not use cloud at all and have your own rack in a locked room with a good security system and/or armed US person guards.
Exactly, enterprise customers almost always use private model endpoints on their cloud provider for any serious deployments. Data stays within the customer's VPC, data security and privacy is guaranteed by the cloud providers.
(ParadeDB maintainer here). This is super cool. Congrats on the project, and I'm excited to see ParadeDB be used to power this kind of use case. If there's anything else you need to ship Omni, don't hesitate to reach out to me!
This is a good time to be offering hybrid search extensions. I just did that myself recently with pgvector for a documentation site.
Does ParadeDB work with Render? They seem to have a whitelist of extensions https://render.com/docs/postgresql-extensions
We just made a blueprint for it! https://github.com/paradedb/render-blueprint
One-click deploy with Render, and we're directly in contact with the core team to get it added to their official docs. I hear the PR is up internally :)
Sweet, nice work
Thanks Philippe! You guys have been super helpful on slack!
Anytime! We have some vector search work coming in the next few weeks/months that I expect you'll find interesting. Stay tuned :)
How are you managing multiplayer and permissions? I see in the docs that you can add multiple users and that queries are filtered by the requesting user such that the user only sees what they have access to. The docs aren't particularly clear on how this is being accomplished.
Does each user do their own auth and the ingest runs for each user using stored user creds, perhaps deduplicating the data in the index, but storing permissions metadata for query time filtering?
Or is there a single "team" level integration credential that indexes everything in the workspace and separately builds a permissions model based on the ACLs from the source system API?
So it depends on the app - e.g., Google has domain-wide delegation where the workspace admin can provide service account creds that allow us to impersonate all users in the workspace and index all their files/email. During indexing, we determine the users/groups who have permissions file and persist that in the db. (It's not perfect, because Google Drive permission model is a bit complex, but I'm working on it.) This model is much simpler than doing per-user OAuth.
In general, the goal is to use an org-wide installation method wherever possible, and record the identify of the user we are impersonating when ingesting data in the ACL. There are some gaps in the permission-gathering step in some of the connectors, I'm still working on fixing those.
How does it compare to Onyx (rebranded from Danswer, with more chat focus, while Danswer was more RAG focus on company docs/comms)?
- https://onyx.app/
- Their rebranded Onyx launch: https://news.ycombinator.com/item?id=46045987
- Their orignal Danswer launch: https://news.ycombinator.com/item?id=36667374
So far both projects are quite similar… the only major difference being the search index. Onyx uses vespa.ai for BM25 and vector search, I decided to go down the Postgres-only route.
Tangential, when you mentioned "Full-text (BM25) and semantic (pgvector) search", what are the significant benefits of the latter? I used to think of BM25 indexes as vectors of documents, which support search, "more like this" etc.
Interesting!
I also started to build something similar for us, as an PoC/alternative to Glean. I'm curious how you handle data isolation, where each user has access to just the messages in their own Slack channels, or Jira tickets from only workspaces they have access to? Managing user mapping was also super painful in AWS Q for Business.
Thank you!
Currently permissions are handled in the app layer - it's simply a WHERE clause filter that restricts access to only those records that the user has read permissions for in the source. But I plan to upgrade this to use RLS in Postgres eventually.
For Slack specifically, right now the connector only indexes public channels. For private channels, I'm still working on full permission inheritance - capturing all channel members, and giving them read permissions to messages indexed from that channel. It's a bit challenging because channel members can change over time, and you'll have to keep permissions updated in real-time.
When a product headlines what it’s built with, it’s does not bode well for customer success.
I've done some RAG using postgres and the vector db extension, look into it if you're doing that type of search; it's certainly simpler than bolting another solution for it.
Yeah, Omni uses Postgres and pgvector for search. ParadeDB is essentially just Postgres with the pgsearch extension that brings in Tantivy, a full-text search engine (like Apache Lucene).
Can it connect to Teams?
Tangeant: Why is integrating with teams SO difficult?
I started parsing its system logs to create entries in our system automatically to book my times - just not todeal with their silly REST api requirements.
Not yet, there’s a Microsoft connector implementation, but it only does Sharepoint, OneDrive, Outlook etc. and I haven’t tested it thoroughly yet. Teams required some special setup to work IIRC, so I skipped it. Will keep it on the roadmap though!
Nice! Could you elaborate on "not just a basic RAG"?
Thank you!
Typical RAG implementations I’ve seen take the user query and directly run it against the full-text search and embedding indexes. This produces sub-par results because the query embedding doesn’t really capture fully what the user is really looking for.
A better solution is to send the user query to the LLM, and let it construct and run queries against the index via tool calling. Nothing too ground-breaking tbh, pretty much every AI search agent does this now. But it produces much better results.
I call this ralphgrep
Multiple pages link to a `API Reference` that returns a 404
Oops, sorry! That page is still a WIP, haven't pushed it yet. The plan was to expose the main search and chat APIs so that users can build integrations with third-party messaging apps (e.g. Slack), but haven't gotten around to properly documenting all the APIs yet.
Can we please not change the meaning of chat to mean agent interface? It was painful to see crypto suddenly meaning token instead if cryptography. Plus i really dont want to “chat” with ai. its a textual interface
Fair point, although I think we have OpenAI to blame for that - for buying chat.com and pointing it to the most popular textual AI interface of them all :)
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[flagged]
I've done small scale experiments with up to 100-500k rows, and did not notice any significant degradation in search query latency - p95 still well under 1s.
I haven't directly compared against Elasticsearch yet, but I plan to do that next and publish some numbers. There's a benchmark harness setup already: https://github.com/getomnico/omni/tree/master/benchmarks, but there's a couple issues with it right now that I need to address first before I do a large scale run (the ParadeDB index settings need some tuning).
we have a pretty intensively used postgres backed app handling thousands of users concurrently. After 6 years and thousands of paying custoners, we are only now approaching to the limits of what it can support on the horizon. TLDR: when you get there, you can hire some people to help you break things off as needed. if you're still trying to prove your business model and carve yoruself a segment of the market, just use postgres
Thanks for sharing! Big part of the reason why I decided on postgres, everything I've read about people using it in prod tells me that most organizations never really grow beyond requiring anything more than what it offers.
Most of the time just re-casting what you want in a horizontally shardable way is the "right" way to do it with any rdbms (if you scale) but at this point you can get boxes on AWS with 32TiB of ram, and most organizations don't have that much total data across their entire suite of stuff (many do, most don't.)
LLM bot.
lol
another AI post...
[dead]
is this AI?
has to be