As a proof-of-concept, I've been using extra tokens to formalize the IRS tax forms and worksheets, mostly just to prove this can be done. The idea is that something like this could be adapted into a publicly maintained version of the IRS Direct File program (RIP). I'm up to an estimated 60% coverage of US federal tax situations so far.
What do you mean when you say 'formalize'? I mean, a full tax document is just each form with each 'line' as an entry by itself and then at the end, all the calculations get performed against that.
Not quite. The formalization covers the IRS's published procedure, i.e. the questions and calculations that go into the published forms, worksheets, and instructions. It's not a formalization of the tax code itself, and it's not a tax calculator, though a client of the underlying code could function as a tax calculator. (The project includes such a client, a CLI tax interviewer, just to exercise the code.)
In more detail:
- every line of every form gets a YAML representation which includes the source text, a data type, and an expression built from a small set of lean atoms, plus a citation pointing to the exact location of the rule in the original official document.
- underneath the yaml layer there's a lean formalization of the atoms showing that they behave as specified, that all comuptations use only the allowed atoms and operations, etc. There's also a typescript evaluator that's used to cross check the lean for agreement, so we have the arithmetic represented two ways.
The critical failure point for an LLM-produced formalization like this is the interpretation of the form text. Since I'm not a tax lawyer and don't intend to become one, this is an explicitly owned weakness and one I've structured the project around. Publicly available tax calculators (PolicyEngine, UsTaxes, OpenTaxSolver, etc.) are used to cross check derived numbers from this system. Mutation testing ensures the test suite is robust against bugs and arithmetic errors. Interpretations are cross checked by different model families (Claude, GPT, Deepseek). Etc.
The idea is to derive confidence from the formal properties of the system itself, and to allow theoretical future users of such a system to derive confidence in the interpretation of specific forms by making it easy to cross check line-level formalizations against the source documents.
There's a lot more detail in the (alas, Claude-authored) README. When I have time I'm going to write my own version of the README expressing my vision of the project, but I'm not there yet.
Some of the interpretations won't be correct without grounding them in old case law. That's the expensive part, in many jurisdictions, the official interpretations by the tax department are not free.
Sort of. The formulas for each line live in the form YAML I described above. The formulas are built out of a set of primitive operations that are defined in lean, with their properties proved there (non-negativity, monotonicity, etc.). There's also a lean evaluator that's run on compiled form data and cross checked with the typescript evaluator. (There's a lot more detail in the README.)
When I was planning the project, I looked into FactGraph to see if it'd be worth building on. I opted not to for a few reasons:
- It's oriented toward the IRS's internal XML API in a way that's not really useful for this project, since no consumer of my code is going to digitally submit to the IRS.
- The semantics of FactGraph are largely undefined, and the maintainers themselves seem to acknowledge that its development trajectory went off in a bad direction and it needs major divisions.
- It's super verbose.
- It's the year of Lean, so I wanted to find an excuse to use Lean.
Other work has been done in this domain, though. The biggest example is Catala[0], though Prof. Sarah Lawsky at Northwestern seems to have made some strides toward formalizing the Internal Revenue Code.
I recently did something similar, using Claude Code to formalize a set of rules for a sporting event as part of building a computer-based scoring system. It's worked very well.
I'm not sure it matters. The IRS killed Direct File under lobbying pressure from the tax prep industry. In my opinion the public is owed a free version of this sort of tool. If the IRS doesn't produce it, then citizens ought to. If Google or Anthropic or whoever wants to offer a free version of this, more power to them, but I don't see why they'd bother. They could have done it decades ago.
And to be clear: my vision here isn't "have an agent do your taxes", it's to have agents build a hardened and thoroughly verified deterministic tool that can help you navigate the (somewhat bewildering) forms and instructions decision tree when filing your own taxes.
This is great, and I appreciate the approach. Yes, tax calculation should be a decision tree. Unfortunately it is truly pretty complicated (just this year I had to abandon FreeTaxUSA for TurboTax because the latter handled an important case I needed better). But I have a lot of faith that we can do this right. Good luck!
Funny, after 10+ years of using TurboTax Online I moved to FreeTaxUSA last year and it's like a breath of fresh air. TurboTax couldn't do what I needed to do (e-file a late return) and FreeTaxUSA had no problem with doing that.
I have a probably stupid question: given that the IRS killed Direct File, even if your software can create ready-to-file (or even ones that need some work) tax returns, isn't that moot without a way to like... get the IRS to accept them?
Why wouldn't the IRS accept a tax return created by your own software? How would they even know it was custom software? Ultimately your tax return is just a stack of papers that gets sent to the IRS.
Software sending efiles to the IRS has to be tested with and approved by the IRS. The IRS is working toward eliminating paper filing as an option. At this point, paid preparers are legally required to efile all returns and if the return gets rejected and they must paper file, they have to include Form 8948, Explanation for Paper Filing. For the IRS to accept a tax return from software the software has to go through this process.
https://www.irs.gov/e-file-providers/how-tax-preparation-sof...
It is truly insane. Major corporations spent billions to prevent this. The worst part is that for the majority of Americans, the government has already internally prepared their tax return before they even file but again the tax preparation industry spends a fortune lobbying Congress to prevent an automatic filing of taxes.
I think you meant to say they spent billions to create this?
In either case the tax prep industry is a pox on our society. But of course this is America, and replacing a perfectly functioning publicly accessible service with a nightmare of private sector bullshit is the most violently American shit ever.
As an American, it is bonkers to me too. Last year my former employer did not issue me a w-2. I called the IRS to ask them what to do. They said they had all the information but can’t release it to me until after the filing deadline. Instead I was told to use form 4852 to estimate and file an amended return if my estimate was wrong once they give me the numbers they have. Worst part is you can’t efile when you file with form 4852.
Anyway, my point is that for 95% of people they have all the information already and you shouldn’t have to file.
This! I am CPA with a tax practice and it makes me unreasonably angry that the tax prep industrial complex has lobbied our government into making the vast majority of people stress over their tax returns or have to pay someone to prepare a return that is already sitting on an IRS computer. No CPA worth their salt wants to do the simple returns but companies like Intuit and HR Block feed off them.
You can use free file fillable forms to submit online. It's a bit of work to transcribe from other software. It also requires a particular interpretation when filling forms (e.g. you need to round before summing your W-2s).
I prepared my taxes this year using Astra + Codex. It did a thorough job on a complicated return, and I sent it to my accountant to fill out the tax form, with everything laid out in a spreadsheet. They sent it back for final review, and I had Codex check it; it was riddled with errors. The refund was nearly $2k lower than it should have been, and they put a bunch of stuff into the wrong fields. Had to go back and forth multiple times to get it 100% correct. Made me wonder how many previous years' returns were wrong.
I feel like we're going to get to a point where it's mandatory to use AI for certain tasks - humans just can't be trusted to get things right that require a level of precision.
I sent my business and personal returns through Claude this year before signing, and it turns out they were a horror show of unforgivably stupid mistakes. Pretty infuriating considering what my now former CPA was charging me.
He amortized the goodwill of businesses I acquired over 5 years instead of 15 (incredibly basic stuff, which Claude flagged with *CRITICAL ERROR*). When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it. I don't know if there's a good response to making that kind of a mistake, but if there is, it's definitely not that one.
Your anecdote seems to clash heavily with the results of the benchmark announced in the article. I suppose that is part of the jagged experience of agentic AI
The benchmark announced in the article seems to be around weird edge cases (which are often important) and whether it can prompt the person with relevant information rather than just listening to what they want to do.
Top tier tax professionals can do pretty well on this benchmark (better than agents) but won't hit 100%.
The kind of people spending time preparing oodles of returns would not do very well on this metric, though (especially midway through that pile of returns, as opposed to "think carefully about this trick question").
The OP said it was a complicated return so anyone assuming OPs anecdote is in line with the article is filling the lines in themselves with a lot of assumptions
I have a non-trivial tax situation, as a US citizen living abroad. I prepare my own tax returns, never having found a cross-border tax preparer I trust, and have for the past few years, fed my draft return along with all the supporting paperwork into Claude. It has:
1. found errors, e.g. where I used the incorrect exchange rate (and managed to work out how I had made that error, which I was impressed by)
2. found items which would appear to be errors except for context it did not know
3. flagged areas where it disagreed with me as to the correct tax treatment, but accepted that the law is ambiguous (and in some cases I preferred its interpretation after additional research and in other cases I did not)
4. flagged areas where it disagreed with IRS guidance, and only acknowledged said guidance after I provided links to it
5. provided highly inaccurate estimated tax calculations, and not been able to fix its estimates even when challenged
So my takeaway is that it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check. (Possibly it is if you just have a W-2 and no other substantial income, but I can't speak for that scenario.)
I have a company abroad in addition to living abroad. It’s kind of a nightmare to do my US taxes and costs me easily $2000 per year right now. I was hoping I could have an agent do them after I have a few years of data and point it at the source statements and the end result forms
I also had a great experience consulting LLM regarding registering a business as a foreigner. Except I'm still very sceptical since it's not my area of expertise and consequences for a mistake are very high.
I'm also in a non-trivial tax situation, GPT Sol did better than my specialist accountant this year. It needed a lot of steering from me and about 2B tokens, but that is better than spending $3-4k in preparation fees. Total time to prepare and handle steering was still less than I normally spend on tax prep with my accountant.
I think what helped is giving it 100+ pdf and csv documents of my full situation so it knew everything about me and could cross-reference with all my bank statements.
So, GPT Sol outputted tax return PDFs that you were able to submit to the IRS? (I'd imagine that you couldn't directly e-file like you could from TurboTax.)
Correct, it first did it in a spreadsheet, then filled out 59 pages of returns and supplement forms and I printed and posted it. It also filed my FBAR.
I don't know that I'd trust it with an FBAR given it's an interactive PDF myself, but good to hear that it was able to do it - maybe I'll get fed up doing the FBAR manually one year and trust an agent to do so.
Ah looks like there's both the PDF and online option; I filed with the PDF version because you can save your history but I guess if Claude can navigate and fill in the online option then I have less quibble about being able to keep progress. Thanks!
Yeah for individual tax preparation that is pretty high. Most people get away with filing for $20-150. I used to have a CPA do my consulting business, tiny 1 person company, and personal taxes for $500.
I no longer do consulting on the side so I just use online tax software. It is about $150 but I do that more out of laziness as it integrates with my brokerage and payroll company to import everything automatically. Also I have carry forward losses and such that it already knows about.
Most people’s taxes are simpler than mine but I am sure there are lots of folks here with much more complex tax situations
I am not a normal case because I don't live in USA. The USA government estimates 50-80 hours to complete all my forms.
I have foreign income and foreign businesses, my partner has foreign income from two countries. The tax law is then based on tax treaties, most tax preparers do not have sufficient knowledge to complete a return, and tax software doesn't handle these cases.
The tax treaties are not clear, tax professionals will disagree on interpretation, and tax benefits in one country will not apply to another.
Then there is a thing called AMT, alternative minimum tax, which is a second way to calculate your taxes that you need to submit as well.
> I have a non-trivial tax situation, as a US citizen living abroad.
Same situation as you. I have a local accountant in the country I'm in plus my US accountant. Claude/ChatGPT have been a huge help for checking, explaining, and finding issues. They also found websites with pertinent information that Google did not, which surprised me.
Hacker's breach Anthropic and dump session logs; they have your software designs, tax records, draft business proposals -- Claude doesn't get all of your information, but it helps with deep details on a significant cross section of it. How do you mitigate this loss?
Try this, it uses a tax engine as the backbone and provides feedback for Claude if it goes off the rails. Albeit it doesn't support every tax scenario.
I appreciate the recommendation, but as it happens I have already solved the estimated issue (mostly by just preparing a full return with estimated next year's figures, which while time-heavy is going to be more accurate). And for what it's worth, I suspect it wouldn't work for me - cross-border is extraordinarily complicated because of the tax treaties (indeed there is a dedicated IRS form to disclose the effect and rationale of any treaty based position), and given there is no obvious note on the documentation for it that there is cross-border support I doubt there is.
(And as I said to another commenter, I think it would be advisable in this thread for people to expressly acknowledge or disclaim any interest in or prior experience with any particular product)
Are you saying I should use this (or it looks like one of their portfolio companies) as my return preparer, or as a replacement for Claude my return-checker? If the former, I would want to know why I should trust it; if the latter, I would want to know why I should pay for their services given I am already playing Anthropic for a subscription plan anyway.
(It would also help to know if you had a financial interest in or prior experience with this company)
My tax accountant mis-filed my taxes a couple years ago, which cost me about $20K (which I learned after I switched to another accountant later). Don't know if Claude / GPT would make the same mistake or not given SOTA today, but feels like there's an opportunity for models to be saving me thousands or tens of thousands of dollars a year.
>mis-filed my taxes a couple years ago, which cost me about $20K
What do you mean by "cost you $20K"? That your tax liability was calculated by the accountant to be higher than it actually should have been? Or that the accountant calculated your liability too low, which was caught and billed by the IRS?
In the first case, you can still file for a refund within 3 years (I'm omitting some technical details), but in the latter case, as mentioned by others, you still owe the tax whether it was correctly reported on your original return or not.
Years ago I had a similar issue. Accountant did something incorrect, increasing my liability by around $50k. Tax date came so I paid it, but moved accountants. New person fixed the issues, refiled, but it took a long time to get the refund back. IIRC, the IRS did include some nominal interest for the time they had the money.
I'm also 0 for 4 on tax preparers / tax accountants, so I concur with you its best to just do it yourself, which is what I now do. Only time I ever got audited (and failed) was when I had employed an accountant, since then I've never had a problem including passing audits. The tax code isn't /that/ complicated for most people's situation, and it's well worth understanding it.
I'm 1 for 6. And my current firm is overworked. I'm moving to do it myself (three companies, fund admin and personal). Recently Found a Tax-Accountant, not a CPA, who does advice only, no paperwork, who I'm using to cross-check my AI tools.
I am a CPA with 20+ years of experience and I feel like I need to defend our industry here.
There is no such thing as "failing" an IRS audit. You either under reported income, overstated deductible expenses, or your "accountant" changed the numbers you provided and committed fraud, which would have resulted in immediate legal proceedings by the IRS against the preparer. So did you "fail" because your preparer committed federal tax fraud or did you underreport your income?
The statement that the one time you used an accountant you got audited implies that you are confusing causation with correlation.
There are about 6 ways that the IRS selects returns for audit but only 3 are actually relevant here.
- Information Matching (ie. if your W2 does not match the W2 submitted by your employer). This mostly used for your basic W2, 1099DIV, etc. kind of taxpayer and is usually just a letter from the IRS telling you your numbers are wrong and you need to amend your return.
- If they are examining a related return (ie. you own a business and another partner in that business is being audited)
- Computer scoring which breaks down into the Discriminant Function System (DIF), basically, how does your return compare to similar returns, and Unreported Income DIF, what is the potential that your return underreported income. This is what is commonly used for businesses and self employed individuals' Schedule C. Basically, when you prepare a business return (1040 Sch. C, 1041, 1065, 1120, 1120S, or 990) you have to report your business industry's NAICS code, and provide a single line explanation what your business does. Using this and the return data, the IRS is able to score a return on the likelihood that profit is understated by comparing a businesses income and expenses to the norm for a similarly sized businesses in that industry. High scoring returns are then reviewed by IRS personnel.
One of those three things happened to you and none of them are the fault of a preparer that is using accurate information provided by the taxpayer.
Finally, you keep saying things like "accountant". You disparaging an entire industry without saying what type of tax preparer you used. Was it a CPA, licensed by a state with experience in tax? An Enrolled Agent, licensed by the IRS directly? Or a tax preparer, an unlicensed person preparing your return either on their own or through something like HR Block or Turbotax? Because if you used an unlicensed tax preparer and are blaming licensed individuals that is like hiring an unlicensed electricians and then when your house burns down from an electrical fire, turning around and saying all electricians are bad
Additionally, "the tax code isn't /that/ complicated"... tax law is so complex that it is the only form of US law in which ignorance of the law is an acceptable legal defense.
And that's true if all you do is have W-2 income, and RSUs/ESPPs. ESPPs were the most complex thing I had to deal with until I stupidly decided to do things and get K-1s.
Fair point, but the actual "tax code" (Title 26 of the US Code) is extremely complex for most people to understand. If he is referring to IRS Form Instructions as the tax code, yes those are usually not very complex. From my perspective K1s, RSUs, & ESPPs are as simple as a W2 with state tax withholding and I think that is why I am struggling with so many of the comments on this post - I simply can't relate.
> From my perspective K1s, RSUs, & ESPPs are as simple as a W2 with state tax withholding and I think that is why I am struggling with so many of the comments on this post - I simply can't relate.
Well, you are an accountant :-)
With my K1, the issue was that it was for a real estate investment, and you have to be careful about things like bonus depreciation/cost segregation etc. One tax accountant I used was not carrying over the loss (or perhaps not applying the prior year's loss, or something like that). Another accountant was deducting that loss from my W-2 income, which I can't do as I'm not a real estate professional.
When I finally brought these points up and got all these corrected, I studied the final result and then continued doing them myself. But it was a pain getting the online tax SW (TurboTax, TaxAct, etc) to mimic what they did. I had to be really careful what things I said "Yes" to and what I didn't. I would never have figured it out on my own by reading the relevant tax publications (which I did peruse).
And then later, I formed a partnership with a friend so we could both invest in something. Now I also have to issue K-1s! Not fun! RSUs are much easier.
> There is no such thing as "failing" an IRS audit.
I consider being audited and the IRS finding corrections to prior returns as a "failure". Once I started doing things myself, I was audited and no corrections found (e.g. a "success"). In the case where I "failed" the audit, it had to do with how expenses were being handled by the accountant. At the time I was a contractor working on a 1099 and was tracking and submitting some business expenses for reimbursement, and some expenses were tracked and taken against my taxes but not reimbursed. What was filed did not align exactly with what was reimbursed, which caused an overstatement of deductions, causing me to owe back taxes. I am not making any claim that they committed fraud, as that's a crime which requires intent, I think they were just incompetent.
> The statement that the one time you used an accountant you got audited implies that you are confusing causation with correlation.
I actually didn't say that, I said the only time I got audited and failed was when I used an accountant, however I realize my style of writing made that unclear. During the time where I received income via a 1099 (and for part of that also had to file from a K-1 briefly for an LLC), I was audited three times in total. Only one time did I "fail", and it was the only time where I had professional assistance. In fact, I found the process of being audited surprisingly kind, the IRS folks involved were very professional and weren't mean about it at all. After my first experience being audited, even though I got the nasty surprise of owing back taxes, I was a lot less fearful of that process. That said, I have never had any understanding of how someone is chosen to be audited, but I have not been selected for an audit since moving to a W-2 for all of my income, which has been quite some time now. The experience of being audited did get me into the habit of storing all my tax related records carefully organized by year long-term, however, which I think is probably a good habit for everyone.
> You disparaging an entire industry without saying what type of tax preparer you used.
My intent isn't to disparage anyone, it's to share my own personal experience. I hired a CPA licensed in the state of Missouri (where I resided at the time) when I previously had a professional handle my taxes.
Someone still needs to take liability for mistakes.
The real issue is that accountant most likely has a track record of those kinds of mistakes - but you have no info on other clients. So you weren’t as informed at the time you chose them vs the new one.
People misunderstand the concept of liability. Yes your tax professional is liabile for fraud or gross misconduct, but if they make a mistake which results in unpaid taxes or fees you're still the one on the hook.
Typically you are on the hook for paying the actual taxes and penalties, but if you acted in good faith on the negligent advice of a tax professional many jurisdictions allow you to recover the costs of penalties and interest accrued as a result of the negligent advice.
The IRS made it crystal clear to me long ago. The tax filer is responsible for everything on the tax return, regardless of the preparer. Of course, accountants can be liable in other ways, but the filer has to pay the fines if the accountant makes a mistake.
From the IRS' point of view, the tax-payer is responsible for paying all of it. You can then turn around and sue to recover any of the fees/fines or penalties incurred as a result of an error made by the accountant, not the actual taxes though.
It seems disingenuous by our government. Create a tax strategy that is so complicated, people feel like they have to hire professionals. However, if the professionals err, it's the citizenry that pays the penalty.
Seems like simplify or just erradicting the tax code would be more just at this point.
You can - literally - have the same government force the negligent CPA pay via the judiciary. That’s what a lawsuit is.
Normally it doesn’t get to that point though. CPAs do not want to be known as the person that misfiled taxes and got a court to rule them negligent. They carry insurance for errors so mostly just end up queitly making it right when there is an error.
yeah i think liability ultimately lies on the individual. of course, you could sue your accountant, but that feels hard, maybe almost as hard as suing the model providers.
At least in Poland it is not viable. Friend's business lost noticeable amount of money because accountant has not filed taxes and lied that they filed and lied that they checked it.
And there was no real path to recovering funds, except small part, after long court case.
At least they were able to fire them without severance pay.
This seems to be usual. Accountants are liable for crimes like embezzlement, but tax authorities hold individual tax payers responsible. If tax payers want to recover losses they have to go to court, which is often risky because accountants will say the client didn't do A, B, and C.
Same story here as many others: I now do my personal and business taxes myself using an application I had claude cook up for me, doing it 100% correct, for South Africa's tax system. Previously I paid thousands to a small accounting firm to do it for me as it can be slightly nerve wrecking, but I had claude explain to me exactly the mechanisms on how everything works, including the tax portal that the government provides to do submissions. Absolutely amazing. Oh and the filing took me 5 minutes to do myself and cost me near zero once it all clicked into place.
Is this just giving a general-purpose agent a workspace full of documents and an assignment, then seeing what happens? I feel this doesn't really say anything. It appears they mostly dumped all of this in base models with little to nothing in the way of infrastructure to support it. If you just randomly grab some person off the street and say 'do taxes' they would probably mess up just as badly. The interesting question would be what the models could do in a well engineered harness focused on taxes. That would be reflective of the performance boundary and how serious we would have to take AI in that field.
Agree. Even a dumb step like "list every rule and deadline that applies, then check each one" would probably catch the missed 2-of-5-years deadline from their own example.
This is about wild west accounting, though, when the user has no idea what they're doing.
When launching my financial business, I spent half a year learning accounting to be sure to understand everything, then I implemented my accounting software, specialized for my activity. And as luck had it, I made it be a CLI. This last few months, I've been using Codex(lately with GPT-6.1 Sol) to do the annoying work - attaching proofs to entries, fixing the dates and amounts issues, etc. It works on on a database copy, generate a script of commands using my CLI to fix and close the previous month, that I then review before executing on my prod database, sometimes asking questions to the agent about why this or how that. … And it's been flawless so far, both on my professional accounting and my personal one. Knowing a bit about what we ask the model to do can go a long way.
> Sharing all my financial info with LLM companies is not something that I like.
The price of convenience and massive time savings is taking some risks on that front. We run a small business. We heavily depend on codex doing all sorts of administrative stuff that in the past either wasn't happening at all, wasn't happening properly, too late, or was being outsourced to people with various degrees of flakiness/incompetence/ineptness. We got rid of all of those and freed up a lot of our time.
As a small business you are always reacting to incoming demands on your time (accountants, taxes, invoices, insurances, etc.) for all sorts of reasons that boil down to having to do a lot of detective work in documents, filling in lots of forms, responding to queries, etc. AI tools can do essentially all of that now faster and better than most people can. Even the complicated stuff that involves fiddling with difficult tools, poorly designed websites, convoluted processes, etc.
Half the success here is getting organized on how and where you store data. A lot of SAAS tools actually make this harder, not easier. It really matters which bank you use, for example. Some banks have decent APIs and tools and export options. Others don't. Same with accountants. Some are more forward thinking than others.
It's a bit of a leap of faith when you start delegating important stuff to LLMs. And then you realize it's mostly fine and that it's doing a better job than you would. You still should double check the output. And you probably should still use an accountant to sign off on things and do some final checks. But don't fill in forms manually or go hunting through your drive for some document or information.
I'm for sure reinventing the wheel here but I've had an LLM generate an app that walks me through my existing tax routine I've been running with for years. Thus no real numbers go to any LLM I just give it my situation and it fills out the form.
The plan now is to retrieve the current year's tax forms then have an LLM compare them to stored versions from last year, and look for variances (e.g. new field, new form requirements, etc) all of which an LLM would excel at (whilst basic math it historically did not).
It does not go far enough. If the government won't allow the IRS to autoprepare the 90%+ returns that they already have the data for than paid tax preparers should be licensed, whether it is as an Enrolled Agent, CPA, or some newly created tax preparer that cannot represent you before the IRS license. The fact that a company like HR Block or Turbotax can hire someone with no experience, sign them up for a PTIN, teach them how to use turbotax, and then call them an "expert" is insane.
As a counterpoint, I ask the big llms every month the same set of tax questions about the last US tax bill and they still can't give me the correct answers.
I started my career fixing the mistakes that my clients' former tax advisers made. It looks like the next generation of tax lawyers will get to do the same except this time the former advisers will be llms.
I find TurboTax is easy enough to use, except that data collection and entering is a huge hassle (to me, merely logging onto each website and getting the corresponding tax doc is painful chore). What would be a good way to collect the data easily? If I solve that problem, tax filing will become an easy task.
The best thing in my opinion is using the TurboTax Desktop software, installing it on a VM and spinning that up only when you need to do taxes. The import/export relies on tax files from the previous season, but they seem to essentially be slightly encrypted archives.
Everybody is saying their tax accountant messed up and the models caught it which makes me think, why aren't tax accountants doing the same thing lot of us are doing - ask LLM to do the prep and they just review and finalize?
Can't be good getting caught with mistakes and losing customers?
A lot of accountants are old school and I'd guess that most folks outside of tech are not nearly as heavily on the AI train.
Additionally, I think the industry and individual practices are probably still figuring out contracts, accountability, and legality. Will their insurance policies cover mistakes made by an AI?
I'd also guess that its one thing for you to feed your financial documents to Claude; would you feel as comfortable if your accountant were feeding it to an unknown model of their choosing? I'm not so sure I'd be.
I follow a tax guy on Twitter and he said it is because of regulations.
I just wonder why Intuit can't have some local model and do it within guidelines.
And you can add me to the list of people who ran my tax filings through AI and found mistakes.
Most were the CPAs fault but I forgot to include a local folder when uploading and codex was able to recognize that my CPA did not include those forms.
I would not advocate replacing CPAs with AI but they can definitely fact check the CPA.
I love this type of benchmarking, especially when it to me was counter intuitive that astra missed it. Got a couple of nice prompts and things to think about for my own tax making moving to a new country, but honestly hard for most people to review if they got the optimal output from taxes.
Since it's really a proving game, you can optimise and take more risks with declaration, and some of it has to be proven in an audit. Tried getting data on the risks, how old cases were ruled but ultimately by accountant gave a fairly different view compared to Opus 5 at the time.
I wasn't able to quickly see what tools/system-prompts were provided in the tasks. [0]
These types of detail-oriented professional tasks seem like they would perform better with task-specific harnesses. The example they give, of Astra missing the property excluded gains requirement, seems easily fixed by scaffolding ensuring that requirements are compiled first, then checked.
If the scaffolding is absent, is this benchmark testing that a model can self-scaffold, or track requirements in-context?
In SAP, tax rules are spread across customizing, tables and custom Z code, and the AI can't see any of that just from the task description. In companies, what's useful is having the AI review and explain, and leaving the decision to whoever knows the process. At least that's how I've been using it in my own work, with great results: going through change requests, trying to understand the impact they might have, and so on.
An AI agent can be right today and wrong tomorrow.
And it all depends on the prompt, We have to ask the agent to pull the latest from the web and use them to perform the calculation, otherwise the agent uses the outdated rules.
Better would have an evaluation mechanism which we should use against the ordinary math code for comparison and flag any mismatch
I had Gemini help me with RMD calculations for an estate I manage. Advice was mostly good. Wrong about withholding for both US and CA. Missed deductions on US. Was accurate on MFJ and spousal death impacts.
I've found it so far more effective for preparation than for planning/advice.
I've got a plan to feed the laws and rules and instructions into the machine before the prompts - is RAG still cool? Maybe I should small-language this?
I wonder if you use opus 5.5 and let it research using the internet what you would get. I use opus 5.5 for legal stuff and it's generally great, but you do have to give it access to the web.
Isn't writing out the example in the beginning that Astra missed basically ensuring that every future model will get this task correct?
I guess my point is these benchmarks unless they are constantly refreshed with new scenarios and not ever having their outcomes written about publicly or available on a public server, will always be prone to being gamed
Not related but I hope QuickBooks learn something and make their software better.
I downloaded 1700 of my transactions and use AI to build an app for it, so I can analyze the transactions and make sure that everything is properly categorized.
It used to take me whole day because of the Quickbooks. It took me less than an hour to go through all the transactions.
I used Grok Bot and found out that I could offset some stock value going down and get back ~$300. I gave it my bank statements, email access (which had my stocks monthly report as attachment) and a few other things.
I'd have honestly never been able to do it myself - its just too tedious.
In Germany, there are companies like Buhl data that let you do your tax return very easily and give a live estimate of your return based on the data you provided. From my experience over the past 5 years, its prediction error is less than 5%.
I'm not sure LLMs will be good at doing taxes any time soon primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this:
"If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?"
LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service.
You just need to give an instruction something along the lines of "any initial calculation or derived value must be computed through the use of a $LANG program."
Pretty sure the LLMs could implement tax calculations better than the average accountant, were they given the rules.
As a veteran tax CPA, I think LLMs can be very good at preparing taxes but I would not trust them to complete a tax return start to finish. An LLM that can take data from tax documents and input that data into the correct boxes of tax preparation software has no need for math, it just has to identify data in a pdf, spreadsheet, or handwritten note and transcribe that data to the correct box in professional tax software that then does the math. In your typical local CPA firm, an LLM could replace the unlicensed preparers/interns/early career stage CPAs in filling out the return for the experienced professional to review and adjust according to their specialized knowledge. As with many industries, I think the LLMs are suited at replacing entry level automated turks but lack the skills necessary to fully complete the project.
ChatGPT and Cursor both failed hard. Even after walking them backwards through the flaws in their answers they just refused to see it. When I stated flat out what the answer was ChatGPT agreed in the end. Cursor did not. Cursors initial reply was -200g. Then I challenged "But I did not tare it after adding the almonds." and again it failed with: "That doesn't change the final reading: it still displays -200g.". When I walked it back further with "If I put the container on the food scale, tare it and remove the container, what does the scale read?" it correctly replied "negative the weight of the empty container" but ... when I asked "So does this impact the agents answer to my original question?" it replied "No, it doesn't change the answer—the scale still reads -200g". LLMs are NOT good at even basic numeric accounting.
You keep saying LLMs can’t do this but I get the correct result each time without drama.
Cursor uses Grok so far as I can tell, which is.. not great and I hope you did not try ChatGPT free or 5.6 or Luna.. you cannot unfortunately just say “Claude” or “ChatGPT”. Their lineup is sufficiently complex - each with their own failure modes - to warrant some further specification.
I think you are “directionally” correct, as these clankers like to say, because language models are indeed not known for their arithmetic wizardry. However, as my correct answers show, they can do it just fine.
As a proof-of-concept, I've been using extra tokens to formalize the IRS tax forms and worksheets, mostly just to prove this can be done. The idea is that something like this could be adapted into a publicly maintained version of the IRS Direct File program (RIP). I'm up to an estimated 60% coverage of US federal tax situations so far.
https://github.com/emmilco/us-tax-model
That's interesting! I've re-upped your Show HN (https://news.ycombinator.com/item?id=50278678) using HN's second-chance mechanism (described https://news.ycombinator.com/item?id=26998308).
(Posting here since we don't have another way to contact you, and of course that thread will be better with you in it...)
Thanks so much!
What do you mean when you say 'formalize'? I mean, a full tax document is just each form with each 'line' as an entry by itself and then at the end, all the calculations get performed against that.
That's neat, what aspect(s) are you formalizing? That a given tax statement is correct?
Not quite. The formalization covers the IRS's published procedure, i.e. the questions and calculations that go into the published forms, worksheets, and instructions. It's not a formalization of the tax code itself, and it's not a tax calculator, though a client of the underlying code could function as a tax calculator. (The project includes such a client, a CLI tax interviewer, just to exercise the code.)
In more detail:
- every line of every form gets a YAML representation which includes the source text, a data type, and an expression built from a small set of lean atoms, plus a citation pointing to the exact location of the rule in the original official document.
- underneath the yaml layer there's a lean formalization of the atoms showing that they behave as specified, that all comuptations use only the allowed atoms and operations, etc. There's also a typescript evaluator that's used to cross check the lean for agreement, so we have the arithmetic represented two ways.
The critical failure point for an LLM-produced formalization like this is the interpretation of the form text. Since I'm not a tax lawyer and don't intend to become one, this is an explicitly owned weakness and one I've structured the project around. Publicly available tax calculators (PolicyEngine, UsTaxes, OpenTaxSolver, etc.) are used to cross check derived numbers from this system. Mutation testing ensures the test suite is robust against bugs and arithmetic errors. Interpretations are cross checked by different model families (Claude, GPT, Deepseek). Etc.
The idea is to derive confidence from the formal properties of the system itself, and to allow theoretical future users of such a system to derive confidence in the interpretation of specific forms by making it easy to cross check line-level formalizations against the source documents.
There's a lot more detail in the (alas, Claude-authored) README. When I have time I'm going to write my own version of the README expressing my vision of the project, but I'm not there yet.
The IRS open sourced their direct-file code after it got cut. IIRC similar work done there. https://github.com/IRS-Public/direct-file
One past thread:
IRS Direct File - https://news.ycombinator.com/item?id=44131901 - May 2025 (65 comments)
Some of the interpretations won't be correct without grounding them in old case law. That's the expensive part, in many jurisdictions, the official interpretations by the tax department are not free.
IRS fillable forms is really good in my opinion it should be more widely known.
Nice. Is the lean used to formalize formulas?
Sort of. The formulas for each line live in the form YAML I described above. The formulas are built out of a set of primitive operations that are defined in lean, with their properties proved there (non-negativity, monotonicity, etc.). There's also a lean evaluator that's run on compiled form data and cross checked with the typescript evaluator. (There's a lot more detail in the README.)
You are one sick puppy. This is a brilliant idea.
Nice. Opus 5.5 is crazy good at engineering ideas into life. Really cool stuff, using "code is data" with Lean
I think the IRS already did some version of this and open sourced it https://github.com/IRS-Public/fact-graph
There's an article about it here https://biggo.com/news/202510160717_IRS-Fact-Graph-Public-Re...
These were artifacts from the IRS Direct File program (RIP)
Maybe your project could (re)productionize this dataset.
When I was planning the project, I looked into FactGraph to see if it'd be worth building on. I opted not to for a few reasons:
- It's oriented toward the IRS's internal XML API in a way that's not really useful for this project, since no consumer of my code is going to digitally submit to the IRS.
- The semantics of FactGraph are largely undefined, and the maintainers themselves seem to acknowledge that its development trajectory went off in a bad direction and it needs major divisions.
- It's super verbose.
- It's the year of Lean, so I wanted to find an excuse to use Lean.
Other work has been done in this domain, though. The biggest example is Catala[0], though Prof. Sarah Lawsky at Northwestern seems to have made some strides toward formalizing the Internal Revenue Code.
[0] https://catala-lang.org/
I recently did something similar, using Claude Code to formalize a set of rules for a sporting event as part of building a computer-based scoring system. It's worked very well.
That's very cool but don't you think the Big AI labs will soon have collected enough data from us that they can easily do this?
I'm not sure it matters. The IRS killed Direct File under lobbying pressure from the tax prep industry. In my opinion the public is owed a free version of this sort of tool. If the IRS doesn't produce it, then citizens ought to. If Google or Anthropic or whoever wants to offer a free version of this, more power to them, but I don't see why they'd bother. They could have done it decades ago.
And to be clear: my vision here isn't "have an agent do your taxes", it's to have agents build a hardened and thoroughly verified deterministic tool that can help you navigate the (somewhat bewildering) forms and instructions decision tree when filing your own taxes.
This is great, and I appreciate the approach. Yes, tax calculation should be a decision tree. Unfortunately it is truly pretty complicated (just this year I had to abandon FreeTaxUSA for TurboTax because the latter handled an important case I needed better). But I have a lot of faith that we can do this right. Good luck!
Funny, after 10+ years of using TurboTax Online I moved to FreeTaxUSA last year and it's like a breath of fresh air. TurboTax couldn't do what I needed to do (e-file a late return) and FreeTaxUSA had no problem with doing that.
I have a probably stupid question: given that the IRS killed Direct File, even if your software can create ready-to-file (or even ones that need some work) tax returns, isn't that moot without a way to like... get the IRS to accept them?
You just mail them in!
If your software can fill out the forms then you can (at the very least) print them and mail them in. The IRS certainly still accepts that.
https://www.irs.gov/filing/where-to-file-paper-tax-returns-w...
Ohhhh I thought Direct File included the mail-in options, not just the online one. Thank you!
Why wouldn't the IRS accept a tax return created by your own software? How would they even know it was custom software? Ultimately your tax return is just a stack of papers that gets sent to the IRS.
Software sending efiles to the IRS has to be tested with and approved by the IRS. The IRS is working toward eliminating paper filing as an option. At this point, paid preparers are legally required to efile all returns and if the return gets rejected and they must paper file, they have to include Form 8948, Explanation for Paper Filing. For the IRS to accept a tax return from software the software has to go through this process. https://www.irs.gov/e-file-providers/how-tax-preparation-sof...
As someone from the Netherlands, this is bonkers to me. I just log in to the tax site, review what they've already filled in for me, and submit.
It is truly insane. Major corporations spent billions to prevent this. The worst part is that for the majority of Americans, the government has already internally prepared their tax return before they even file but again the tax preparation industry spends a fortune lobbying Congress to prevent an automatic filing of taxes.
I think you meant to say they spent billions to create this?
In either case the tax prep industry is a pox on our society. But of course this is America, and replacing a perfectly functioning publicly accessible service with a nightmare of private sector bullshit is the most violently American shit ever.
As an American, it is bonkers to me too. Last year my former employer did not issue me a w-2. I called the IRS to ask them what to do. They said they had all the information but can’t release it to me until after the filing deadline. Instead I was told to use form 4852 to estimate and file an amended return if my estimate was wrong once they give me the numbers they have. Worst part is you can’t efile when you file with form 4852.
Anyway, my point is that for 95% of people they have all the information already and you shouldn’t have to file.
This! I am CPA with a tax practice and it makes me unreasonably angry that the tax prep industrial complex has lobbied our government into making the vast majority of people stress over their tax returns or have to pay someone to prepare a return that is already sitting on an IRS computer. No CPA worth their salt wants to do the simple returns but companies like Intuit and HR Block feed off them.
The first option on Form 8948 is "Taxpayer chose to file this return on paper" - is there any evidence paper filing is going away permanently?
You can use free file fillable forms to submit online. It's a bit of work to transcribe from other software. It also requires a particular interpretation when filling forms (e.g. you need to round before summing your W-2s).
I think that assuming work can be done more easily in the future should not stop people from doing work that needs to be done in the present.
(If a painter knew that the camera was coming out I would not want them to throw their canvas away)
> (If a painter knew that the camera was coming out I would not want them to throw their canvas away)
That's not a fair comparison because the painter produces different work, whereas tax computations will be the same.
[citation needed]
Plus there is little chance it will be done in the future by any large entity because of the entrenched interested in the area.
This is amazing. I’ve wanted something like this for a long time. I’d pay for it even.
I prepared my taxes this year using Astra + Codex. It did a thorough job on a complicated return, and I sent it to my accountant to fill out the tax form, with everything laid out in a spreadsheet. They sent it back for final review, and I had Codex check it; it was riddled with errors. The refund was nearly $2k lower than it should have been, and they put a bunch of stuff into the wrong fields. Had to go back and forth multiple times to get it 100% correct. Made me wonder how many previous years' returns were wrong.
I feel like we're going to get to a point where it's mandatory to use AI for certain tasks - humans just can't be trusted to get things right that require a level of precision.
I sent my business and personal returns through Claude this year before signing, and it turns out they were a horror show of unforgivably stupid mistakes. Pretty infuriating considering what my now former CPA was charging me.
He amortized the goodwill of businesses I acquired over 5 years instead of 15 (incredibly basic stuff, which Claude flagged with *CRITICAL ERROR*). When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it. I don't know if there's a good response to making that kind of a mistake, but if there is, it's definitely not that one.
> When I asked about it, he said he had caught the error in his review but forgot to save after he fixed it.
Obviously total bullshit.
Did you fire him?
Ask for damages?
Yes I fired him, and I offered him the option to take 50% off his bill or chase me for payment. He chose the former.
> my now former CPA
Assume yes, fired.
You all have zero concerns sending the AI providers your personal and business financial data?
Your anecdote seems to clash heavily with the results of the benchmark announced in the article. I suppose that is part of the jagged experience of agentic AI
The benchmark announced in the article seems to be around weird edge cases (which are often important) and whether it can prompt the person with relevant information rather than just listening to what they want to do.
Top tier tax professionals can do pretty well on this benchmark (better than agents) but won't hit 100%.
The kind of people spending time preparing oodles of returns would not do very well on this metric, though (especially midway through that pile of returns, as opposed to "think carefully about this trick question").
The OP said it was a complicated return so anyone assuming OPs anecdote is in line with the article is filling the lines in themselves with a lot of assumptions
> Made me wonder how many previous years' returns were wrong
Feed previous year returns into Astra and find out! It's not too late for amended returns, you have 3? years from original date of filing
[dead]
I have a non-trivial tax situation, as a US citizen living abroad. I prepare my own tax returns, never having found a cross-border tax preparer I trust, and have for the past few years, fed my draft return along with all the supporting paperwork into Claude. It has:
1. found errors, e.g. where I used the incorrect exchange rate (and managed to work out how I had made that error, which I was impressed by)
2. found items which would appear to be errors except for context it did not know
3. flagged areas where it disagreed with me as to the correct tax treatment, but accepted that the law is ambiguous (and in some cases I preferred its interpretation after additional research and in other cases I did not)
4. flagged areas where it disagreed with IRS guidance, and only acknowledged said guidance after I provided links to it
5. provided highly inaccurate estimated tax calculations, and not been able to fix its estimates even when challenged
So my takeaway is that it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check. (Possibly it is if you just have a W-2 and no other substantial income, but I can't speak for that scenario.)
I have a company abroad in addition to living abroad. It’s kind of a nightmare to do my US taxes and costs me easily $2000 per year right now. I was hoping I could have an agent do them after I have a few years of data and point it at the source statements and the end result forms
According to these benchmarks. It's not there yet
form 5471?
Among a few others yes
I also had a great experience consulting LLM regarding registering a business as a foreigner. Except I'm still very sceptical since it's not my area of expertise and consequences for a mistake are very high.
you were consulted by an LLM?
He said he's a foreigner, almost certainly a mistranslation.
Embarrassing...
I'm also in a non-trivial tax situation, GPT Sol did better than my specialist accountant this year. It needed a lot of steering from me and about 2B tokens, but that is better than spending $3-4k in preparation fees. Total time to prepare and handle steering was still less than I normally spend on tax prep with my accountant.
I think what helped is giving it 100+ pdf and csv documents of my full situation so it knew everything about me and could cross-reference with all my bank statements.
So, GPT Sol outputted tax return PDFs that you were able to submit to the IRS? (I'd imagine that you couldn't directly e-file like you could from TurboTax.)
Correct, it first did it in a spreadsheet, then filled out 59 pages of returns and supplement forms and I printed and posted it. It also filed my FBAR.
I don't know that I'd trust it with an FBAR given it's an interactive PDF myself, but good to hear that it was able to do it - maybe I'll get fed up doing the FBAR manually one year and trust an agent to do so.
They changed the FBAR page recently, no longer an interactive PDF.
Ah looks like there's both the PDF and online option; I filed with the PDF version because you can save your history but I guess if Claude can navigate and fill in the online option then I have less quibble about being able to keep progress. Thanks!
Amazing story!! Honestly, thanks for sharing so completely.
Is it unusual for Americans to spend that much on tax returns?
Yeah for individual tax preparation that is pretty high. Most people get away with filing for $20-150. I used to have a CPA do my consulting business, tiny 1 person company, and personal taxes for $500.
I no longer do consulting on the side so I just use online tax software. It is about $150 but I do that more out of laziness as it integrates with my brokerage and payroll company to import everything automatically. Also I have carry forward losses and such that it already knows about.
Most people’s taxes are simpler than mine but I am sure there are lots of folks here with much more complex tax situations
I am not a normal case because I don't live in USA. The USA government estimates 50-80 hours to complete all my forms.
I have foreign income and foreign businesses, my partner has foreign income from two countries. The tax law is then based on tax treaties, most tax preparers do not have sufficient knowledge to complete a return, and tax software doesn't handle these cases.
The tax treaties are not clear, tax professionals will disagree on interpretation, and tax benefits in one country will not apply to another.
Then there is a thing called AMT, alternative minimum tax, which is a second way to calculate your taxes that you need to submit as well.
It's unusual to spend that much even for a small business tax returns except maybe in major cities
> I have a non-trivial tax situation, as a US citizen living abroad.
Same situation as you. I have a local accountant in the country I'm in plus my US accountant. Claude/ChatGPT have been a huge help for checking, explaining, and finding issues. They also found websites with pertinent information that Google did not, which surprised me.
Similarly, had Claude look at my past returns and found mistakes in my records.
I had it prepare an analysis and then found a professional to confirm everything and then handle doing the amendment.
Definitely useful as a second pair of eyes.
Hacker's breach Anthropic and dump session logs; they have your software designs, tax records, draft business proposals -- Claude doesn't get all of your information, but it helps with deep details on a significant cross section of it. How do you mitigate this loss?
https://claude.com/marketplace/connectors/aiwyn-tax
Try this, it uses a tax engine as the backbone and provides feedback for Claude if it goes off the rails. Albeit it doesn't support every tax scenario.
I appreciate the recommendation, but as it happens I have already solved the estimated issue (mostly by just preparing a full return with estimated next year's figures, which while time-heavy is going to be more accurate). And for what it's worth, I suspect it wouldn't work for me - cross-border is extraordinarily complicated because of the tax treaties (indeed there is a dedicated IRS form to disclose the effect and rationale of any treaty based position), and given there is no obvious note on the documentation for it that there is cross-border support I doubt there is.
(And as I said to another commenter, I think it would be advisable in this thread for people to expressly acknowledge or disclaim any interest in or prior experience with any particular product)
check out https://www.multiplierholdings.com/
it's the most ai-native tax experience for US citizens abroad
Are you saying I should use this (or it looks like one of their portfolio companies) as my return preparer, or as a replacement for Claude my return-checker? If the former, I would want to know why I should trust it; if the latter, I would want to know why I should pay for their services given I am already playing Anthropic for a subscription plan anyway.
(It would also help to know if you had a financial interest in or prior experience with this company)
My tax accountant mis-filed my taxes a couple years ago, which cost me about $20K (which I learned after I switched to another accountant later). Don't know if Claude / GPT would make the same mistake or not given SOTA today, but feels like there's an opportunity for models to be saving me thousands or tens of thousands of dollars a year.
>mis-filed my taxes a couple years ago, which cost me about $20K
What do you mean by "cost you $20K"? That your tax liability was calculated by the accountant to be higher than it actually should have been? Or that the accountant calculated your liability too low, which was caught and billed by the IRS?
In the first case, you can still file for a refund within 3 years (I'm omitting some technical details), but in the latter case, as mentioned by others, you still owe the tax whether it was correctly reported on your original return or not.
Years ago I had a similar issue. Accountant did something incorrect, increasing my liability by around $50k. Tax date came so I paid it, but moved accountants. New person fixed the issues, refiled, but it took a long time to get the refund back. IIRC, the IRS did include some nominal interest for the time they had the money.
Sounds like an opportunity which could end up costing or saving $10k and which may be difficult to evaluate.
My obligatory comment about my experience with tax accountants:
https://blog.nawaz.org/posts/2022/Nov/do-you-trust-your-tax-...
I'm also 0 for 4 on tax preparers / tax accountants, so I concur with you its best to just do it yourself, which is what I now do. Only time I ever got audited (and failed) was when I had employed an accountant, since then I've never had a problem including passing audits. The tax code isn't /that/ complicated for most people's situation, and it's well worth understanding it.
I'm 1 for 6. And my current firm is overworked. I'm moving to do it myself (three companies, fund admin and personal). Recently Found a Tax-Accountant, not a CPA, who does advice only, no paperwork, who I'm using to cross-check my AI tools.
I am a CPA with 20+ years of experience and I feel like I need to defend our industry here.
There is no such thing as "failing" an IRS audit. You either under reported income, overstated deductible expenses, or your "accountant" changed the numbers you provided and committed fraud, which would have resulted in immediate legal proceedings by the IRS against the preparer. So did you "fail" because your preparer committed federal tax fraud or did you underreport your income?
The statement that the one time you used an accountant you got audited implies that you are confusing causation with correlation. There are about 6 ways that the IRS selects returns for audit but only 3 are actually relevant here.
- Information Matching (ie. if your W2 does not match the W2 submitted by your employer). This mostly used for your basic W2, 1099DIV, etc. kind of taxpayer and is usually just a letter from the IRS telling you your numbers are wrong and you need to amend your return.
- If they are examining a related return (ie. you own a business and another partner in that business is being audited)
- Computer scoring which breaks down into the Discriminant Function System (DIF), basically, how does your return compare to similar returns, and Unreported Income DIF, what is the potential that your return underreported income. This is what is commonly used for businesses and self employed individuals' Schedule C. Basically, when you prepare a business return (1040 Sch. C, 1041, 1065, 1120, 1120S, or 990) you have to report your business industry's NAICS code, and provide a single line explanation what your business does. Using this and the return data, the IRS is able to score a return on the likelihood that profit is understated by comparing a businesses income and expenses to the norm for a similarly sized businesses in that industry. High scoring returns are then reviewed by IRS personnel.
One of those three things happened to you and none of them are the fault of a preparer that is using accurate information provided by the taxpayer.
Finally, you keep saying things like "accountant". You disparaging an entire industry without saying what type of tax preparer you used. Was it a CPA, licensed by a state with experience in tax? An Enrolled Agent, licensed by the IRS directly? Or a tax preparer, an unlicensed person preparing your return either on their own or through something like HR Block or Turbotax? Because if you used an unlicensed tax preparer and are blaming licensed individuals that is like hiring an unlicensed electricians and then when your house burns down from an electrical fire, turning around and saying all electricians are bad
Additionally, "the tax code isn't /that/ complicated"... tax law is so complex that it is the only form of US law in which ignorance of the law is an acceptable legal defense.
In his defense, he did say "... for most people".
And that's true if all you do is have W-2 income, and RSUs/ESPPs. ESPPs were the most complex thing I had to deal with until I stupidly decided to do things and get K-1s.
Fair point, but the actual "tax code" (Title 26 of the US Code) is extremely complex for most people to understand. If he is referring to IRS Form Instructions as the tax code, yes those are usually not very complex. From my perspective K1s, RSUs, & ESPPs are as simple as a W2 with state tax withholding and I think that is why I am struggling with so many of the comments on this post - I simply can't relate.
> From my perspective K1s, RSUs, & ESPPs are as simple as a W2 with state tax withholding and I think that is why I am struggling with so many of the comments on this post - I simply can't relate.
Well, you are an accountant :-)
With my K1, the issue was that it was for a real estate investment, and you have to be careful about things like bonus depreciation/cost segregation etc. One tax accountant I used was not carrying over the loss (or perhaps not applying the prior year's loss, or something like that). Another accountant was deducting that loss from my W-2 income, which I can't do as I'm not a real estate professional.
When I finally brought these points up and got all these corrected, I studied the final result and then continued doing them myself. But it was a pain getting the online tax SW (TurboTax, TaxAct, etc) to mimic what they did. I had to be really careful what things I said "Yes" to and what I didn't. I would never have figured it out on my own by reading the relevant tax publications (which I did peruse).
And then later, I formed a partnership with a friend so we could both invest in something. Now I also have to issue K-1s! Not fun! RSUs are much easier.
> There is no such thing as "failing" an IRS audit.
I consider being audited and the IRS finding corrections to prior returns as a "failure". Once I started doing things myself, I was audited and no corrections found (e.g. a "success"). In the case where I "failed" the audit, it had to do with how expenses were being handled by the accountant. At the time I was a contractor working on a 1099 and was tracking and submitting some business expenses for reimbursement, and some expenses were tracked and taken against my taxes but not reimbursed. What was filed did not align exactly with what was reimbursed, which caused an overstatement of deductions, causing me to owe back taxes. I am not making any claim that they committed fraud, as that's a crime which requires intent, I think they were just incompetent.
> The statement that the one time you used an accountant you got audited implies that you are confusing causation with correlation.
I actually didn't say that, I said the only time I got audited and failed was when I used an accountant, however I realize my style of writing made that unclear. During the time where I received income via a 1099 (and for part of that also had to file from a K-1 briefly for an LLC), I was audited three times in total. Only one time did I "fail", and it was the only time where I had professional assistance. In fact, I found the process of being audited surprisingly kind, the IRS folks involved were very professional and weren't mean about it at all. After my first experience being audited, even though I got the nasty surprise of owing back taxes, I was a lot less fearful of that process. That said, I have never had any understanding of how someone is chosen to be audited, but I have not been selected for an audit since moving to a W-2 for all of my income, which has been quite some time now. The experience of being audited did get me into the habit of storing all my tax related records carefully organized by year long-term, however, which I think is probably a good habit for everyone.
> You disparaging an entire industry without saying what type of tax preparer you used.
My intent isn't to disparage anyone, it's to share my own personal experience. I hired a CPA licensed in the state of Missouri (where I resided at the time) when I previously had a professional handle my taxes.
Someone still needs to take liability for mistakes.
The real issue is that accountant most likely has a track record of those kinds of mistakes - but you have no info on other clients. So you weren’t as informed at the time you chose them vs the new one.
People misunderstand the concept of liability. Yes your tax professional is liabile for fraud or gross misconduct, but if they make a mistake which results in unpaid taxes or fees you're still the one on the hook.
Typically you are on the hook for paying the actual taxes and penalties, but if you acted in good faith on the negligent advice of a tax professional many jurisdictions allow you to recover the costs of penalties and interest accrued as a result of the negligent advice.
How much money does that take?
How long is a piece of string?
Twice the length from one end to the middle.
Yeah, I looked him up just now actually, and he has a 3-star rating on Yelp... Which I think is pretty low for accountants?
I think the red flag should have been when he printed out all my tax returns and made me pick them up in person, and didn't even send a digital copy.
In this case accountant DID NOT take liability for mistakes.
It's not the accountant that paid $20k for his mistake but the accountant's client.
[dead]
i'm a bit unclear who has the liability even when you use a CPA. accountants are humans that sometimes make mistakes.
The IRS made it crystal clear to me long ago. The tax filer is responsible for everything on the tax return, regardless of the preparer. Of course, accountants can be liable in other ways, but the filer has to pay the fines if the accountant makes a mistake.
From the IRS' point of view, the tax-payer is responsible for paying all of it. You can then turn around and sue to recover any of the fees/fines or penalties incurred as a result of an error made by the accountant, not the actual taxes though.
It seems disingenuous by our government. Create a tax strategy that is so complicated, people feel like they have to hire professionals. However, if the professionals err, it's the citizenry that pays the penalty.
Seems like simplify or just erradicting the tax code would be more just at this point.
You can - literally - have the same government force the negligent CPA pay via the judiciary. That’s what a lawsuit is.
Normally it doesn’t get to that point though. CPAs do not want to be known as the person that misfiled taxes and got a court to rule them negligent. They carry insurance for errors so mostly just end up queitly making it right when there is an error.
I believe it's a global thing.
In some jurisdictions, it may help with criminal but not civil liability.
In NZ the liability is generally on you and not the accountant.
> Someone still needs to take liability for mistakes.
I am not aware of setups available to me that would transfer liability onto accountant.
So in either case I am taking liability.
(also, I am pretty sure that if corporation loses 10 millions over accountant making bad decisions, they are not going to recover that funds either)
yeah i think liability ultimately lies on the individual. of course, you could sue your accountant, but that feels hard, maybe almost as hard as suing the model providers.
Yes, at least in the US the individual must sign their own return.
At least in Poland it is not viable. Friend's business lost noticeable amount of money because accountant has not filed taxes and lied that they filed and lied that they checked it.
And there was no real path to recovering funds, except small part, after long court case.
At least they were able to fire them without severance pay.
This seems to be usual. Accountants are liable for crimes like embezzlement, but tax authorities hold individual tax payers responsible. If tax payers want to recover losses they have to go to court, which is often risky because accountants will say the client didn't do A, B, and C.
> Accountants are liable for crimes like embezzlement
and even then, if they already lost all money you may be unable to recover it
[dead]
[dead]
Same story here as many others: I now do my personal and business taxes myself using an application I had claude cook up for me, doing it 100% correct, for South Africa's tax system. Previously I paid thousands to a small accounting firm to do it for me as it can be slightly nerve wrecking, but I had claude explain to me exactly the mechanisms on how everything works, including the tax portal that the government provides to do submissions. Absolutely amazing. Oh and the filing took me 5 minutes to do myself and cost me near zero once it all clicked into place.
Is this just giving a general-purpose agent a workspace full of documents and an assignment, then seeing what happens? I feel this doesn't really say anything. It appears they mostly dumped all of this in base models with little to nothing in the way of infrastructure to support it. If you just randomly grab some person off the street and say 'do taxes' they would probably mess up just as badly. The interesting question would be what the models could do in a well engineered harness focused on taxes. That would be reflective of the performance boundary and how serious we would have to take AI in that field.
The goalposts shifted so hard here I think we're playing rugby instead of football
Agree. Even a dumb step like "list every rule and deadline that applies, then check each one" would probably catch the missed 2-of-5-years deadline from their own example.
How do you think 99.95% of people who do taxes with gen AI are doing it?
This is about wild west accounting, though, when the user has no idea what they're doing.
When launching my financial business, I spent half a year learning accounting to be sure to understand everything, then I implemented my accounting software, specialized for my activity. And as luck had it, I made it be a CLI. This last few months, I've been using Codex(lately with GPT-6.1 Sol) to do the annoying work - attaching proofs to entries, fixing the dates and amounts issues, etc. It works on on a database copy, generate a script of commands using my CLI to fix and close the previous month, that I then review before executing on my prod database, sometimes asking questions to the agent about why this or how that. … And it's been flawless so far, both on my professional accounting and my personal one. Knowing a bit about what we ask the model to do can go a long way.
I'm surprised Sonnet 5.5 did noticeably better than Opus and Fable. Same pattern in their other benchmarks.
I like their benchmarks. They ask the right, not easy questions: https://surgehq.ai/blog/sudo-l7
Has anyone verified their figures to see if they hold up?
It didn't really do any better, the difference is well within the error bars for their benchmark.
Sharing all my financial info with LLM companies is not something that I like.
EDIT: good point that paying accountant does not ensure avoiding it
wonderful, you made a good point that make me dislike situation even more
> Sharing all my financial info with LLM companies is not something that I like.
The price of convenience and massive time savings is taking some risks on that front. We run a small business. We heavily depend on codex doing all sorts of administrative stuff that in the past either wasn't happening at all, wasn't happening properly, too late, or was being outsourced to people with various degrees of flakiness/incompetence/ineptness. We got rid of all of those and freed up a lot of our time.
As a small business you are always reacting to incoming demands on your time (accountants, taxes, invoices, insurances, etc.) for all sorts of reasons that boil down to having to do a lot of detective work in documents, filling in lots of forms, responding to queries, etc. AI tools can do essentially all of that now faster and better than most people can. Even the complicated stuff that involves fiddling with difficult tools, poorly designed websites, convoluted processes, etc.
Half the success here is getting organized on how and where you store data. A lot of SAAS tools actually make this harder, not easier. It really matters which bank you use, for example. Some banks have decent APIs and tools and export options. Others don't. Same with accountants. Some are more forward thinking than others.
It's a bit of a leap of faith when you start delegating important stuff to LLMs. And then you realize it's mostly fine and that it's doing a better job than you would. You still should double check the output. And you probably should still use an accountant to sign off on things and do some final checks. But don't fill in forms manually or go hunting through your drive for some document or information.
Your accountant sure will.
Most accountants will not in fact share client data with an LLM company.
Curious where you're getting this confidence from.
I'm for sure reinventing the wheel here but I've had an LLM generate an app that walks me through my existing tax routine I've been running with for years. Thus no real numbers go to any LLM I just give it my situation and it fills out the form.
The plan now is to retrieve the current year's tax forms then have an LLM compare them to stored versions from last year, and look for variances (e.g. new field, new form requirements, etc) all of which an LLM would excel at (whilst basic math it historically did not).
Some inference providers have a Zero Data Retention policy
Worst case scenario what does OpenAI do with your tax return?
- their data will be partially leaked in general, including my financial info
- someone who hates me personally will get access to it (see case where someone put their ex-spouse on no-fly-list etc)
- leak my private info via new version of LLM was trained on it
- sell it to everyone interested (maybe sold after their bankruptcy)
- give it to NSA and whoever else
from obvious ones
[flagged]
Oh look, yet another reason to use local inference.
There's also: https://github.com/column-tax/tax-calc-bench
There is currently a bill in Congress to finally allow regulation of paid tax preparers. Not sure if addresses AI, but probably not.
https://www.accountingtoday.com/news/new-bill-calls-for-irs-...
There is also an IRS form (14157) to report a return preparer complaint, which of course does not include an option to report a faulty AI.
It does not go far enough. If the government won't allow the IRS to autoprepare the 90%+ returns that they already have the data for than paid tax preparers should be licensed, whether it is as an Enrolled Agent, CPA, or some newly created tax preparer that cannot represent you before the IRS license. The fact that a company like HR Block or Turbotax can hire someone with no experience, sign them up for a PTIN, teach them how to use turbotax, and then call them an "expert" is insane.
As a counterpoint, I ask the big llms every month the same set of tax questions about the last US tax bill and they still can't give me the correct answers.
I started my career fixing the mistakes that my clients' former tax advisers made. It looks like the next generation of tax lawyers will get to do the same except this time the former advisers will be llms.
I find TurboTax is easy enough to use, except that data collection and entering is a huge hassle (to me, merely logging onto each website and getting the corresponding tax doc is painful chore). What would be a good way to collect the data easily? If I solve that problem, tax filing will become an easy task.
The best thing in my opinion is using the TurboTax Desktop software, installing it on a VM and spinning that up only when you need to do taxes. The import/export relies on tax files from the previous season, but they seem to essentially be slightly encrypted archives.
Everybody is saying their tax accountant messed up and the models caught it which makes me think, why aren't tax accountants doing the same thing lot of us are doing - ask LLM to do the prep and they just review and finalize?
Can't be good getting caught with mistakes and losing customers?
A lot of accountants are old school and I'd guess that most folks outside of tech are not nearly as heavily on the AI train.
Additionally, I think the industry and individual practices are probably still figuring out contracts, accountability, and legality. Will their insurance policies cover mistakes made by an AI?
I'd also guess that its one thing for you to feed your financial documents to Claude; would you feel as comfortable if your accountant were feeding it to an unknown model of their choosing? I'm not so sure I'd be.
I follow a tax guy on Twitter and he said it is because of regulations.
I just wonder why Intuit can't have some local model and do it within guidelines.
And you can add me to the list of people who ran my tax filings through AI and found mistakes.
Most were the CPAs fault but I forgot to include a local folder when uploading and codex was able to recognize that my CPA did not include those forms.
I would not advocate replacing CPAs with AI but they can definitely fact check the CPA.
Not even getting AI to do the work, but just do basic review of it.
I love this type of benchmarking, especially when it to me was counter intuitive that astra missed it. Got a couple of nice prompts and things to think about for my own tax making moving to a new country, but honestly hard for most people to review if they got the optimal output from taxes.
Since it's really a proving game, you can optimise and take more risks with declaration, and some of it has to be proven in an audit. Tried getting data on the risks, how old cases were ruled but ultimately by accountant gave a fairly different view compared to Opus 5 at the time.
I wasn't able to quickly see what tools/system-prompts were provided in the tasks. [0]
These types of detail-oriented professional tasks seem like they would perform better with task-specific harnesses. The example they give, of Astra missing the property excluded gains requirement, seems easily fixed by scaffolding ensuring that requirements are compiled first, then checked.
If the scaffolding is absent, is this benchmark testing that a model can self-scaffold, or track requirements in-context?
0 - https://huggingface.co/datasets/surgeai/DAYJOB-tax/tree/main...
In SAP, tax rules are spread across customizing, tables and custom Z code, and the AI can't see any of that just from the task description. In companies, what's useful is having the AI review and explain, and leaving the decision to whoever knows the process. At least that's how I've been using it in my own work, with great results: going through change requests, trying to understand the impact they might have, and so on.
An AI agent can be right today and wrong tomorrow.
And it all depends on the prompt, We have to ask the agent to pull the latest from the web and use them to perform the calculation, otherwise the agent uses the outdated rules.
Better would have an evaluation mechanism which we should use against the ordinary math code for comparison and flag any mismatch
I had Gemini help me with RMD calculations for an estate I manage. Advice was mostly good. Wrong about withholding for both US and CA. Missed deductions on US. Was accurate on MFJ and spousal death impacts.
I've found it so far more effective for preparation than for planning/advice.
I've got a plan to feed the laws and rules and instructions into the machine before the prompts - is RAG still cool? Maybe I should small-language this?
I wonder if you use opus 5.5 and let it research using the internet what you would get. I use opus 5.5 for legal stuff and it's generally great, but you do have to give it access to the web.
Gemini has access to the web. It should have had access to this, if not even already known some of these rules.
Admittedly, it could be promting issue, lots of context for these decisions.
Isn't writing out the example in the beginning that Astra missed basically ensuring that every future model will get this task correct?
I guess my point is these benchmarks unless they are constantly refreshed with new scenarios and not ever having their outcomes written about publicly or available on a public server, will always be prone to being gamed
Not related but I hope QuickBooks learn something and make their software better.
I downloaded 1700 of my transactions and use AI to build an app for it, so I can analyze the transactions and make sure that everything is properly categorized. It used to take me whole day because of the Quickbooks. It took me less than an hour to go through all the transactions.
I used Grok Bot and found out that I could offset some stock value going down and get back ~$300. I gave it my bank statements, email access (which had my stocks monthly report as attachment) and a few other things.
I'd have honestly never been able to do it myself - its just too tedious.
The failures may be spotted much faster if there is a visual presentation to group all agents' execution paths.
Article I hoped to read: Evaluating AI agents on 50 professional tax evasions
Useless benchmark as it seems to be evaluating literally only the weights.
What is human baseline? How it stacks against 10% LLM accuracy?..
wonder if we should bring back https://taxheaven3000.com/ in the age of ai
Ah, yes, the 'dude trust us' bench.
i want to always have a fast way of reviewing my tax returns without having to hassle my accountant or hire another one.
In Germany, there are companies like Buhl data that let you do your tax return very easily and give a live estimate of your return based on the data you provided. From my experience over the past 5 years, its prediction error is less than 5%.
They exist in USA - eg: freetaxusa.com
Kinda points to another thing... that our tax code is insanely bad.
So are the accountants completely fucked yet?
Not yet, and not completely.
Some accountants need to make judgement calls. LLMs are not great at many judgement calls yet.
10% of the time it works every time
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
[flagged]
I'm not sure LLMs will be good at doing taxes any time soon primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this: "If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?" LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service.
You just need to give an instruction something along the lines of "any initial calculation or derived value must be computed through the use of a $LANG program."
Pretty sure the LLMs could implement tax calculations better than the average accountant, were they given the rules.
In my experience agents will reach for numpy for any non trivial math without being instructed
It's the judgement of "trivial" where I find issues.
Most calculations are trivial.
It is also trivial to specify deterministic tool calling for all math
Exactly. That's why I specify it that way.
[flagged]
As a veteran tax CPA, I think LLMs can be very good at preparing taxes but I would not trust them to complete a tax return start to finish. An LLM that can take data from tax documents and input that data into the correct boxes of tax preparation software has no need for math, it just has to identify data in a pdf, spreadsheet, or handwritten note and transcribe that data to the correct box in professional tax software that then does the math. In your typical local CPA firm, an LLM could replace the unlicensed preparers/interns/early career stage CPAs in filling out the return for the experienced professional to review and adjust according to their specialized knowledge. As with many industries, I think the LLMs are suited at replacing entry level automated turks but lack the skills necessary to fully complete the project.
Look at multiple posters here explaining that their accountants made O($10k) errors that were discovered by LLMs.
I’m not sure the evidence is on the side of the CPAs. Maybe CPAs with LLM assistance.
“The scale reads −(100 g + the weight of the empty container).”
I literally gave it your question. What models did you test?
ChatGPT and Cursor both failed hard. Even after walking them backwards through the flaws in their answers they just refused to see it. When I stated flat out what the answer was ChatGPT agreed in the end. Cursor did not. Cursors initial reply was -200g. Then I challenged "But I did not tare it after adding the almonds." and again it failed with: "That doesn't change the final reading: it still displays -200g.". When I walked it back further with "If I put the container on the food scale, tare it and remove the container, what does the scale read?" it correctly replied "negative the weight of the empty container" but ... when I asked "So does this impact the agents answer to my original question?" it replied "No, it doesn't change the answer—the scale still reads -200g". LLMs are NOT good at even basic numeric accounting.
You keep saying LLMs can’t do this but I get the correct result each time without drama.
Cursor uses Grok so far as I can tell, which is.. not great and I hope you did not try ChatGPT free or 5.6 or Luna.. you cannot unfortunately just say “Claude” or “ChatGPT”. Their lineup is sufficiently complex - each with their own failure modes - to warrant some further specification.
I think you are “directionally” correct, as these clankers like to say, because language models are indeed not known for their arithmetic wizardry. However, as my correct answers show, they can do it just fine.
Size, unfortunately, matters.
Free version of Gemini also gets it right
[dead]
[dead]
[dead]