Even if you are nearing retirement and have a large 401(k) or IRA balance, you may not have as much money as you think. Lots of people don’t realize that they owe taxes on their accumulations.

You have benefited throughout your investing years by not having to pay taxes on contributions into traditional 401(k) plans and IRA accounts and on the investment returns on those contributions. Deferring taxes for decades has real value. We usually come out ahead, even after we pay taxes on the withdrawals. And higher income people benefit the most.

The problem is that higher income investors tend not to take taxes into account when assessing how well prepared they are for retirement.

For some, the tax bill will be quite high, as shown in the table below. The group under consideration is working households ages 55-64 with assets in a 401(k) plan in 2016 — the most recent year for the Federal Reserve’s Survey of Consumer Finances. The income levels that define the deciles in the first column are for all working households. The second column reports the median combined 401(k) and IRA balances for those households in each decile that have a 401(k). The third column, which is based on an earlier study, shows the estimated tax rate in retirement that will be faced by households in each decile. The final column shows the after-tax 401(k)/IRA amounts that households in each decile will have to support themselves.

Notice that after the bottom half of the income distribution, the next four deciles, which hold about 45% of the assets, will pay about 22% to the federal government. The top decile holds the remaining 44% and will pay about a third of their assets in taxes.

The point of showing this table is to remind us that many will likely have less in retirement asset than we think. This makes tax-smart investing and financial planning all the more important.