When the Model Is the Algorithm: Testing AI Visibility in a Moving Menu

On September 2, Google added Gemini 3.8 Flash to the model menu in AI Mode. Paid subscribers on Google AI Pro and Ultra can now pick it from the plus icon in the search bar. It is the third Flash model Google has released in six weeks. Gemini 3.7 Flash joined the same menu on August 14.

That sounds like a product note for subscribers. For anyone measuring how their organization shows up in AI search, it is more than that. The model behind AI Mode now changes on Google's model release schedule. And a paid user can choose which model answers the question.

For years, "the algorithm" meant one ranking system that everyone saw, give or take personalization. In AI Mode, the answer a person reads depends partly on which model wrote it. That changes how visibility should be tested and reported.

What changed this month

Here is the current setup, as far as Google has described it.

• Free users get the default. Gemini 3.5 Flash has been the default model in AI Mode since Google I/O in May. Nothing announced in September changes that. Most people who use AI Mode never see a model menu.

• Paid users get a choice. AI Pro and Ultra subscribers can pick Auto, a Flash model, or Pro. Google has not said which model Auto uses.

• The menu changes quickly. In the screenshot Robby Stein, Google's VP of Product for Search, shared on September 2, Gemini 3.7 Flash no longer appears. A model that was available three weeks ago may already be gone.

• The documentation lags. Google's help page for the AI Mode model menu still lists only "Fast" and "Pro."

Why the model matters for citations

Each model handles a query in its own way. It may break the question into different sub-queries. It may weigh sources differently, or decide differently whether an answer needs a link at all. A citation test run on 3.7 Flash in August and one run on 3.8 Flash in September came from different models. If the results differ, you cannot tell whether your content changed or the model did.

This matters most for organizations that started tracking AI citations this year. A university checking whether its program pages appear in answers may see a swing from one month to the next. So may a health system checking its condition pages. Some of that swing may come from the model, not the pages.

It also means paid and free users can see different answers to the same question. Your executives may be paid subscribers. Most of your audience is not. When a leader shares a screenshot of an AI Mode answer that leaves you out, the first question to ask is which model produced it.

How to test without chasing the menu

The fix is not to test every model. It is to make every test record what produced it.

• Log the model with every result. Record the date, the query, the account type, and the model selected. Without that, month-over-month comparisons are guesses.

• Test the default first. The free default is what most of your audience sees. Make a free account your main test series. Treat paid-model checks as a second view.

• Keep the question set fixed. Ask the same questions each month. When the default model changes, mark the date in your log, the same way you would mark a core update.

• Read trends, not single answers. One answer is a sample. A pattern across twenty questions over three months is evidence.

• Keep watching referral traffic. The AI referral segments in your analytics still show what actually reaches your site. Citations matter. Visits and conversions matter more.

What does not change

None of this calls for new content tactics. The qualities that earn citations have been steady across model generations. A direct answer near the top of the page helps. So do a named and qualified author, clear structure, and facts your organization will stand behind. We made the same case in January, when we described citation rate as the new ranking. Faster model turnover makes those basics more valuable, because tactics tuned to one model may not survive the next.

The next thing to watch is whether 3.8 Flash replaces 3.5 Flash as the default. Earlier Flash models followed that path, and a default change would reach every free user at once. Google has not said whether or when that will happen. If it does, expect citation patterns to shift in the weeks after. Give them time to settle before drawing conclusions.

The model is now one more variable in AI search, alongside the query, the user, and the page. Treat it like any other variable. Write it down, hold everything else steady, and wait for enough data before you act.