Features / Log analyser
Log analyser: see what Google and AI bots really do on your site
SEOSage, a desktop app for Mac and Windows, reads your server's access logs and shows which bots visit, which visits are wasted, which pages they never ask for, and how it changes day by day. The log files stay on your computer.
Jump to: The screen · What it shows · Wasted visits · Log files · How it works · Questions
Drop in your logs, see every bot
Open Log analyser in a project's menu, under Search and traffic, after AI Visibility. Drop in your server's log files, or press Add log files. SEOSage reads them on your computer and shows who visits.
Example data: a made-up shop and made-up numbers. Your own logs show what really happens on your site.
What the Log analyser shows
- Who visits: every bot, with its visits, its share of all visits, its error share, the pages it asked for and the last day it was seen. Normal visitors are only counted, never listed.
- Real or fake: each bot's address is checked against the lists that Google, Bing, Apple, DuckDuckGo, OpenAI, Perplexity and Common Crawl publish for their bots. A bot is shown as Real, Claimed only or No list.
- Wasted visits: bot visits spent on missing or broken pages (4xx and 5xx), on moved pages (3xx) and on endless variations of an address with ?parameters, for each bot, and the addresses that waste the most.
- Pages never visited: pages from your latest crawl that load, can be indexed and are not blocked, which the bot you pick never asked for in the logs.
- Trends: visits for each day, for a group of bots or for one bot.
- Export CSV: every view can be saved as a CSV file.
What it does not do
The Log analyser reads files you give it. It does not follow a log live and it changes nothing on your site.
A server log only holds the visits your own server answered. If you use a CDN such as Cloudflare or CloudFront, use its logs, because a CDN answers many visits before they reach your server.
Claimed only is how fake bots look, but it is also what every bot looks like when your logs show your proxy's address instead of the visitor's. Some bots publish no address list, so they show No list and nothing is claimed about them.
Pages never visited depends on the dates in your logs. A short log makes it look worse than it is. Google, Bing, Apple, DuckDuckGo, OpenAI, ChatGPT, Perplexity, Claude and Common Crawl are names of their owners. SEOSage is not affiliated with them.
Find the visits that are wasted
Bots have a limited amount of attention for your site. The Wasted visits view shows how much of it goes to pages that are missing, broken or moved, or to endless variations of an address. The Pages never visited view shows the pages they skip.
Example data: a made-up shop and made-up numbers.
Which log files it reads
SEOSage recognises the format by itself. Files can be plain or zipped (.gz), and large files are read line by line, so a very big log does not need to fit in memory.
- Apache and Nginx in the common "combined" format, with or without a host column.
- JSON lines from Nginx, Cloudflare, Akamai-style exports and similar.
- IIS and Amazon CloudFront W3C files, which start with a Fields line.
- Amazon load balancer logs.
- CSV files with a header row, such as Cloudflare exports.
A file that is not a web server log is refused with its first line shown, so you can see what it is. Lines that cannot be read are counted and skipped, never guessed. The log must show the visitor's name (the "user agent"), because that is how a bot is told from a person.
What SEOSage keeps
Only counts for each day, bot and page address. It does not keep the log lines or the visitors' addresses, and normal visitors are counted without being listed. Clear logs removes everything for the project.
How fast it is
In our tests a 1 GB log (about 6 million lines) was read in about 20 seconds on a Mac, zipped or not. Press Stop at any time: nothing from the file being read is kept.
How it works in three steps
- Get your log files.Download the access log from your host or your CDN. Your host can tell you where it is.
- Drop them in.Drop the files on the page, or press Add log files. Add each file once.
- Read the four views.Who visits, Wasted visits, Pages never visited and Trends. Crawl the site first if you want Pages never visited.
Related: AI Visibility · Rank tracking · Compare crawls.
Questions about the Log analyser
What is the Log analyser?
A page in each SEOSage project that reads your server's access logs and shows what search engines and AI bots actually do on your site: which bots visit, which visits are wasted, which pages they never ask for, and how it changes day by day.
Where do I get my log files?
From your host or your CDN. Most hosts let you download the access log from their control panel, or you can ask their support. If you use a CDN such as Cloudflare or CloudFront, use its logs, because a CDN answers many visits before they reach your server.
Which log formats can it read?
Apache and Nginx logs in the combined format, JSON lines (Nginx, Cloudflare, Akamai-style exports and similar), IIS and Amazon CloudFront files that start with a Fields line, Amazon load balancer logs, and CSV files with a header row. Files can be plain or zipped (.gz). SEOSage recognises the format by itself, and a file it cannot read is refused with its first line shown.
Are my log files uploaded?
No. The files are read on your own computer. SEOSage keeps only counts for each day, bot and page address. It does not keep the log lines or the visitors' addresses, and normal visitors are counted without being listed. Clear logs removes everything for the project.
How does it tell a real bot from a fake one?
It matches the bot's name, then checks the address against the lists that Google, Bing, Apple, DuckDuckGo, OpenAI, Perplexity and Common Crawl publish for their bots. A bot is shown as Real, Claimed only (the name was used from another address, which is how fake bots look) or No list (the owner publishes none, so nothing is claimed). A copy of the lists comes with SEOSage and Update lists downloads the current ones.
Why does every bot show Claimed only?
Your logs probably show the address of a proxy or CDN instead of the visitor's address. Use the original visitor address in the log, or the log from your CDN, and the bots are checked against the right address.
What are wasted visits?
Visits where a bot asked for a page that is missing or broken (4xx and 5xx), a page that has moved (3xx), or one of an endless number of variations of an address with ?parameters. The Wasted visits view shows the share for each bot and the addresses that waste the most.
How does Pages never visited work?
It compares the pages of your latest crawl that load, can be indexed and are not blocked with the pages the bot you pick asked for in your logs. You need a crawl of the site first. A short log makes the list longer than it should be, so the dates covered are shown next to it.
How big a log can it read?
A large one. In our tests a 1 GB log of about 6 million lines was read in about 20 seconds on a Mac, zipped or not. Files are read line by line, so they never have to fit in memory. Press Stop at any time and nothing from the file being read is kept.
Can I add the same log twice, or a log that has grown?
The same file added twice is recognised and skipped. A log that has grown since you added it would count its earlier lines twice, so press Clear logs and add it again.
Which bots does it recognise?
Googlebot and other Google crawlers, Bingbot, Applebot, DuckDuckBot, YandexBot, Baiduspider, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Bytespider, Amazonbot and Meta-ExternalAgent. Any other visitor that names itself a bot, crawler or spider is listed under Other bots.
Does it cost extra, and does it work on Windows?
It is included in SEOSage, with no extra fee and no limit on the size of your logs, and it works on Mac and Windows.
More SEOSage features
Every check and tool in the app, each with its own page.
SEO crawler Crawl settings Page audit Change tracking Compare crawls Rank tracking AI Visibility Search Console report Google Analytics report Core Web Vitals Robots.txt and sitemap audit Backup and restore Local site audit Custom extraction Internal linking tool Orphan pages Link strength Canonical tag checker Title tag width checker URL structure checker Security headers check Keyword cannibalization Search Console quick wins Competitor analysis Weekly SEO to-do list Fix and re-check White-label reports Ask Claude All features
Try it on your own site
Start your 14-day free trial for Mac or Windows. No card needed. Then from $9 a month.
