Website Links
Your website is usually the fastest way to teach your agent. Add individual pages or let Ultimo Bots crawl your site, and the text on each page becomes knowledge your agent can answer from. Automatic rescraping then keeps those pages up to date on a schedule.
Website links live under Knowledge Base → Website Links in the dashboard. Each link is one page of your site. Ultimo Bots reads the readable text from that page, stores it, and uses it to answer visitor questions. Images, navigation menus, headers, and footers are stripped out, so the agent learns from the actual content rather than the page chrome.


Add pages or crawl your site
Click Add Links to open the add dialog. It has two tabs, depending on whether you want Ultimo Bots to find pages for you or you already know the exact URLs.
Crawl Links (discover pages automatically)
Enter your site address, for example
https://www.your-company.com, or the address of your sitemap in Website Link or Sitemap URL, then click Fetch. Ultimo Bots crawls the site and comes back with the pages it found, split into three groups: Newly found (add them with Add), Already trained (read them again with Rescrape), and No longer available (pages you have that the crawl did not find, which you can Remove). You tick the pages you want in each group. Nothing changes until you confirm.Manual Input (paste exact URLs)
Switch to Manual Input and paste one exact page address per Website Link field. Use the button to add more fields. This skips discovery and simply adds the specific pages you list, which is handy when you only want a handful of known pages.
/sitemap.xml. Starting from the sitemap lets the crawler use your own list of pages instead of guessing from links, which tends to find the right pages faster.How crawling works
When you crawl a site, Ultimo Bots does not blindly follow every link. It works in a clear order so it lands on real content pages and ignores assets and boilerplate.
- Sitemap first. The crawler checks your
robots.txtand the usual sitemap locations (such as/sitemap.xmland/sitemap_index.xml). If it finds a sitemap, it uses that list of pages as the starting point. - Otherwise it follows links. If there is no sitemap, it starts at your homepage and follows links it finds, going a couple of levels deep into the site.
- Same site only. It stays on your domain. Links to other websites, social profiles, and external tools are skipped.
- Content, not files. It skips images, videos, stylesheets, scripts, and downloadable files such as PDFs, plus common privacy and legal-notice pages. To add a PDF or other file as knowledge, use Documents instead.
https://www.your-company.com/blog) or paste extra URLs manually. Every page you add still counts toward your plan's knowledge-source limit.Page status
Each page shows a status icon, and a date that tells you when the page was last read successfully. Reading happens in the background, so a page may sit in the scraping state for a short while before it is ready.
| Status | What it means |
|---|---|
| Scraping | The page is being read right now. It shows a spinner and cannot be previewed or selected until it finishes. |
| Completed | The page was read successfully and its text is part of your agent's knowledge. You can preview it or rescrape it. |
| Failed | The page could not be read, for example because it was unreachable or had no usable text. The reason is shown next to the page. Ultimo Bots tries it automatically up to three times. After that you can select it and click Rescrape, or remove it. A failed page does not count toward your limit. |
Preview the extracted content
Knowing a page was added is not the same as knowing what your agent read from it. Click the View Data button on any completed page to open the exact text Ultimo Bots extracted. This is the single most useful check before you trust an answer.
- If the preview shows the real page text, the agent has what it needs.
- If the preview is mostly empty, very short, or full of cookie-banner or menu text, the page probably needs JavaScript to load, or its content sits behind something the crawler cannot reach. Rescrape it, or add the same information as a document or Q&A.
The search box at the top of the list filters by URL, which is the fastest way to find a single page when you have crawled a large site.
Keep pages up to date
Your website changes over time, but the text Ultimo Bots stored does not change by itself. There are three ways to bring it up to date. The first two work on every plan.
- Rescrape chosen pages by hand.
- Check for updates to scan your site once and review what changed.
- Automatic rescraping on a monthly or daily schedule (Boost and Ultimo).
Rescrape selected pages
Tick one or more pages (or a whole domain or folder) and click the Rescrape button above the list. The pages return to the scraping state and are read again in the background. Pages that are still being read cannot be selected.
Check for updates
Click Check for updates next to Add Links. There is no address to type: Ultimo Bots scans the websites you entered under Websites to keep updated (see below), otherwise your agent's website address, otherwise the domains of the pages you already have (up to five websites per check). It then shows what it found in three groups:
- Add new pages on your site that your agent does not know yet.
- Refresh existing pages that are still on your site. Pages whose text has not changed are skipped automatically, so this is quick.
- Remove pages that no longer exist. This group starts unticked, and removal asks for a separate confirmation. If part of your site could not be read completely, removals are held back and nothing is deleted.
Nothing changes until you click Update knowledge (N pages).
Next to the button, a freshness line tells you when your website knowledge was last refreshed: Website knowledge refreshed N days ago, refreshed today, or not refreshed yet. From 45 days on it turns into a warning, even if automatic rescraping is on, because a schedule that has not refreshed anything for that long is not doing its job (for example after a plan change, see below).
Automatic rescraping
Automatic rescraping reads your website again on a schedule, so your agent keeps answering from current content without you coming back to do it by hand. It can also add pages that are new on your site and remove pages that have disappeared.
| Plan | Automatic rescraping |
|---|---|
| Free, Smart | Not included (Rescrape and Check for updates still work) |
| Boost | Monthly |
| Ultimo | Monthly or daily |
Turn it on
Switch on Activate automatic rescraping
The switch sits above the list of pages. On the Free and Smart plans, switching it on opens a Premium Feature dialog that shows Boost+ as the required plan and an Upgrade Now button instead.
A schedule is created
The first time you switch it on, Ultimo Bots creates a Monthly schedule with every page you already have ticked, and with adding new pages and removing old pages both switched off. The settings dialog then opens so you can adjust it.
Adjust and save
Set the options described below and click Save. To change them later, click the sliders button next to the switch. It only shows while the switch is on.
The Automatic Rescraping Settings dialog

| Setting | What it does |
|---|---|
| Frequency | Monthly or Daily. Daily needs the Ultimo plan; on Boost it is greyed out and marked Ultimo plan required with an Upgrade button. |
| Websites to keep updated | One Website address per field, for example yourcompany.com and shop.yourcompany.com. Click the button (Add another website) for more and the icon to remove one. An address can also be a section, such as yourcompany.com/blog; then only pages under that path belong to it. www. or no www. makes no difference. Up to 20 websites are processed. |
| Scan for New URLs During Rescraping | Switch Fetch New Links. When on, each run also adds pages that are new on the websites above. |
| Remove Old Links During Rescraping | Switch Remove Old Links. When on, each run deletes pages that are no longer found on the websites above and are gone (not found, or redirected elsewhere). |
| Other pages | A checklist of your pages that are not under any website above, with a search box and Select All. Tick the ones that should be refreshed as well. It only appears when there are such pages. |
Cancel closes the dialog without changes; Save stores the schedule.
When it runs, and what your plan has to do with it
- Monthly runs on the 1st of each calendar month.
- Daily runs every day.
- Your plan is checked at every run. A monthly schedule needs Boost or Ultimo, a daily schedule needs Ultimo. If the plan no longer matches, the run is skipped. After a downgrade from Ultimo to Boost, a daily schedule therefore stops running: open the settings and switch it to Monthly.
What gets refreshed
Each run reads again:
- every page you have under a website in Websites to keep updated, and
- every ticked page under Other pages.
What happens to each page
| Case | What a run does |
|---|---|
| Existing page | Read again. If its text has not changed, only the date next to it updates. If the text changed, the page is indexed again and the new content replaces the old one. |
| New page (with Fetch New Links) | Each website above is crawled (up to 2,000 links), and every page under it that your agent does not know yet is added and read. These pages carry the description Auto-discovered during rescraping and count toward your knowledge-source limit like any other page. |
| Removed page (with Remove Old Links) | Deleted only if it sits under a website above, the crawl no longer finds it, that crawl was healthy (it worked, found pages, and was not cut off at the 2,000-link limit), and the page itself is gone (it answers "not found" or redirects to another page). A page that still opens is kept. If a crawl looks incomplete, nothing is deleted for that website. Pages under Other pages are never removed automatically. |
| Page that cannot be read | Marked as failed with the reason shown, and tried automatically up to three times. It is not deleted. |
What you see after a run
- The status icon of each page updates while it is read.
- The date next to each page shows its last successful refresh.
- The freshness line next to Check for updates shows when your website knowledge was last refreshed.
- There is no e-mail after a scheduled run. Check the page list or the freshness line instead.
Remove pages
To drop pages, tick them (or a whole domain or folder) and click the trash button above the list. There is no delete button on the individual rows. A confirmation step guards the deletion. Removing a page deletes its stored text and frees a slot against your knowledge limit, so the agent immediately stops using that page in its answers. Removal is permanent. If you only want to update a page rather than drop it, rescrape it instead of removing and re-adding it.
What it does not do
Website links are powerful, but the crawler reads public pages the way an ordinary visitor would. A few honest limits are worth knowing before you rely on it.
Next steps
Round out your agent knowledge with the other sources in the knowledge base: