Website Links

Your website is usually the fastest way to teach your agent. Add individual pages or let Ultimo Bots crawl your site, and the text on each page becomes knowledge your agent can answer from. Automatic rescraping then keeps those pages up to date on a schedule.

11 min read Updated

Website links live under Knowledge Base → Website Links in the dashboard. Each link is one page of your site. Ultimo Bots reads the readable text from that page, stores it, and uses it to answer visitor questions. Images, navigation menus, headers, and footers are stripped out, so the agent learns from the actual content rather than the page chrome.

One page in, one source out. Every page counts as one knowledge source toward your plan limit, which is shared across website links, documents, Q&As, and imported files. The bar at the top right of the page shows how many sources you use. At the limit, Add Links offers an upgrade instead. See the Knowledge Base overview for the limit per plan.
The Website Links screen with the Add Links button, the Activate automatic rescraping switch, and the list of pages grouped by domain
The Website Links screen. Pages are grouped by domain and folder, with the Activate automatic rescraping switch above the list.
The Website Links toolbar with Add Links, Check for updates, the refreshed-today note and automatic rescraping switched on
Keep your website knowledge fresh right from the toolbar.

Add pages or crawl your site

Click Add Links to open the add dialog. It has two tabs, depending on whether you want Ultimo Bots to find pages for you or you already know the exact URLs.

  1. Crawl Links (discover pages automatically)

    Enter your site address, for example https://www.your-company.com, or the address of your sitemap in Website Link or Sitemap URL, then click Fetch. Ultimo Bots crawls the site and comes back with the pages it found, split into three groups: Newly found (add them with Add), Already trained (read them again with Rescrape), and No longer available (pages you have that the crawl did not find, which you can Remove). You tick the pages you want in each group. Nothing changes until you confirm.

  2. Manual Input (paste exact URLs)

    Switch to Manual Input and paste one exact page address per Website Link field. Use the button to add more fields. This skips discovery and simply adds the specific pages you list, which is handy when you only want a handful of known pages.

Pasting a sitemap gives the cleanest results. Most sites have one at /sitemap.xml. Starting from the sitemap lets the crawler use your own list of pages instead of guessing from links, which tends to find the right pages faster.

How crawling works

When you crawl a site, Ultimo Bots does not blindly follow every link. It works in a clear order so it lands on real content pages and ignores assets and boilerplate.

  • Sitemap first. The crawler checks your robots.txt and the usual sitemap locations (such as /sitemap.xml and /sitemap_index.xml). If it finds a sitemap, it uses that list of pages as the starting point.
  • Otherwise it follows links. If there is no sitemap, it starts at your homepage and follows links it finds, going a couple of levels deep into the site.
  • Same site only. It stays on your domain. Links to other websites, social profiles, and external tools are skipped.
  • Content, not files. It skips images, videos, stylesheets, scripts, and downloadable files such as PDFs, plus common privacy and legal-notice pages. To add a PDF or other file as knowledge, use Documents instead.
Up to 1,000 pages per crawl. A single crawl in the add dialog gathers up to 1,000 pages. If your site is larger, crawl specific sections (for example https://www.your-company.com/blog) or paste extra URLs manually. Every page you add still counts toward your plan's knowledge-source limit.
Discovery runs in the background. Larger sites can take a couple of minutes to scan. Keep the dialog open while it runs, since closing the window cancels the request. For most pages plain reading is enough, and Ultimo Bots automatically falls back to a full browser render for pages that need JavaScript to show their content.

Page status

Each page shows a status icon, and a date that tells you when the page was last read successfully. Reading happens in the background, so a page may sit in the scraping state for a short while before it is ready.

StatusWhat it means
ScrapingThe page is being read right now. It shows a spinner and cannot be previewed or selected until it finishes.
CompletedThe page was read successfully and its text is part of your agent's knowledge. You can preview it or rescrape it.
FailedThe page could not be read, for example because it was unreachable or had no usable text. The reason is shown next to the page. Ultimo Bots tries it automatically up to three times. After that you can select it and click Rescrape, or remove it. A failed page does not count toward your limit.

Preview the extracted content

Knowing a page was added is not the same as knowing what your agent read from it. Click the View Data button on any completed page to open the exact text Ultimo Bots extracted. This is the single most useful check before you trust an answer.

  • If the preview shows the real page text, the agent has what it needs.
  • If the preview is mostly empty, very short, or full of cookie-banner or menu text, the page probably needs JavaScript to load, or its content sits behind something the crawler cannot reach. Rescrape it, or add the same information as a document or Q&A.

The search box at the top of the list filters by URL, which is the fastest way to find a single page when you have crawled a large site.

Keep pages up to date

Your website changes over time, but the text Ultimo Bots stored does not change by itself. There are three ways to bring it up to date. The first two work on every plan.

  • Rescrape chosen pages by hand.
  • Check for updates to scan your site once and review what changed.
  • Automatic rescraping on a monthly or daily schedule (Boost and Ultimo).

Rescrape selected pages

Tick one or more pages (or a whole domain or folder) and click the Rescrape button above the list. The pages return to the scraping state and are read again in the background. Pages that are still being read cannot be selected.

Check for updates

Click Check for updates next to Add Links. There is no address to type: Ultimo Bots scans the websites you entered under Websites to keep updated (see below), otherwise your agent's website address, otherwise the domains of the pages you already have (up to five websites per check). It then shows what it found in three groups:

  • Add new pages on your site that your agent does not know yet.
  • Refresh existing pages that are still on your site. Pages whose text has not changed are skipped automatically, so this is quick.
  • Remove pages that no longer exist. This group starts unticked, and removal asks for a separate confirmation. If part of your site could not be read completely, removals are held back and nothing is deleted.

Nothing changes until you click Update knowledge (N pages).

Next to the button, a freshness line tells you when your website knowledge was last refreshed: Website knowledge refreshed N days ago, refreshed today, or not refreshed yet. From 45 days on it turns into a warning, even if automatic rescraping is on, because a schedule that has not refreshed anything for that long is not doing its job (for example after a plan change, see below).

Automatic rescraping

Automatic rescraping reads your website again on a schedule, so your agent keeps answering from current content without you coming back to do it by hand. It can also add pages that are new on your site and remove pages that have disappeared.

PlanAutomatic rescraping
Free, SmartNot included (Rescrape and Check for updates still work)
BoostMonthly
UltimoMonthly or daily

Turn it on

  1. Switch on Activate automatic rescraping

    The switch sits above the list of pages. On the Free and Smart plans, switching it on opens a Premium Feature dialog that shows Boost+ as the required plan and an Upgrade Now button instead.

  2. A schedule is created

    The first time you switch it on, Ultimo Bots creates a Monthly schedule with every page you already have ticked, and with adding new pages and removing old pages both switched off. The settings dialog then opens so you can adjust it.

  3. Adjust and save

    Set the options described below and click Save. To change them later, click the sliders button next to the switch. It only shows while the switch is on.

Switching off only pauses. Turning the switch off stops the scheduled runs but keeps your settings. Switch it back on and the same schedule continues.

The Automatic Rescraping Settings dialog

The Automatic Rescraping Settings dialog with Monthly frequency, one website to keep updated, Fetch New Links switched on and the list of other pages
Pick the frequency, the websites to keep updated and what happens to new and removed pages.
SettingWhat it does
FrequencyMonthly or Daily. Daily needs the Ultimo plan; on Boost it is greyed out and marked Ultimo plan required with an Upgrade button.
Websites to keep updatedOne Website address per field, for example yourcompany.com and shop.yourcompany.com. Click the button (Add another website) for more and the icon to remove one. An address can also be a section, such as yourcompany.com/blog; then only pages under that path belong to it. www. or no www. makes no difference. Up to 20 websites are processed.
Scan for New URLs During RescrapingSwitch Fetch New Links. When on, each run also adds pages that are new on the websites above.
Remove Old Links During RescrapingSwitch Remove Old Links. When on, each run deletes pages that are no longer found on the websites above and are gone (not found, or redirected elsewhere).
Other pagesA checklist of your pages that are not under any website above, with a search box and Select All. Tick the ones that should be refreshed as well. It only appears when there are such pages.

Cancel closes the dialog without changes; Save stores the schedule.

When it runs, and what your plan has to do with it

  • Monthly runs on the 1st of each calendar month.
  • Daily runs every day.
  • Your plan is checked at every run. A monthly schedule needs Boost or Ultimo, a daily schedule needs Ultimo. If the plan no longer matches, the run is skipped. After a downgrade from Ultimo to Boost, a daily schedule therefore stops running: open the settings and switch it to Monthly.

What gets refreshed

Each run reads again:

  • every page you have under a website in Websites to keep updated, and
  • every ticked page under Other pages.
New and removed pages need a website address. Finding new pages and removing old ones only ever happens for the websites listed under Websites to keep updated. If you leave that list empty, nothing new is found and nothing is removed, even with Fetch New Links and Remove Old Links switched on. Only the ticked pages are refreshed. To keep a whole site in sync, enter its address.

What happens to each page

CaseWhat a run does
Existing pageRead again. If its text has not changed, only the date next to it updates. If the text changed, the page is indexed again and the new content replaces the old one.
New page (with Fetch New Links)Each website above is crawled (up to 2,000 links), and every page under it that your agent does not know yet is added and read. These pages carry the description Auto-discovered during rescraping and count toward your knowledge-source limit like any other page.
Removed page (with Remove Old Links)Deleted only if it sits under a website above, the crawl no longer finds it, that crawl was healthy (it worked, found pages, and was not cut off at the 2,000-link limit), and the page itself is gone (it answers "not found" or redirects to another page). A page that still opens is kept. If a crawl looks incomplete, nothing is deleted for that website. Pages under Other pages are never removed automatically.
Page that cannot be readMarked as failed with the reason shown, and tried automatically up to three times. It is not deleted.

What you see after a run

  • The status icon of each page updates while it is read.
  • The date next to each page shows its last successful refresh.
  • The freshness line next to Check for updates shows when your website knowledge was last refreshed.
  • There is no e-mail after a scheduled run. Check the page list or the freshness line instead.

To drop pages, tick them (or a whole domain or folder) and click the trash button above the list. There is no delete button on the individual rows. A confirmation step guards the deletion. Removing a page deletes its stored text and frees a slot against your knowledge limit, so the agent immediately stops using that page in its answers. Removal is permanent. If you only want to update a page rather than drop it, rescrape it instead of removing and re-adding it.

What it does not do

Website links are powerful, but the crawler reads public pages the way an ordinary visitor would. A few honest limits are worth knowing before you rely on it.

Pages must be public and reachable over HTTPS. The crawler cannot sign in, so anything behind a login, a password, a paywall, or a members area will not be read. Every page you add becomes an entry straight away, then it is read in the background. A page that cannot be opened or that has no usable text ends up marked as failed, so you can spot it and remove it.
Very JavaScript-heavy pages can come back thin. Ultimo Bots falls back to a full browser render when a page needs it, but some single-page apps and interactive widgets still expose little readable text. Always preview such pages, and if the text is missing, add that content as a document or a Q&A instead.
A crawl is not a full mirror of your site. A crawl in the add dialog collects up to 1,000 pages and goes a couple of levels deep when there is no sitemap, and it skips assets, downloadable files, and common legal pages. Use View Data to confirm that a page you care about was read the way you expect.

Next steps

Round out your agent knowledge with the other sources in the knowledge base: