Add Website Data Sources
Give your AI Chatbot content to answer from: add a website address, a sitemap, a list of pages, files, pasted text or Q&A pairs, then start indexing each source yourself.
An AI Chatbot answers from the sources you add to it. A source is a place Flidu reads text from: a page on your website, a sitemap, a list of page addresses, an uploaded file, pasted text, or question-and-answer pairs. This guide covers each source type, how to add one, and how to get it indexed so Ask AI can use it.
Where To Add A Source
In the dashboard, open AI Chatbots, choose your assistant, and open the Data Sources tab. Select Add Source. A panel titled Add a Source opens and lists the source types. The same panel is used in step 2, Data Sources, when you create your AI assistant.
The workspace Owner and Admins can add, edit and remove sources. Other members can view them.
The Source Types
| Source Type | What You Enter | Good To Know |
|---|---|---|
| Website URL | One public web address | Follow links on this site is on by default: Flidu also reads the pages this page links to on the same site, one level deep. Switch it off to index that one page only. |
| Sitemap | The public address of an XML sitemap | Flidu uses the sitemap to discover pages. |
| Manual URL List | One page address per line | Up to 25 pages per list, all on the same website. |
| File Upload | A .txt, .md, .csv or .html file | Up to 2 MB each. The readable text is indexed and the original file is not stored. |
| Manual Text | Text pasted into the Content box | Between 40 and 100,000 characters. |
| Manual Q&A | Your own question-and-answer pairs | From AI Chatbot Starter. A question can be up to 300 characters and an answer up to 2,000. The assistant answers from the pairs like any other source. |
Every source also has an optional Source name so you can recognise it later. Web addresses must be public http or https pages. Addresses that include a username and password, and private or local addresses, are refused.
Add A Source And Start Indexing
- On the Data Sources tab, select Add Source.
- Choose a source type, for example Website URL.
- Fill in the field for that type: the address, the list of pages, the file, the text or your Q&A pairs.
- Select Add source. The source appears as a card with the status Draft.
- On the source card, select Start indexing.
A website source is queued first: its button reads Indexing queued and then Indexing pages. Files, pasted text and Q&A pairs are indexed directly when you select Start indexing. Each source card shows three counts, Found, Indexed and Not added, and, once the source has been indexed, the date it was last indexed. If a source could not be indexed, the card offers Retry indexing.
How Much Of Your Site A Source Covers
A Manual URL List indexes exactly the pages you list, within your indexed-pages limit. Website URL and Sitemap sources index a limited batch of pages per crawl, not your whole site, so always check the Indexed Content list on the same tab to see which pages were read. Review Indexed Pages explains that list. To make sure particular pages are covered, name them in a Manual URL List.
When Flidu indexes a page it reads the main page text and leaves out what sits in the page’s header, navigation, footer and sidebar elements. A fact that appears only in your header or footer, such as a phone number or opening hours, should also be written in the body of a page. Flidu reads the page as your server sends it, so text that only appears after scripts run is not read, and pages that your robots.txt file disallows are skipped.
Source Limits By Plan
Limits are counted per assistant. Website URL, Sitemap, Manual URL List and Manual Text sources share one count. Files and Manual Q&A entries each have their own.
| AI Chatbot Plan | Website, Sitemap, URL List Or Text Sources | Files | Manual Q&A Entries | Indexed Pages |
|---|---|---|---|---|
| Free | 1 | 1 | Not included | 50 |
| Starter | 3 | 5 | 10 | 250 |
| Pro | 10 | 10 | 25 | 500 |
| Business | 25 | 25 | 50 | 1,000 |
For Agency and Enterprise, see Pricing. Website pages count toward the indexed-pages limit of the assistant; AI Chatbot Allowances explains how the allowances work. When a count is used up, the Add a Source panel tells you, and a new source of that kind is not added until you remove one or upgrade AI Chatbot.
Keep Your Sources Current
Flidu re-reads a website source only when you select Recrawl on its card. After a crawl, the card shows when Recrawl is available again, and each plan has a monthly recrawl allowance that the assistant’s Usage tab counts. If a refresh does not finish, the previously indexed version stays in use.
- Edit on a source card reopens its form. A source that is queued or being indexed is read-only until it finishes.
- A file’s card shows Manage in place of Edit. A file cannot be edited in place: remove it and upload the new version.
- Remove, then confirm, deletes a source from the assistant together with its indexed content. A source cannot be removed while it is queued or being indexed.
If A Source Does Not Index
- The card still says Draft. Indexing has not been started. Select Start indexing.
- The card says Needs attention. Check that the pages are public and reachable, then select Retry indexing.
- Fewer pages than you expected. Open Indexed Content and use the Ignored & Skipped and Failed filters to see the reason for each page.
- Add Source is unavailable inside the Create AI Chatbot flow. Once indexing has begun, add further sources from the assistant’s Data Sources tab instead.
For a step-by-step check, see My Pages Are Not Being Indexed.