RAG over SharePoint and Confluence: Architecture, Effort and Timeline

A RAG chatbot over SharePoint and Confluence is mostly a connector and permissions project with a language model at the end. Retrieval and generation work much as in any RAG system. The effort goes into four things: pulling content incrementally through Microsoft Graph and the Confluence REST API without getting throttled, turning Office files and Confluence storage format into clean text that keeps its structure, reproducing who may see what at query time, and handling deletes, moves and permission changes quickly enough that the bot never quotes a page someone has just locked down. A focused first release over selected sites and spaces usually takes 10 to 16 weeks. Full coverage of a large tenant with strict security trimming takes longer, and your permission model drives that more than the choice of model.

This article covers what is specific to these two sources. For the general pipeline (embedding models, vector stores, reranking, prompts), see our guide to building a RAG chatbot for internal documents. API limits and product features were checked against vendor documentation as of October 2026; both vendors change them often, so verify before you size a project.

Why are SharePoint and Confluence harder than a folder of PDFs?

A folder of PDFs is flat, static and has one access rule. These two systems are none of that.

Confluence content is a tree, and the parent carries meaning: a page called “Rollout” under “Project Falcon / 2025 / Phase 2” says nothing on its own. Pages have versions, often hundreds, and only the current one should normally be indexed. Much of the useful content sits in macros: include macros that pull text from another page, expand sections, page properties tables that act as a small database, Jira issue lists, code blocks. Attachments hang off pages and inherit their visibility.

SharePoint is several storage types under one name. Document libraries hold Office files and PDFs with version history. Modern site pages are stored as web part data, not documents. Lists hold structured records people use as small databases: FAQs, contract registers, product catalogues. Every Microsoft Teams team has a SharePoint site behind it, private and shared channels get sites of their own, and OneDrive is SharePoint with one personal site per user. Microsoft Loop adds another layer: Microsoft’s Purview eDiscovery documentation states that Loop workspaces, Copilot Pages and Copilot Notebooks store content in SharePoint Embedded containers rather than ordinary team sites. Loop content is still awkward to reach programmatically, and for most first releases it is reasonable to exclude it and say so.

Then there are permissions. Both systems let people restrict individual items below the site or space level, and both have sharing mechanisms that grant access outside the group structure. A chatbot that ignores this is a data leak with a friendly interface.

Connecting to SharePoint through Microsoft Graph

Microsoft Graph is the API to use for SharePoint Online and OneDrive. Microsoft’s guidance on avoiding throttling in SharePoint Online recommends it over CSOM and the older SharePoint REST API, because Graph calls have a fixed cost and the older APIs usually consume more.

The connector starts with an inventory of in-scope sites and, per site, their document libraries (drives in Graph terms) and lists. Keep this inventory as data and refresh it, because site owners will add libraries during the project.

Incremental sync with delta queries

The core of the sync is driveItem: delta. The first call on /drives/{drive-id}/root/delta enumerates the library in pages and ends with an @odata.deltaLink; calling that link later returns only what changed. Microsoft’s reference for the function lists behaviours a connector must handle:

• Deleted items arrive with a deleted facet and must be removed from the index.
• The feed shows each item’s latest state, and the same item can appear more than once; use the last occurrence.
• parentReference has no path, because renaming a folder does not return its descendants. Track items by ID, never by path.
• If a token is stale or server state has changed, the service returns 410 Gone with a resync code and a URL for a fresh enumeration. For a read-only index that means a full re-scan and reconciliation.

SharePoint lists have their own listItem: delta. Microsoft also notes that delta is the only reliable way to enumerate a whole library, since paging through folder children can miss items written during the scan.

Change notifications

Webhooks cut polling. Graph change notifications support driveItem on a drive root and SharePoint list resources, and according to Microsoft’s change notifications overview, subscriptions on both expire after at most 42,300 minutes (just under 30 days), so you need a renewal job. A notification only says that something changed; the connector then calls the delta link. Microsoft gives an average latency under one minute and a maximum of six hours for both resources, so a scheduled delta run every few minutes stays in the design as the safety net.

Throttling

SharePoint Online throttles on resource units. Microsoft’s throttling guidance charges 1 unit for a single-item request, a delta call with a token or a file download, 2 for multi-item queries and writes, and 5 for any permission operation, including $expand=permissions. The default budget per app per tenant, for tenants with up to 1,000 licences, is 1,250 units per minute and 1,200,000 per day, rising with licence count. Throttled calls get HTTP 429 or 503 with a Retry-After header that must be honoured; ignoring it extends the throttling, and persistent offenders can be blocked. Microsoft also asks integrators to identify traffic with a User-Agent like NONISV|CompanyName|AppName/Version and says such traffic is prioritised.

Two consequences follow. The initial crawl of a large tenant is bound by the daily budget, so it runs for days and belongs outside business hours in the tenant’s time zone. And permission reads cost five units each: a design that reads permissions for every file on every sync hits the ceiling long before content does.

Least privilege with Sites.Selected

Sites.Read.All lets an app read every site in the tenant, including all OneDrives, and many security teams refuse that for a chatbot. Microsoft’s overview of selected permissions describes the alternative: Sites.Selected, Lists.SelectedOperations.Selected, ListItems.SelectedOperations.Selected and Files.SelectedOperations.Selected. Consent alone grants nothing. An administrator then assigns a role (read, write, owner or fullcontrol) to the app per site, list or file with POST /sites/{site-id}/permissions, and can revoke it per site.

Sites.Selected with read access per site is the right default for a scoped rollout, but it has a price. Microsoft’s delta documentation says that reliable tracking of sharing changes (the Prefer: deltashowsharingchanges header, combined with deltashowremovedasdeleted and deltatraversepermissiongaps) needs Sites.FullControl.All. Vendors face the same trade-off: Glean’s SharePoint connector documentation asks for Sites.FullControl.All and warns that narrower scopes slow down permission updates. Settle this with your security team in the first week: narrow rights and slower permission sync, or broad rights, faster sync and a harder approval.

Connecting to Confluence: Cloud REST API v2, CQL and Data Center

The Confluence Cloud REST API v2 is the current interface for pages, blog posts, attachments, labels, versions, children and spaces. It uses cursor pagination (follow the next link in the Link header) and returns page bodies via the body-format parameter, usually storage (XHTML-based storage format) or atlas_doc_format (JSON). The v1 expand parameter does not apply.

Search is the gap. As of October 2026 there is no v2 search group, and CQL still runs through the v1 endpoint /wiki/rest/api/search; Atlassian staff have said in the developer community that it will only be deprecated once a v2 equivalent exists. CQL is what a connector uses to find changes since the last run:

type in (page, blogpost, attachment)
  and space in ("HR", "ENG", "QM")
  and lastmodified >= "2026-10-08 06:00"
order by lastmodified asc

Reading page restrictions may also still require a v1 endpoint (/wiki/rest/api/content/{id}/restriction), depending on your app type; check the current reference before committing to a design.

Confluence Cloud has no tenant-wide webhook for REST integrations comparable to Graph’s. Forge apps can subscribe to product events such as avi:confluence:updated:page and the trashed and restored events, but Atlassian’s public issue tracker has reports of delayed, out-of-order delivery and an open request for permanent-deletion events. Use events as a hint, a scheduled CQL sync as the source of truth, and a periodic full reconciliation to catch purges.

Atlassian rate limits

Atlassian enforces a points-based quota for Jira and Confluence Cloud REST APIs from 2 March 2026. According to its rate limiting documentation, it applies to Forge, Connect and OAuth 2.0 (3LO) apps, while API-token traffic stays under the older burst limits. Each request costs a base point plus points per returned object: one for pages, spaces and attachments, two for users, groups and permissions. The default tier is a global pool of 65,000 points per hour; a per-tenant tier scales with plan and users up to 500,000 points per hour and requires Atlassian review. Quotas reset at the top of each UTC hour, and once spent, every request is denied until the reset. Separate burst limits are stricter on permission, search and admin endpoints. Responses carry X-RateLimit-Remaining and X-RateLimit-NearLimit, and a 429 comes with Retry-After.

So: request only the fields you need, fetch bodies only for changed pages, spread the initial crawl across hours, and decide early between an OAuth app and a technical account with an API token, because that choice changes which limits apply.

Confluence Data Center

Data Center uses the older REST API (/rest/api/content, expand, offset paging), with CQL via /rest/api/content/search. Rate limiting exists but is set by the customer’s admin, per Atlassian’s page on improving instance stability with rate limiting, so you agree a budget instead of reading it from documentation. The real work is network access: the connector runs inside the customer network or reaches the instance via VPN or private link.

Plan for change. Atlassian stopped selling Data Center to new customers on 30 March 2026, and the products reach end of life on 28 March 2029, when licences become read-only. Build the Confluence connector as a swappable adapter so a later move to Cloud does not mean rewriting the pipeline.

Permission-aware retrieval: the part that decides whether you can go live

Every chunk in the index has to carry enough information to decide, at question time, whether the asking user may see it. There are two basic approaches and a combination.

We recommend the hybrid for most enterprise deployments. Late binding alone fails on broad questions, where the top 50 candidates come from a restricted HR space and the user gets an empty answer.

What has to be synced

In SharePoint, Graph returns permissions per item, covering Entra ID users and groups, SharePoint site groups and sharing links. Each must resolve to principals your filter can match. Entra ID groups expand through /groups/{id}/transitiveMembers, which flattens nesting, and the group: delta function tracks membership changes. SharePoint site groups (owners, members, visitors) are not Entra groups; their membership is read from the site and often contains Entra groups in turn. Sharing links are the hard case: an “anyone in the organisation with the link” link means that whoever receives it can open the file. Many teams treat such files as visible only to explicitly granted principals, which is stricter than SharePoint. Decide this deliberately and write it down.

Because permission reads cost five resource units, read them only where something changed. The Prefer: hierarchicalsharing header on delta returns sharing information only for roots of permission hierarchies and items whose sharing changed.

Confluence has three layers: space permissions, page view restrictions and inheritance. Atlassian’s documentation on page restrictions states that a view restriction is inherited by all descendants, and a page with its own restriction under a restricted parent requires the user to satisfy both. A connector that reads only the page’s own restriction will expose children of restricted parents. This happens in practice: Microsoft’s admin documentation for its Copilot connector for Confluence Cloud stated that parent page permissions are not considered, and an issue in Atlassian’s public tracker describes child pages of restricted parents surfacing through it. Microsoft’s roadmap lists hierarchical ACL support for that connector, so check its current state, but the lesson for a custom build stands: compute effective permissions by walking the ancestor chain. Confluence groups also need mapping to Entra ID; with SCIM provisioning through Atlassian Guard this is usually clean, without it you maintain a lookup table.

At question time the backend needs the user’s full principal set across Entra ID, SharePoint site groups and Confluence. Expanding it live on every question is slow, so cache it per user for minutes, not hours, and invalidate on group delta changes.

The oversharing problem

Permission-aware retrieval enforces the permissions you have, not the ones you meant to have. Microsoft 365 Copilot made this visible: because it answers from everything a user can technically open, years of broad sharing links and “Everyone except external users” sites became searchable in plain language. Microsoft responded with a Copilot deployment blueprint whose first pillar is remediating oversharing, using SharePoint Advanced Management (data access governance reports, site access reviews, Restricted Content Discovery) and Microsoft Purview sensitivity labels.

A custom chatbot inherits the same problem. Start from an allow-list of sites and spaces rather than the whole tenant, which Sites.Selected enforces technically, and exclude content above a chosen sensitivity label unless the use case needs it. An oversharing report before go-live is cheap.

Parsing: turning storage format and Office files into usable text

Confluence storage format is XHTML with Atlassian namespaces: ac:structured-macro for macros, ac:parameter for their settings, ri:page and ri:attachment for references. The parser needs a rule for every macro in use:

Count macro usage across in-scope spaces before building. A few dozen types usually cover nearly everything, and Marketplace macros often have no text in storage format at all, only parameters.

On the SharePoint side, DOCX, PPTX and XLSX parse reliably; the work is preserving structure (Word heading styles, slide titles, sheet names, table headers). PDFs range from clean exports to scanned contracts that need OCR, and PDF tables cause most extraction errors. Modern pages come from Graph as a canvas of web parts; text web parts parse cleanly, custom ones often keep their content elsewhere. Lists are structured data: indexing rows as labelled text works for FAQ lists, but a contract register with 20,000 rows is better served by a tool that queries the list than by retrieval.

Chunking by page structure

Fixed-size chunking throws away what makes these sources useful. Chunk along the document’s own structure: one chunk per heading section, split long sections, merge tiny ones into their parent, and prefix each chunk with its context path so the embedding and the model see where it came from:

[Confluence | Space: Quality Management | Path: Processes > Supplier audits > 2026 checklist]
[Section: On-site audit > Documentation review]
Last modified: 2026-09-14 | Labels: iso-9001, supplier, checklist
...section text...

Word heading styles define sections; slides become one chunk each with title and notes; spreadsheets one chunk per logical table with headers repeated. Attachments get their parent page’s path, which is often the only thing that makes “Anlage_3_final_v2.pdf” retrievable.

Metadata, freshness, deletes and moves

Store on every chunk: source system, site or space, full path, author and last modifier, created and modified dates, version, Confluence labels or SharePoint content type and managed metadata, sensitivity label, language, a stable source ID and the URL for citations.

Freshness matters more in wikis than in archives, because outdated pages are rarely deleted. Two pages called “Travel policy”, from 2021 and 2026, will both match. Boost recent content when relevance is close, show the last-modified date with each citation, exclude archived spaces, and tell the model to name the conflict and cite both dates when sources disagree.

Deletes and moves are where connectors fail quietly. Graph delta reports deletions explicitly. Confluence moves pages to the trash, and a later purge may produce no event at all. A move changes the path and, in Confluence, possibly inherited restrictions, so it must trigger a permission recomputation for the whole subtree. A nightly reconciliation that lists all IDs per space and drive and deletes whatever no longer exists catches what incremental sync misses.

German and English content in one index

German process documentation, English engineering pages and mixed pages often sit side by side. Three choices handle this. Use a multilingual embedding model so a German question finds an English page; BGE-M3 (Chen et al., 2024, Beijing Academy of Artificial Intelligence) and multilingual E5 (Wang et al., 2024, Microsoft) were built for cross-lingual retrieval. Add BM25 keyword search with a German analyser that splits compounds, so “Reisekostenabrechnung” meets “Reisekosten”, and because part numbers and form names are where vector search is weakest. And store each chunk’s language, answer in the language of the question and quote sources in the original. Test cross-lingual questions explicitly; that is where surprises appear.

Architecture overview

The components are vendor-neutral; each box can be managed or self-hosted, depending on your hosting rules.

[SharePoint Online / OneDrive]          [Confluence Cloud or Data Center]
   Graph: delta + webhooks                 REST v2 + CQL (v1 search) + events
            |                                          |
            v                                          v
   +-------------------- Connector workers ------------------------+
   | inventory | change detection | fetch | throttling + retry     |
   | ACL reader (permissions, restrictions, ancestors, groups)     |
   +---------------------------------------------------------------+
            |                                 |
            v                                 v
   [Raw store: original + version]    [Identity service: Entra ID,
            |                           site groups, Confluence groups]
            v
   [Parsing + OCR] -> [Structure-aware chunking] -> [Embedding]
            |
            v
   [Search index: vectors + BM25 + metadata + allowed_principals]
            ^
            |  filter by user principals, recency boost, language
   [Query service] <- [Chat UI / Teams app] <- user (Entra ID SSO)
            |
            v
   [Rerank] -> [Live permission check on cited sources] -> [LLM]
            -> answer with citations + logs, feedback, evaluation store

The query path in pseudocode:

def answer(question, user):
    principals = identity_cache.get_or_expand(user.id)   # user + transitive groups
    candidates = index.hybrid_search(
        query=question,
        filter={"allowed_principals": principals, "status": "current"},
        boost={"last_modified": "recency"},
        top_k=50,
    )
    ranked = reranker.rank(question, candidates)[:8]
    allowed = [c for c in ranked if source_acl.can_read(user, c.source_id)]
    if not allowed:
        return no_answer_message(question)
    return llm.generate(question, context=allowed, cite=True)

Two details get forgotten. Users must sign in with the identity the permissions refer to, which means Entra ID single sign-on plus a mapping to Confluence accounts. And logs contain questions and retrieved text, so they need the same access controls and retention as the content.

Hosting in the EU, GDPR and the works council

Three hosting questions matter: where the index and content copies live, where the model runs, and who can read the logs. A custom build can put all three in an EU region or on your own servers, with an open-weight model in the same environment if your policy rules out external model APIs. Microsoft completed its EU Data Boundary in February 2025, and Atlassian offers data residency for Confluence Cloud, so the sources can already be EU-resident; copying their content into a US-hosted service undoes that. Our LLM hosting services page covers the options.

Under GDPR the index is a new copy of personal data: author names, mentions, HR pages, meeting notes. Document purpose and legal basis, make deletions in the source reach the index within a defined time, sign processing agreements along the chain, and consider a data protection impact assessment if HR spaces are in scope. Under the EU AI Act, an internal knowledge assistant is generally not high-risk unless it feeds decisions about employees, which falls under the employment category in Annex III; the AI literacy duty in Article 4 has applied since February 2025.

In Germany, bring in the works council (Betriebsrat) early. Under § 87(1) no. 6 BetrVG it has a co-determination right over technical systems objectively suitable for monitoring behaviour or performance, whatever the employer intends. A chatbot that logs who asked what qualifies. The Hamburg Labour Court’s decision of 16 January 2024 (24 BVGa 1/24), which found no such right when staff used ChatGPT through private accounts, is often quoted, but it turned on the employer having no access to usage data; commentators note that a company-provided tool with company accounts is a different case. A works agreement on logging scope, pseudonymisation, retention and a ban on performance evaluation is the normal route, and starting it in the first weeks keeps it off the critical path.

Buy or build: Microsoft 365 Copilot, Copilot Studio, Atlassian Rovo, Glean

Off-the-shelf products handle this problem well, and for many companies one of them is the right answer.

Buying is usually better when most content sits in one ecosystem, you are committed to that vendor’s cloud, and a general assistant is the goal. A custom build is justified when:

• knowledge is split across both systems and neither vendor’s connector to the other meets your permission requirements;
• Confluence or SharePoint runs on-premises, or content may not leave your infrastructure;
• special formats carry the value: drawings, scanned forms, page properties used as data, large lists, custom macros;
• you need control over the model (an EU-hosted open-weight or fine-tuned model), ranking rules or citation format, for example for regulated quality documentation;
• the assistant is part of a larger application or agent workflow rather than general employee search.

A middle path is common: Copilot or Rovo for general search, plus a narrow custom assistant for one high-value domain. Our article on fine-tuning versus RAG covers when a domain model is worth adding.

Evaluation: how to know it works before employees do

Build the test set before the pipeline is finished: 150 to 300 questions written by people from the departments whose content is indexed, each with expected source pages and a reference answer. Include the cases that break these systems: answers in attachments, in tables, under restricted parents, on recently moved pages, in the other language, or with an outdated and a current version.

Measure retrieval and generation separately. For retrieval, check whether the expected page is in the top 5 and top 10, broken down by source, space and language. For answers, grade faithfulness and correctness with human reviewers or an LLM judge calibrated on a human-graded sample; the RAGAS framework (Es et al., presented at EACL 2024) defines common metrics. Add a permission suite in which test users with different group memberships ask questions whose answers sit in restricted content. The pass criterion there is zero leaks, not a percentage. Rerun everything after each change to parsing, chunking, model or prompt.

After launch, review unanswered questions and negative feedback weekly. Often the fix is in the content, not the pipeline: three contradictory pages, and someone has to decide which is right.

Project phases and timeline

For a first production release over defined sites and spaces, with permission-aware retrieval and a few hundred to a few thousand users:

With overlap, that is roughly 10 to 16 weeks for a focused scope, and 5 to 8 months for a whole tenant, Data Center behind a firewall, heavy OCR or several user groups with different rules. What moves the timeline most, in rough order: permission complexity (broken inheritance, sharing links, deep Confluence restrictions, unlinked identities); approval lead times for admin consent, Atlassian app approval, security review and the works agreement, which is calendar time rather than work and the most common cause of slippage; content condition (scans, Marketplace macros, duplicates); volume against throttling budgets; and on-premises hosting of model and index.

Checklist for IT before kick-off

• Entra ID app registration with agreed Graph permissions (Sites.Selected plus per-site grants, or broader), admin consent, certificate-based authentication.
• In-scope SharePoint sites, Teams and libraries, and explicit decisions on OneDrive, Loop, private channels and lists.
• GroupMember.Read.All or equivalent for group expansion, approved by the identity team.
• Confluence Cloud: admin approval of the OAuth app or a technical account with API token, in-scope spaces, and whether SCIM links accounts to Entra ID.
• Confluence Data Center: network path, a personal access token for a read-only technical user, the instance’s rate limit settings.
• A test site and test spaces with restricted pages, broken inheritance and sharing links, plus three or four test users with different memberships.
• Which sensitivity labels are excluded from indexing.
• Hosting target (EU region or on-premises) for index and model.
• Contacts in data protection and the works council, with a first meeting date.
• Logging and retention rules for questions, answers and retrieved text.

FAQ

Can a RAG chatbot respect SharePoint and Confluence permissions?

Yes, if each chunk carries its allowed users and groups and queries are filtered by the user’s expanded memberships, with a live check on cited sources before answering. Confluence permissions must be computed through the page’s ancestors, because view restrictions are inherited.

How quickly do changes reach the index?

SharePoint changes typically appear within minutes with change notifications plus frequent delta runs, although Microsoft documents a maximum notification latency of six hours. Confluence is usually synced by CQL every few minutes, and a nightly reconciliation removes deleted and purged content.

Do we need Sites.Read.All?

No. Sites.Selected with read access per site is enough for content. Tracking permission changes efficiently at scale needs broader rights, so it is a trade-off your security team should make knowingly.

Does it work with Confluence Data Center or SharePoint Server?

Yes. The connectors use the older REST APIs and run inside your network, and index and model can run on your infrastructure. Atlassian ends Data Center support in March 2029, so keep the Confluence connector replaceable.

Are Microsoft 365 Copilot or Rovo not enough?

Often they are, when knowledge sits mainly in one ecosystem and the vendor’s model and hosting are acceptable. A custom build pays off when content spans both systems, runs on-premises, includes special formats, or when you need control over model, ranking and data location.

Does the works council have to agree?

In Germany, usually yes: a chatbot that logs user questions is objectively suitable for monitoring under § 87(1) no. 6 BetrVG. A works agreement on logging, retention and a ban on performance evaluation is the normal route.

Working with AI Superior

AI Superior is an AI engineering company in Darmstadt, founded in 2019 by AI scientists. We build RAG systems, chatbots and AI agents for B2B clients, including the connector and permission work described here, hosted in the EU or on your servers. Our AI integration services cover connecting SharePoint, Confluence and other enterprise systems to language models; our AI chatbot development services cover the assistant itself, from evaluation to rollout. For a private chatbot that answers with document references and runs on-premises or in a private cloud, see AiSuperiorGPT and the custom LLM-enabled chatbot we built for a consulting firm.

Sources and further reading

• Microsoft Graph documentation: “Use delta query to track changes in Microsoft Graph data”, “driveItem: delta”, “Set up notifications for changes in resource data”, “Overview of Selected Permissions in OneDrive and SharePoint”
• Microsoft, “Avoid getting throttled or blocked in SharePoint Online”
• Microsoft, Copilot deployment blueprint for a secure and governed data foundation; Copilot connector documentation for Confluence Cloud; Copilot Studio quotas and limits; Purview eDiscovery data sources
• Microsoft, “Microsoft completes landmark EU Data Boundary”, February 2025
• Atlassian: Confluence Cloud REST API v2 reference, rate limiting documentation, Forge product events, “Page restrictions”, “Improving instance stability with rate limiting”, Data Center end-of-life timeline, Rovo third-party connectors guide
• Glean, SharePoint and Confluence connector documentation
• Chen et al. (2024), “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings”
• Wang et al. (2024), “Multilingual E5 Text Embeddings: A Technical Report”
• Es et al. (2024), “RAGAS: Automated Evaluation of Retrieval Augmented Generation”, EACL 2024
• Arbeitsgericht Hamburg, decision of 16 January 2024, 24 BVGa 1/24
• Regulation (EU) 2024/1689 (AI Act), Article 4 and Annex III