REME Indexer — Help

Complete guide to crawling, indexing, managing domains, chat, and plans.

Jump to:Quick start Login / Sign-up The form Your Domains Client Text Chat widget Plans & limits Account Settings Guest rules Admin Server config Troubleshooting

Quick start

Open the dashboard at/.
Paste a URL into theStart URL field (e.g. example.com).
ClickStart Index.
Watch progress in theLive Log overlay.
Once finished, the domain appears inYour Domains — open the AI chat to talk about it.

Tip

Your last Start URL and Client Text are automatically restored from your account (if logged in) or session (if guest).

Login / Create account

Login

Click theLogin button in the header bar.
Enter your email and password, then submit.
The page reloads so Socket.IO reconnects with your auth cookie.

Create account

TheCreate account button is visible when the server allows sign-up ( REME_ALLOW_SIGNUP=true).
Fill in first name, last name, email, and password (min 8 characters).
On success you arelogged in immediately; any guest state (Start URL, Client Text) is migrated to your new account.

Logout

When logged in, the Login button changes toLogout (your name).
Logging out creates a fresh guest session.

Security note

Sign-up can be disabled in production by leaving REME_ALLOW_SIGNUP unset or false.

The form

Start URL

Type or paste any website URL. The input auto-normalises it (addshttps:// if missing, strips trailing slashes, etc.).
This is the "seed" page — the crawler starts here and discovers linked pages.

Start Index button

Sits inline with the URL input for a streamlined workflow.
Click once to kick off a full crawl + index run.
While a job is running, apage blocker overlay appears and the Live Log panel opens automatically.

Live Log

A slide-in panel that streams real-time stdout/stderr from the running job.
Status phases (Crawl, Distill, Embed, etc.) and progress counters are shown inline.
The log auto-scrolls during the run. Close it with the ✕ button after the job finishes.

Under the hood

The form emits index:start over Socket.IO. The server streams index:log, index:phase, and index:progress events. The job resolves with index:done or index:error.

Server-side defaults

Crawl mode (hybrid/static/dynamic) and max pages are configured on the server via environment variables — no form fields needed. See the Server config section.

Your Domains (authenticated users)

After you log in and run at least one crawl, theYour Domains panel appears below the form.

Domain cards

Each card shows asnapshot thumbnail, the domain name, start URL, and last-updated timestamp.
Two action buttons per card:

View | Use — selects the domain, loads its Client Text and chat settings, and opens the settings accordion.
Re-crawl — opens a confirmation modal and re-runs the crawler on this domain. Existing crawl data is replaced, the index is regenerated, and the knowledge pack is rebuilt. Chat history remains intact.

Settings accordion (per domain)

Expanded when you clickView | Use. Contains:

Contact page — set a contact-page path for the domain (saved to the server).
Active Links — extract discoverable links ( domains:links:extract), then select up to 10 active links. Selections auto-save.
Chat Widget Theme — Light, Dark, Girly, Blush, or Console.
Chat Widget Position — Top Left, Top Right, Bottom Left, or Bottom Right.
Welcome Message — customise the first greeting visitors see when the chat opens. Max 300 characters. Leave blank for the default ("Hi! I'm the {domain} AI assistant…"). A live preview shows how the message will look.
Iframe URL — a read-only embed snippet you can copy and paste into any website.
Client Text — the editor moves here when a domain is selected, titled "Client Text (added to {domain})".
Knowledge Pack — loaded on demand; shows the domain's extracted knowledge (title, about, services, FAQs, pricing, etc.).
Logs — view archived extract and index logs in a unified chronological picker.

Plan badge

TheYour Domains header shows a plan badge (e.g. "Free · 1/1 sites").
When usage equals the limit, the badge turnsred.

Client Text (auto-saved)

Optional context or instructions that will be sent with the next crawl run. Use it to guide the scraper or add information not available on the website.

Edits areauto-saved after a short typing pause (~1 second debounce).
Pasting text triggers animmediate save.
Max length:20,000 characters. A character counter is shown below the editor.
TheClear button wipes the text and saves immediately.

Storage

Logged-in users: stored per-domain in MongoDB. Switching domains loads that domain's text.
Guests: stored in the session (ephemeral).

If you go offline, the status shows "Offline (will save when connected)" and the save is queued for reconnection.

Chat widget

When you can use it

The chat iframe isonly rendered for authenticated users.
It needs adomain context — this comes from your last run's Start URL or the domain you selected via "View | Use".

Features

Floating launcher button that opens an AI chat popover.
Header shows the domain name, with a theme selector and close button.
5 themes: Light, Dark, Girly, Blush, Console.
4 positions: Top Left, Top Right, Bottom Left, Bottom Right.
Theme and position are saved per-domain so each site can have its own look.
Welcome message — set a custom greeting in the Settings tab. Leave blank for the auto-generated default. Max 300 characters.

If chat says "no site" or "no domain"

Run the indexer at least once, or click "View | Use" on an existing domain.
The chat widget refreshes its domain automatically after runs — no manual reload needed.

Embedding on your website

Use the Iframe URL from the domain settings accordion to embed the chat widget on any external website.

Plans & limits

Every account belongs to a workspace with a plan tier. The plan determines how many websites you can index and how many chat messages you can send per month.

Plan tiers

Free — 1 website, 200 messages/month, $0, no file uploads
Basic — 1 website, 2,000 messages/month, $29/mo, file uploads: 5 files / 10 MB each
Pro — 5 websites, 20,000 messages/month, $79/mo, file uploads: 20 files / 25 MB each
Scale — 20 websites, 100,000 messages/month, $249/mo, file uploads: 100 files / 50 MB each

Feature flags by plan

Basic+: Analytics, Remove Branding, Custom Themes, File Uploads (5 files / 10 MB)
Pro+: Multi-Language, API Access, Priority Support, File Uploads (20 files / 25 MB)
Scale: White Label, File Uploads (100 files / 50 MB)

What happens when you hit a limit?

The server blocks the action and returns an error code (WEBSITE_LIMIT_REACHED or PLAN_LIMIT_REACHED).
APlan Limit modal appears showing your current plan, usage, and a prompt to upgrade.
The Live Log shows a ⚠️ warning and the status changes to "Limit reached".
Re-indexing domains you already own always passes the capacity check — limits only apply tonew domains.

Account Settings

Click theAccount button in the header bar (visible only when logged in) to open the Account Settings modal.

My Plan tab

Shows your current workspace plan at a glance:

Plan name — Free, Basic, Pro, or Scale.
Billing status pill — colour-coded indicator (free, active, trial, past_due, canceled).
Price — your monthly cost.
Websites bar — how many indexed sites you have vs. your plan limit. Turns red at 100%.
Messages bar — chat messages used this month vs. the monthly allowance.
Features checklist — ✓ / ✗ for each feature flag (Analytics, Remove Branding, Multi-Language, API Access, White Label, Priority Support, Custom Themes).

Upgrade tab

Displays all available plans side by side, sorted by price. Each plan card shows limits, features, and an action button.

Current Plan — disabled button on your active plan.
Upgrade — switches to a higher-priced plan.
Switch — switches to a lower-priced plan (downgrade).

Downgrade protection

If you own more websites than the target plan allows, the switch is blocked. Remove extra domains first, then try again.

Profile tab

Two forms for managing your account data:

Profile form

Email — read-only, shown for reference.
First name / Last name — editable. Saved with the "Save Profile" button. The logout button label updates immediately to reflect your new name.
Newsletter — checkbox to opt in or out of email updates.

Change password

Enter yourCurrent password and a New password (minimum 8 characters).
ClickChange Password to update. The form resets on success.

Keyboard shortcut

Press Escape to close the Account modal at any time.

Guest rules

Domain locking

After a guest starts the first crawl, the session islocked to that registrable domain (eTLD+1).
Subdomains of the locked domain are allowed (e.g. locked toexample.com → you can still run www.example.com or blog.example.com).
Once locked, the Start URL input becomes read-only and shows the locked domain.

Existing domains

If a guest tries to scrape a domain that already exists in the database, the server blocks the run and asks the user to create an account.

What guests cannot access

The "Your Domains" panel
The chat widget
Admin dashboard

Why these restrictions?

Guest locking prevents anonymous re-indexing and abuse. Once a domain is known, scraping state must be tied to a real account.

Admin dashboard

Available at/admin for users with the admin role. An "Admin" link appears in the header bar.

Becoming an admin

Set theADMIN_EMAIL environment variable to an email address.
On server start, that user is automatically promoted to admin.

Dashboard tabs

Overview — stat cards (Total Users, Workspaces, Domains, Messages This Month) and a plan distribution chart.
Users — searchable, paginated table of all users. Promote or demote users to/from admin.
Workspaces — paginated table of all workspaces. Edit plan tier and billing status.
Plans — read-only reference table of all plan tiers with limits and feature flags.

Server configuration

Key environment variables configured inecosystem.config.cjs or your deployment environment.

Crawl settings

CRAWL_MODE — default crawl strategy: hybrid (default), static, or dynamic.
CRAWL_MAX_PAGES — maximum pages per crawl (default: 10).
CRAWL_REQUEST_DELAY_MS — delay between HTTP requests in ms (default: 500).

Server

INDEXER_PORT — listening port (default: 4099).
REME_CHAT_SOCKET_URL — chat backend Socket.IO URL.
OPENAI_API_KEY — OpenAI key for embeddings.
EMBEDDING_MODEL — model name (default: text-embedding-3-small).

Sessions & auth

SESSION_SECRET — express-session secret.
REME_SESSION_COOKIE_NAME — cookie name (default: reme_indexer_sid).
REME_ALLOW_SIGNUP — enable public registration (default: true).
ADMIN_EMAIL — auto-promote this user to admin on startup.

Crawl mode explained

Hybrid — tries a fast static fetch first; falls back to a headless browser when JS rendering is needed.
Static — plain HTTP requests only. Fastest, best for server-rendered sites.
Dynamic — always uses a headless browser (Puppeteer). Slower but handles JS-heavy SPAs.

Troubleshooting

"Disconnected" indicator stays red

Ensure the server is running (defaulthttp://localhost:4099).
Check that cookies are allowed — sessions require thereme_indexer_sid cookie.
If behind a reverse proxy, verifyTRUST_PROXY=true and WebSocket upgrade support.

Start button does nothing / times out

The UI times out after 45 seconds if no ack is received. Check server logs.
Verify MongoDB connectivity — the session store and domain data depend on it.

"Plan Limit Reached" modal

You have hit your plan's website or message quota.
Re-crawling an existing domain doesnot count as a new website.
Contact an admin to upgrade your workspace plan.

Client Text too long

Keep it under20,000 characters. The counter at the bottom of the editor shows current length.

Chat says "no domain"

Run the indexer at least once, or select a domain via "View | Use" in Your Domains.

Guest locked to wrong domain

Guest sessions lock to the first domain you crawl. Create an account to unlock access to any domain.

Still stuck?

Open the browser DevTools Console and look for Socket.IO errors. Then check server logs with pm2 logs domain-indexer.