What it does
Every so often we visit public restaurant websites, usually the site a restaurant lists on its public business profile, and read the few pages that matter: menus, specials, happy hours and opening hours, along with the images and PDF menus those pages show.
We turn what the restaurant has posted into a consistent format (days, times, items and prices) so people can discover deals near them on our map. We always point back to the restaurant, and we recommend checking with it before you go.
It does not log in, fill in forms, place orders, or get past paywalls, logins or security checks, and it never solves CAPTCHAs. It does not collect information about a site’s visitors.
How often it visits
We usually check a restaurant’s site about once a month, and less often when the restaurant has no specials listed. We may come back sooner when a site’s specials change every week or when a visit fails. Restaurants can ask us to check sooner, or less often: see For restaurants.
How it behaves
- It follows
robots.txt, includingAllow,DisallowandCrawl-delay. It obeys both the*rules and any rules forLucoBot: if either disallows a page, we don’t fetch it. - It loads one page at a time from a site, pauses between pages, and backs off when a server responds with 429 or 503.
- It reads only a limited number of pages per visit and stays on the restaurant’s own site.
- Pages are loaded in a standard web browser, so content that appears through JavaScript can be read the way a visitor sees it. In your server logs our visits look like an ordinary browser.
Controlling access
To stop us from visiting your site, add this to your robots.txt:
User-agent: LucoBot
Disallow: /To slow us down instead (we honor delays of up to 30 seconds):
User-agent: LucoBot
Crawl-delay: 10We never use a copy of your robots.txt older than a day, so changes apply within a day.
Contact
Questions, corrections, or a request to stop visiting your site: hello@seeyathere.com.