What is the Keto App crawler?

Simply put, the Keto App crawler scans content from participating websites and builds a time-ordered feed that is synced with Keto App on mobile.

Unless we have invited a blogger to participate and we have received explicit permission we will not crawl a website.

Does it copy content from participating sites?

No, it only extracts data used to index the original content. All recourses (e.g. images) are served by the original website.

How often does it crawl?

The Keto App crawler performs a 2-level scan:

  • A quick scan to capture the latest updates is performed once every 10 minutes for each website. This typically involves a handful of HTTP requests.
  • A full scan to capture all updates is performed once a day for each website. This typically involves scanning all the content.

The crawler throttles all requests to ensure minimal impact on the websites crawled.

How can I identify the Keto App crawler?

You can ensure it's Keto App crawler visiting your website by:

  • The user-agent string: keto-app 01 October 2018, please see https://ketodietapp.com/keto-app/crawler-info.
  • The request ip-address resolves to the same address as the domain KetoDiet.com.

You can use either or both methods to whitelist the crawler if needed. This is only required if you're blocking unknown web clients.

  1. Home
  2. Keto App crawler info