Crawler URLs

How Crawler works
Each time when visitor visit cacheable page (according to the settings Cacheable Actions, Allowed Pages, Ignored Pages, Max. Allowed Page Depth), extension automatically save page url to file var/log/fpc.log ( for logged in users var/log/fpclogged.log). Every 15 minutes this log is exported into a database, therefore you can see these URLs in the Crawler URLs table (System->Full Page Cache->Crawler URLs). Crawler scans URls from these table, and if Url is not in the cache, he adds them to the cache.
If "Add crawled ulrs directly in database" enabled FPC will add urls directly in database.

To enable the Crawler, go to System > Configuration > Full Page Cache and set Enabled at the Crawler Settings list.

Full list of these urls you can find at System > Full Page Cache > Crawler URLs and System > Full Page Cache > Crawler URLs for logged in users, where:

URL - full url to page
Cache Id - unique identifier for this page
Popularity (number of visits) - number of visits (priority for crawler)

Sort by page type - sorts URLs list by page type. To use advanced settings for sort by page type, follow these steps:

Go to System > Configuration > Mirasvit Extensions > Full Page Cache, open Crawler Settings list.
At the line Sort Crawler urls by set value Custom Order and you will see a Sort by page type option.
Add from the list Cachable Actions with the highest priority (from top to the bottom). There are 6 types of Cachable Actions are available:
- cms/index_index - home page
- cms/index_index+ - home page with GET parameters. For example: http://example.com?data=1&data1=2)
- catalog/product_view - all product pages. For example: http://www.example.com/accessories/eyewear/aviator-sunglasses.html
- catalog/product_view+ - all product pages with GET parameters. For example: http://www.example.com/accessories/eyewear/aviator-sunglasses.html?data=4&data4=5)
- catalog/category_view - all category pages. For example: http://example.com/accessories/eyewear.html
- catalog/category_view+ all category pages with GET parameters. For example: http://example.com/accessories/eyewear.html?limit=24
  
  If option Sort by product attribute is also configured, settings at Sort by page type have higher priority.
  Unassigned Cacheble actions in the list will be sorted by popularity

Sort by product attribute - sorts URLs list by product attributes. To use advanced settings for sort by product attribute, follow these steps:

Go to System > Configuration > Mirasvit Extensions > Full Page Cache, open Crawler Settings list.
At the line Sort Crawler urls by set value Custom Order and you will see a Sort by product attribute option. Select from the list Attribute and set appropriate Value for the sort order. You can set a few values separated by comma. If field Value leave empty, then it will check only attribute existence in the product.
At the Crawler URLs list, you will see product attribute values like 1, 2, 3. Each number response for the number of the Sort by product attribute position.
If option Sort by page type is also configured, settings at Sort by page type have higher priority.
~! To make changes work, you need to remove all URLs at System > Full Page Cache > Crawler URLs and totally flush Magento cache.

Customer group - Customer group for current url (only for "Crawler URLs for logged in users").
Store id - Store id for current url.
Currency - Currency for current url.
Cache Status - indicates the current cache status for this page
- Pending - page not available in cache yet
- In Cache - page already in cache

Depending on crawler cron expression, crawler visits all not cached URLs and fill cache for these URLs.

Was this page helpful?

Your feedback about this content is important.
Let us know what you think.