Skip to main content

How to add URLs to the pages table

To add URLs to pages table, use sources. After the extension is installed, the Visitors' actions source is available and used by default. However, you can also create and use other sources to meet your needs.

Adding URLs with Visitors' actions​

It tracks the pages opened by customers in the browser. New uncached pages will be added to the Pages table, while the popularity counter will increase for pages already added to the table.

If you want to quickly get a list of URLs and do not need popularity data, use the command below:

bin/magento mirasvit:cache-warmer:crawl

The pages obtained in this way will be marked as collected through the Visitors' actions source too. We recommend using this source to add URLs to the Pages table and prioritize warming up the most visited pages by customers.

Adding URLs with Crawler​

To quickly import all URLs of the site, use the Crawler source.

  1. Navigate to System -> Page Cache Warmer -> Sources, then click the Add source button. Enter a source name, set Type to Crawler, and enable the source.

  2. Connect to your server via SSH and clear your store cache:

    bin/magento cache:clean

  3. Run crawler command. Use additional keys for specific queries.

    bin/magento mirasvit:cache-warmer:crawl

    The Crawler runs in multiple cycles. In the first cycle, it will scan the homepage and gather all the links. In the second and subsequent cycles, the crawler will visit the other pages of your site.

    If you have many URLs, the crawling process may take some time. The speed of crawling depends on the server’s performance, page caching time, and varies for each site.

    To prevent the scan from stopping due to an SSH connection drop, use nohup in your command:

    nohup php -f bin/magento mirasvit:cache-warmer:crawl &

    The result of the command will be saved in the file nohup.out.

    The extension may not add a URL if it was excluded using exclusion configuration settings. In this case you will need update your extension settings.

Adding URLs with File source​

To quickly add specified URLs from the site, use the File source.

  1. Prepare a TXT or CSV file with URLs from your store, placing each URL on a new line.

  2. Navigate to System -> Page Cache Warmer -> Sources, then click the Add source button. Enter a source name, set Type to File, add your file from previous step and enable the source.

  3. Connect to your server via SSH and run the synchronization command. Replace 2 with the ID of your file source:

    bin/magento mirasvit:cache-warmer:sync-source --source-id 2

    All previously uncached URLs from your file will be added to pages table. The extension may not add a URL if it was already cached before synchronization or if it was excluded using exclusion configuration settings. In this case you will need clear cache or update your extension settings and run command again.

    Use additional keys for specific queries.

Adding URLs with Sitemap source​

Use this source to add URLs from sitemap to the pages table.

  1. Navigate to Marketing -> SEO and Search -> Site Map and make sure that sitemap.xml is created and contains up-to-date information. If the file does not exist, create it.

  2. Navigate to System -> Page Cache Warmer -> Sources, then click the Add source button. Enter a source name, set Type to Sitemap, choose your sitemap in the dropdown and enable the source.

  3. Connect to your server via SSH and run the synchronization command. Replace 2 with the ID of your sitemap source:

    bin/magento mirasvit:cache-warmer:sync-source --source-id 2

    Use additional keys for specific queries.

    note

    The extension has a separate job that runs once a day and automatically synchronizes File and Sitemap sources. This allows you to automatically add new URLs to the pages table.