If data isn’t stored in RAM, then plenty of disk space will be required, with adequate RAM and ideally SSDs. It’s important to remember that crawling remains a memory intensive process regardless of how data is stored. The reporting and filters are all instant as well, although sorting and searching at huge scale will take some time. For example, you can scroll through 8 million page titles, as if it was 800. We have a hate for pagination, so we made sure the SEO Spider is powerful enough to allow users to view data seamlessly still. Here are 10 million URLs crawled, of 26 million (with 15 million sat in the queue) for example. The default crawl limit is now set at 5 million URLs in the SEO Spider, but it isn’t a hard limit, the SEO Spider is capable of crawling significantly more (with the right hardware). This is why version 9.0 now allows users to choose to save to disk in a database, which enables the SEO Spider to crawl at truly unprecedented scale for any desktop application while retaining the same, familiar real-time reporting and usability. However, storing data in memory also has downsides, notably crawling at scale. The SEO Spider has traditionally used RAM to store data, which has enabled it to have some amazing advantages helping to make it lightning fast, super flexible, and providing real-time data and reporting, filtering, sorting and search, during crawls. In our last update, we released a new user interface, in this release we have a new and extremely powerful hybrid storage engine.
Our team have been busy in development working on exciting new features. I’m delighted to announce the release of Screaming Frog SEO Spider 9.0, codenamed internally as ‘8-year Monkey’.
0 kommentar(er)
