site spider tool

User 2347821 Photo


Registered User
1 post

I was using the Site Spider Tool yesterday and finding it quite useful. The big annoyance is that once it reports a list of problem urls, I'm unable to edit any pages while the spider result page is open. When I try to copy/paste or export the data to a spreadsheet the information doesn't transfer in a usable form. So, I was forced to leave HTML Editor open and make the corrections in Notepad :rolleyes:. Never the less, the spider was still useful.

However... today when I try to spider any of my sites, the tool give an "ignored/skipped" response on all the urls I enter. What happened?????
User 184085 Photo


Ambassador
1,707 posts

Not a tool I have a lot of experience with, but I was able to spider my site and export the spider results to a CSV file.

Do you have links to your sites so we can take a look??
Volunteering to help :)
http://www.tbaygeek.ca
My HTML play area
http://www.tbaygeek.ca/test/
User 235128 Photo


Registered User
33 posts

I just "discovered" the tool myself and played with it a bit. (Using CC HTML Editor v12.8 build 691)

Really useful tool! Right now, I'm just using the tool to Benchmark a site.

Like the OP, the first thing I noticed was that the spider dialog prevents any further use of the Editor. I guess I can work around this by launching another instance of the Editor, but it would be nice not to have to do this.

Had no issues exporting to CSV.

Wishlist:
    1) It would be nice if the spider could save and import/reload the scan results. For my site, it processed some 17,000 files and it takes quite a bit of time to process all of those pages. It would be very useful if I could save and reload the scan data into the spider.
    2) Right now it seems external links are ignored. It would be nice if the spider would report the server status of the external links. (i.e. 404 or 200) That way I can easily see if any of the external links referenced by my site are "broken".
    3) The opened Spider tool should not prevent the subsequent use of the Editor instance from which it was opened.

Bug:
Sometimes, the spider reports blank/empty Description and/or Keywords when there are, in fact, keywords and descriptions in the appropriate meta tags on the page.

Looking forward to using the Spider and Websight Insight for site cleanup and (hopefully :D) SEO improvements.



User 38401 Photo


Senior Advisor
10,951 posts

With the new Website Insight App being released soon, I am thinking the Spider Tool will probably end up going away as that new app is going to be pretty dang awesome! Watch for it and catch the deals on it which I think may have already been started being emailed. The prices aren't going to be remaining that low so catch it while it's cheap, it's definitely going to be a hot new item :)
User 235128 Photo


Registered User
33 posts

Jo Ann wrote:
With the new Website Insight App being released soon, I am thinking the Spider Tool will probably end up going away as that new app is going to be pretty dang awesome! Watch for it and catch the deals on it which I think may have already been started being emailed. The prices aren't going to be remaining that low so catch it while it's cheap, it's definitely going to be a hot new item :)


Already played with the Website Insight App - and purchased it right away :D

Actually, I'm hoping to see more of a merger of the two apps, Spider and Insight, rather than having the Spider just go away:
    Right now Insight only targets one page url at a time - Spider gets 'em all with one shot.

    I like the Spider functionality where I can click on one of the urls and it opens right up in the browser - Insight does not have this functionality (though you can right click, go to url).

    Insight has a 'broken links' tab - I'll point the app to a page with broken external links and see if they show up in this tab. Right now, there is no indication in Spider as to whether or not an external link is 'broken'.

    In Insight, we can import a previous scan and continue working where we left off. Spider does not have the ability to import a previous scan.

CoffeeCup would have a killer app if they can manage to merge these two together!



Have something to add? We’d love to hear it!
You must have an account to participate. Please Sign In Here, then join the conversation.