Diggernaut can run JS routines

How to execute JavaScript snippet in the middle of scraping process?

Sometimes, when you scrape something, it may happen that you need to calculate some parameter to do request for some page. There may be a lot of different cases, when you need to do some calc, but most of them can be solved using “eval” command.

Let imagagine that we need to get current time in seconds since Epoch and use it in some URL.
There are few ways to do it:

1:

Lets keel URL value in the variable, evaluate JS code and return result of JS code to the register, to do it you need to use closure and your closure should return value.

2:

Another way to do it is to write value of some JS variable to your digger variable. You can use argument “assign” for it.

3:

And third way to do it is to pass base URL to JS routine as variable and then your closure will return complete URL.

Please note that if you are using digger’s variables in JS routine, you need to use quotes, because they will become literal values before engine evaluate JS routine.
You can of course create more complex functions and even run scripts you extract from script tag on page, but you should understand that it will not work with DOM and your function should not run too long.

If you do too many recursive calls or your routine runs too long you will see following errors:

Leave a Reply

Your email address will not be published. Required fields are marked *