Home / Questions / 486511
Accepted
NoahVerner
Asked: 2022-09-25 16:09:12 +0800 CST 2022-09-25 16:09:12 +0800 CST

How can I create a dynamic explicit wait using selenium in Python?

  • 772

I created a Python program to webscrape data on a website, when executing it sometimes it threw me a problem , I explain:

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,'/Este_es_el_XPATH')))

I set the above explicit wait to get a particular piece of data from the website.

Most of the time the program ran through the process, it worked just fine.

However, there were a few occasions where the following error was generated:

TimeoutException

Which made my program stop and I had to repeat all the long process that I had already done.

The seconds wait parameter 10is not always effective.

I would like to know if there is a way to set this parameter as a variable that will always get the load time (in seconds) of a fully loaded page, so that I can run the above explicit wait without problems.

You could simply have the process repeat " forcibly " until the element's visibility is finally loaded so that you can scrape the data, such as through the use of tryand blocks exception.

But I don't know what I should write in the block exceptionto repeat what I put in it over tryand over again until the element is visible and can be scraped.

python

1 Answers

  1. Best Answer

    To do it as you say (using the try , except ), you can establish a variable that will fulfill the function of a counter, it will increase until the time you indicate passes (maximum value of the counter), that same variable, always the you'll have to set before you start looking for the first element of all, for example:

    x = 0 # Contador

driver.get(url)

while x < 100: # intentar encontrar el elemento en 10 segundos, sino, continua con el código
    try:
        elemento = driver.find_element_by_name('name')
        elemento.click() # O el evento que desees ejecutar

        x = 0 # Ya encontró el elemento, así que el contador vuelve a 0
        break # Sale del bucle

    except: # en caso de no encontrarlo

        time.sleep(0.1) # Espera 100 milisegundos (decima parte de un segundo), recuerda importar el modulo time: import time
        x += 1 # Aumenta el contador

    As you see, the sleeps have (0.1), this to find it and act faster, but you could also replace them with (1), and also remove a 0 at the beginning of the while (this will slow down the search).

    If in the future another element has to be fetched, it would be the same, only you will no longer need to declare the counter, since it was already set to 0 before.

    I hope and it works for what you are looking for, I am not an expert, but this is a practical way to carry out such a procedure.

    • 0