Having trouble scraping data from website with Selenium



  • I've tried a couple different routes to get this to work but I've had no luck.

    1)

    driver= webdriver.PhantomJS(executable_path=r'C:\Users\x\Downloads\phantomjs-2.1.1-windows\phantomjs-2.1.1-windows\bin\phantomjs.exe')
    driver.get('https://www.drf.com/race-results')
    
    elements = driver.find_elements_by_xpath("//div[@class='handicappingBottom']")
    

    for item in elements: print(item)

    *Returns an empty list

    I then read up a bit and feel like the webpage isn't fully loading those items so I tried multiple variants of this:

    elements = ui.WebDriverWait(driver, 45).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "handicappingBottom")))
    

    Example of code snippet from webpage:

    Ideally I'm trying to scrape the current horse race locations into a list in a program I'm mocking up. Any help at all would be greatly appreciated. I'm new-ish to python and even newer than that to webscraping. Thanks a ton in advance!



  • Hi I am not sure why you are printing that

    try the below code if you want to get the displayed text from those elements:

    value= driver.find_elements_by_xpath("//div[@class='handicappingBottom']").get_attribute('textContent');
    
    print(value);
    

    if this doesn't work, then try in your code:

    for item in elements: print(item.get_attribute('textContent'))
    

    I made it working using protractor:

    enter image description here

    If you just need any tool for achieving this, then you could use protractor:

    Steps to use protractor:

    1. Install nodejs first, it will install npm along with it which is like pip for nodejs : https://nodejs.org/en/download/
    2. now install protractor:

    npm install -g protractor

    1. Now create a file called conf.js and add the below code in it and save
     exports.config = {
    
    specs: ['test.js'],
    
    capabilities: {
      'browserName': 'chrome',
    },
    
    onPrepare: function(){
        browser.waitForAngularEnabled(false);
        browser.driver.manage().window().maximize();
    },
    
      jasmineNodeOpts: {
        showColors: true, // Use colors in the command line report.
        defaultTimeoutInterval: 10000000,
    
    },
    SELENIUM_PROMISE_MANAGER: false,
    }
    
    1. Now create a file test.js and copy-paste below code in it :
    describe('Protractor Demo App', function() {
    it('should have a title @Regression @Smoke @test', async   function() {
        await browser.get('https://www.drf.com/race-results');
        let a=await element.all(by.xpath('//div[@class="handicappingBottom"]')).getAttribute('textContent');
        console.log(a);
      });
      });
    
    1. Keep both conf.js and test.js in the same folder and run below command in cmd:

    protractor conf.js



Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2