Having trouble scraping data from website with Selenium

  • I've tried a couple different routes to get this to work but I've had no luck.


    driver= webdriver.PhantomJS(executable_path=r'C:\Users\x\Downloads\phantomjs-2.1.1-windows\phantomjs-2.1.1-windows\bin\phantomjs.exe')
    elements = driver.find_elements_by_xpath("//div[@class='handicappingBottom']")

    for item in elements: print(item)

    *Returns an empty list

    I then read up a bit and feel like the webpage isn't fully loading those items so I tried multiple variants of this:

    elements = ui.WebDriverWait(driver, 45).until(EC.visibility_of_all_elements_located((By.CLASS_NAME, "handicappingBottom")))

    Example of code snippet from webpage:

    Ideally I'm trying to scrape the current horse race locations into a list in a program I'm mocking up. Any help at all would be greatly appreciated. I'm new-ish to python and even newer than that to webscraping. Thanks a ton in advance!

  • Hi I am not sure why you are printing that

    try the below code if you want to get the displayed text from those elements:

    value= driver.find_elements_by_xpath("//div[@class='handicappingBottom']").get_attribute('textContent');

    if this doesn't work, then try in your code:

    for item in elements: print(item.get_attribute('textContent'))

    I made it working using protractor:

    enter image description here

    If you just need any tool for achieving this, then you could use protractor:

    Steps to use protractor:

    1. Install nodejs first, it will install npm along with it which is like pip for nodejs : https://nodejs.org/en/download/
    2. now install protractor:

    npm install -g protractor

    1. Now create a file called conf.js and add the below code in it and save
     exports.config = {
    specs: ['test.js'],
    capabilities: {
      'browserName': 'chrome',
    onPrepare: function(){
      jasmineNodeOpts: {
        showColors: true, // Use colors in the command line report.
        defaultTimeoutInterval: 10000000,
    1. Now create a file test.js and copy-paste below code in it :
    describe('Protractor Demo App', function() {
    it('should have a title @Regression @Smoke @test', async   function() {
        await browser.get('https://www.drf.com/race-results');
        let a=await element.all(by.xpath('//div[@class="handicappingBottom"]')).getAttribute('textContent');
    1. Keep both conf.js and test.js in the same folder and run below command in cmd:

    protractor conf.js

Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2