Selenium only loading JavaScript and not full HTML (Python)



  • I am trying to find the search bar in this link: https://www.takealot.com using the class "search-field" from:

    input class="search-field " type="text" placeholder="Search for products, brands..." value=""
    

    My Code:

    from selenium import webdriver
    import time
    
    driver = webdriver.Firefox()
    driver.get("https://www.takealot.com/")
    
    print(driver.page_source)
    
    time.sleep(100)
    
    driver.quit()
    

    I have set sleep to 100 hoping that the HTML was taking long to load but I only seem to get the below output:

    <html dir="ltr" lang="en"><head>
    <link rel="preload" href="https://shopfront.takealot.com/static/js/app-loader.js" as="script">
    <link rel="preconnect" href="//media.takealot.com" crossorigin="">
    <link rel="preconnect" href="https://shopfront.takealot.com" crossorigin="">
    <link rel="preconnect" href="//api.takealot.com" crossorigin="">
    <link rel="preconnect" href="//static.takealot.com" crossorigin="">
    <title>Takealot.com: Online Shopping | SA's leading online store</title>
    <link rel="shortcut icon" href="https://www.takealot.com/favicon.ico?20110717" type="image/x-icon">
    <meta property="twitter:account_id" content="4503599630296419">
    <meta name="description" content="South Africa's leading online store. Fast, reliable delivery to your door. Many ways to pay. Shop anything you can imagine: TVs, laptops, cellphones, kitchen appliances, toys, books, beauty &amp; more. Shop the mobile app anytime, anywhere." data-react-helmet="true"><link rel="canonical" href="https://www.takealot.com/" data-react-helmet="true"><link rel="alternate" media="handheld" href="https://m.takealot.com/" data-react-helmet="true"><link rel="alternate" href="android-app://fi.android.takealot/app/takealot.com/home" data-react-helmet="true"><meta name="robots" content="INDEX,FOLLOW" data-react-helmet="true"><meta property="og:site_name" content="Takealot.com" data-react-helmet="true"><meta property="og:title" content="Takealot.com: Online Shopping | SA's leading online store" data-react-helmet="true"><meta property="og:description" content="South Africa's leading online store. Fast, reliable delivery to your door. Many ways to pay. Shop anything you can imagine: TVs, laptops, cellphones, kitchen appliances, toys, books, beauty &amp; more. Shop the mobile app anytime, anywhere." data-react-helmet="true"><meta property="og:image" content="https://www.takealot.com/static/images/logo_transparent.png" data-react-helmet="true"><script type="application/ld+json">
      {
        "@context": "http://schema.org",
        "@type": "WebSite",
        "url": "https://www.takealot.com",
        "name": "Takealot.com",
        "potentialAction": {
            "@type": "SearchAction",
            "target": "https://www.takealot.com/all/?qsearch={search_term_string}",
            "query-input": "required name=search_term_string"
        }
      }
    </script>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <style>
            html, body {
              background-color: #f4f4f4;
            }
            .sf {
              min-height: 400px;
            }
          </style>
    <link href="https://shopfront.takealot.com/56ef9861e86851c6e164b4f9ac61aaf391be03e0/static/js/runtime-752252ceefc11a26036e.js" rel="preload" as="script"><link href="https://shopfront.takealot.com/56ef9861e86851c6e164b4f9ac61aaf391be03e0/static/js/vendors~app-79405fe02e9a75a07916.js" rel="preload" as="script"><link href="https://shopfront.takealot.com/56ef9861e86851c6e164b4f9ac61aaf391be03e0/static/js/app-0aaac12d85774ebc557d.js" rel="preload" as="script"><script src="https://shopfront.takealot.com/56ef9861e86851c6e164b4f9ac61aaf391be03e0/static/js/runtime-752252ceefc11a26036e.js"></script></head>
    <body id="body">
    <div class="sf " id="shopfront-app"><style>
        @keyframes load-in {
          0% {
            opacity: 0;
          }
    
          100% {
            opacity: 1;
          }
        }
    
        @keyframes load8 {
          0% {
            transform: rotate(0deg);
          }
    
          100% {
            transform: rotate(360deg);
          }
        }
    
        .appload-wrapper {
          animation: load-in 200ms linear;
          box-sizing: border-box;
          height: 100%;
          min-height: 300px;
          position: relative;
          transform: translateZ(0);
        }
    
        .apploader {
          animation: load8 .8s infinite linear;
          border-bottom: 3px solid rgba(77, 77, 79, .2);
          border-left: 3px solid #0b79bf;
          border-radius: 50%;
          border-right: 3px solid rgba(77, 77, 79, .2);
          border-top: 3px solid rgba(77, 77, 79, .2);
          box-sizing: border-box;
          display: block;
          font-size: 10px;
          height: 30px;
          left: calc(50% - 15px);
          position: absolute;
          text-indent: -9999em;
          top: calc(50% - 15px);
          transform: translateZ(0);
          width: 30px;
        }
    
        .apploader::after {
          border-radius: 50%;
          box-sizing: border-box;
          height: 30px;
          width: 30px;
        }
        </style>
        <div class="appload-wrapper">
          <div class="apploader"></div>
        </div></div>
    <script src="https://shopfront.takealot.com/static/js/app-loader.js" onload="window.loadShopfront('https://shopfront.takealot.com');" type="text/javascript">
          </script>
    
    
    </body></html>
    


  • To set the value in search field Induce WebDriverWait() and wait for element_to_be_clickable() and use below css selector.

    driver.get("https://www.takealot.com/")
    WebDriverWait(driver,20).until(EC.element_to_be_clickable((By.CSS_SELECTOR,"input.search-field"))).send_keys("USB")
    

    You need to import below libraries.

    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    


Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2