Multi-point passwording pages



  • I'm writing a walk for online plays that will provide useful data. There are two methods that return the value of specific weapons to the market for one country and for all countries (74) together.

    public void getCheapestWeapon(Weapon weapon) {
        //long result = 0;
        Map<String, Integer> countryAndId = allCountries.getCountryAndId();
        for (Map.Entry<String, Integer> item : countryAndId.entrySet()) {
            //long start = System.currentTimeMillis();
            getCheapestWeapon(item.getKey(), weapon);
            //long finish = System.currentTimeMillis();
            //System.out.println(finish - start);
            //result += finish - start;
        }
        //System.out.println(result);
    }
    

    public void getCheapestWeapon(String country, Weapon weapon) {
    long start = System.currentTimeMillis();
    String link = MARKET_LINK
    + allCountries.getCountryId(country)
    + "/" + weapon.getPlaceOnMarket()
    + "/" + weapon.getQuality()
    + "/" + ADDITIONAL_LINK;
    try {
    Document page = Jsoup.connect(link)
    .userAgent("Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36")
    .cookies(cookies)
    .timeout(1000)
    .get();
    Elements table = page.getElementsByClass("price_sorted");
    Element row = table.select("tr").first();
    Elements columns = row.getElementsByTag("td");
    Element priceCell = columns.get(3);
    String priceStr = priceCell.text();
    double price = Double.parseDouble(priceStr.substring(0, priceStr.length() - 4));
    System.out.println(country + " - " + price);
    } catch (IOException e) {
    e.printStackTrace();
    }
    //long finish = System.currentTimeMillis();
    //System.out.println(finish - start);
    }

    The problem is, when data are available to all countries, the time for passwording is an average of 16 to 18 seconds, which, please, is very much.
    Proceed in the first method to put the weapon into a separate flow. Nothing good. How can one-time passwording of all 74 pages be implemented?



  • Current (work results getCheapestWeapon It's easier to use standard. https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ExecutorService.html

    public void getCheapestWeapon(Weapon weapon) {
        final int THREADS = 4;
        ExecutorService pool = Executors.newFixedThreadPool( THREADS );
    
    Map&lt;String, Integer&gt; countryAndId = allCountries.getCountryAndId();
    for (Map.Entry&lt;String, Integer&gt; item : countryAndId.entrySet()) {
        pool.execute( () -&gt; getCheapestWeapon( item.getKey(), weapon ) );
    }
    
    pool.shutdown();
    

    }

    If the operation is not a one-time operation, the creation and shutdown of the service is a method. The main delay, most likely because of the network (74 requests, even if the package goes from the client to the 50ms server, eats 7 seconds), the number of service processes may exceed the number of kernels (please pilot).

    As an alternative, try to get more data from the server than the number of queries, e.g. removing the filter by country if you need the lowest price.




Suggested Topics

  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2
  • 2