java - Error using HtmlUnit 2.18 for crawl webpages -
i have following code:
webclient webclient = new webclient(browserversion.getdefault()); htmlpage page; list<htmlanchor> anchor=new arraylist<htmlanchor>(); try { system.out.println("querying"); page = webclient.getpage("https://www.amazon.com/gp/goldbox"); anchor = page.getanchors(); for(htmlanchor s:anchor) { system.out.println(s.getattribute("href")); } system.out.println("success"); }
querying
exception in thread "main" java.lang.nosuchfielderror: instance @ org.apache.http.impl.io.defaulthttprequestwriterfactory.<init>(defaulthttprequestwriterfactory.java:52) @ org.apache.http.impl.io.defaulthttprequestwriterfactory.<init>(defaulthttprequestwriterfactory.java:56) @ org.apache.http.impl.io.defaulthttprequestwriterfactory.<clinit>(defaulthttprequestwriterfactory.java:46) @ org.apache.http.impl.conn.managedhttpclientconnectionfactory.<init>(managedhttpclientconnectionfactory.java:82) @ org.apache.http.impl.conn.managedhttpclientconnectionfactory.<init>(managedhttpclientconnectionfactory.java:95) @ org.apache.http.impl.conn.managedhttpclientconnectionfactory.<init>(managedhttpclientconnectionfactory.java:104) @ org.apache.http.impl.conn.managedhttpclientconnectionfactory.<clinit>(managedhttpclientconnectionfactory.java:62) @ org.apache.http.impl.conn.poolinghttpclientconnectionmanager$internalconnectionfactory.<init>(poolinghttpclientconnectionmanager.java:572) @ org.apache.http.impl.conn.poolinghttpclientconnectionmanager.<init>(poolinghttpclientconnectionmanager.java:174) @ org.apache.http.impl.conn.poolinghttpclientconnectionmanager.<init>(poolinghttpclientconnectionmanager.java:158) @ org.apache.http.impl.conn.poolinghttpclientconnectionmanager.<init>(poolinghttpclientconnectionmanager.java:149) @ org.apache.http.impl.conn.poolinghttpclientconnectionmanager.<init>(poolinghttpclientconnectionmanager.java:125) @ com.gargoylesoftware.htmlunit.httpwebconnection.createconnectionmanager(httpwebconnection.java:972) @ com.gargoylesoftware.htmlunit.httpwebconnection.getresponse(httpwebconnection.java:161) @ com.gargoylesoftware.htmlunit.webclient.loadwebresponsefromwebconnection(webclient.java:1321) @ com.gargoylesoftware.htmlunit.webclient.loadwebresponse(webclient.java:1238) @ com.gargoylesoftware.htmlunit.webclient.getpage(webclient.java:346) @ com.gargoylesoftware.htmlunit.webclient.getpage(webclient.java:415) @ com.gargoylesoftware.htmlunit.webclient.getpage(webclient.java:400) @ crawler.htmlunitcrawl.main(htmlunitcrawl.java:29)
what might mistake?
you have conflicting classpath
, code works fine clean installation.
please remove httpcomponents
.jars, , use provided htmlunit.
also, can see version used by:
class<?> klass = defaulthttprequestwriterfactory.class; string location = klass.getprotectiondomain().getcodesource().getlocation().tostring(); system.out.println(location);
which should give location of httpcore-4.4.1.jar
in case.
Comments
Post a Comment