顯示具有 WebRequest 標籤的文章。 顯示所有文章
顯示具有 WebRequest 標籤的文章。 顯示所有文章

[Java]Http Get與Post函式2(HttpURLConnection)

序言

之前我發的一篇[Http Get與Post函式]中已經實現了可維持Session狀態的HTTP連線函式,但我發現了Apache提供的這個Client在Windows下會有連線數上限的問題(聽說好像調整XP連線數可以解決)。
另外,有的時候使用HTTP連線也不一定需要維持Session狀態,這時候其實就不需要另外加入Apache的http client的JAR。
因此我還是整理一下如果一個不維持Session狀態的HTTP Client連線怎麼達成。

函式原始碼

參數轉換函式
package common.control;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.HashMap;

public class HTTPParseFunc {

/**
* hashMapToString
*
* @param map
* @param charset編碼
* ,如HTTP.UTF_8
* @return
* @throws UnsupportedEncodingException
*/
@SuppressWarnings("unchecked")
public static String hashMapToString(HashMap<String, String> map,
String charset) throws UnsupportedEncodingException {
StringBuffer result = new StringBuffer();
java.util.Iterator it = map.entrySet().iterator();
boolean isfirst = true;
while (it.hasNext()) {
java.util.Map.Entry entry = (java.util.Map.Entry) it.next();
if (isfirst) {
isfirst = false;
} else {
result.append("&");
}
result
.append(URLEncoder.encode(entry.getKey().toString(),
charset));
result.append("=");
result.append(URLEncoder.encode(entry.getValue().toString(),
charset));
}
return result.toString();
}
/**
* 將inputStream轉為String
*
* @param is
* inputStream
* @param charset
* 編碼,如HTTP.UTF_8
* @return inputStream的內容
* @throws UnsupportedEncodingException
*/
public static String inputStream2String(InputStream is, String charset)
throws UnsupportedEncodingException {
BufferedReader in = new BufferedReader(new InputStreamReader(is,
charset));
StringBuffer buffer = new StringBuffer();
String line = "";
try {
boolean isfirst = true;
while ((line = in.readLine()) != null) {
if (!isfirst) {
buffer.append("\n");
} else {
isfirst = false;
}
buffer.append(line);
}
} catch (IOException e) {
e.printStackTrace();
}

return buffer.toString();
}

/**
* HTTP 傳輸參數分割
* @param param 如name1=value1&name2=value2
* @return
* @throws UnsupportedEncodingException
*/
public static ArrayList<String[]> paramToArray(String param)
throws UnsupportedEncodingException {
ArrayList<String[]> arr = null;
String[] p = param.split("&");
if (param.toLowerCase().contains("&amp;")) {
ArrayList<String> p2 = new ArrayList<String>();
int j = 0;
for (int i = 0; i < p.length; i++) {
if (p[i].toLowerCase().startsWith("amp;")) {
p2.set(j - 1, p2.get(j - 1) + "&amp;" + p[i].substring(4));
j--;
}
p2.add(p[i]);
j++;
}
p2.toArray(p);
}

for (int i = 0; i < p.length; i++) {
String[] item = p[i].split("=");
if (item.length == 2) {
if (arr == null)
arr = new ArrayList<String[]>();
// item[0]=URLDecoder.decode(item[0],charset);
// item[1]=URLDecoder.decode(item[1],charset);
arr.add(item);
}
}
return arr;
}
}

連線參數模型
package common.model;

public class HTTPResponse {
String html=null;
Integer statusCode=null;
public String getHtml() {
return html;
}
public void setHtml(String html) {
this.html = html;
}
public Integer getStatusCode() {
return statusCode;
}
public void setStatusCode(Integer statusCode) {
this.statusCode = statusCode;
}
}


HTTP連線函式
package common;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.PrintWriter;
import java.io.UnsupportedEncodingException;
import java.net.HttpURLConnection;
import java.net.Proxy;
import java.net.URL;
import java.util.HashMap;

import common.control.HTTPParseFunc;
import common.model.HTTPResponse;



public class HTTPBaseIO2 {
public enum Method {
get, post
}
/**
* 送出Request
* @param urlpath URL
* @param method HTTPBaseIO2.Method.get/post
* @param params HashMap<Name, Value>
* @param charset UTF-8
* @param timeout 連線逾時null:不設限/millisecond
* @return 回傳的html
* @throws UnsupportedEncodingException
*/
public static HTTPResponse doSend(String urlpath, Method method,
HashMap<String, String> params, String charset,boolean isAutoRedirect, Integer timeout,Proxy proxy) throws UnsupportedEncodingException {
HTTPResponse result=new HTTPResponse();
String param=null;
if(params!=null){
param=HTTPParseFunc.hashMapToString(params, charset);
}
if(method==Method.post){
result=sendPost(urlpath,param,charset,isAutoRedirect,timeout,proxy);
}else{
result=sendGet(urlpath,param,charset,isAutoRedirect,timeout,proxy);
}
return result;
}
/**
* 向指定URL發送GET方法的請求
*
* @param url
* 發送請求的URL
* @param param
* 請求參數,請求參數應該是name1=value1&name2=value2的形式。
* @return URL所代表遠程資源的響應
*/
public static HTTPResponse sendGet(String url, String params, String charset,boolean isAutoRedirect, Integer timeout,Proxy proxy) {
HTTPResponse result=new HTTPResponse();
HttpURLConnection conn=null;
BufferedReader in = null;
try {
String urlName = url;
if(params!=null) urlName+= "?" + params;
URL realUrl = new URL(urlName);

HttpURLConnection.setFollowRedirects(isAutoRedirect);
// 打開和URL之間的連接
if(proxy!=null)
conn = (HttpURLConnection)realUrl.openConnection(proxy);
else
conn = (HttpURLConnection)realUrl.openConnection();
if(timeout!=null) conn.setConnectTimeout(timeout);
// 設置通用的請求屬性
conn.setRequestProperty("accept", "*/*");
conn.setRequestProperty("connection", "Keep-Alive");
conn.setRequestProperty("user-agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)");
// 建立實際的連接
conn.connect();
result.setHtml(HTTPParseFunc.inputStream2String(conn.getInputStream(), charset));
} catch (Exception e) {
e.printStackTrace();
result=null;
}
// 使用finally塊來關閉輸入流
finally {
try {
if (conn != null){
if(conn.getResponseCode()!=HttpURLConnection.HTTP_OK)
result=null;
result.setStatusCode(conn.getResponseCode());
}
} catch (Exception e) {
}
try {
if (in != null) {
in.close();
}
if (conn != null){
conn.disconnect();
}
} catch (IOException ex) {
ex.printStackTrace();
}
}
return result;
}

/**
* 向指定URL發送POST方法的請求
*
* @param url
* 發送請求的URL
* @param param
* 請求參數,請求參數應該是name1=value1&name2=value2的形式。
* @return URL所代表遠程資源的響應
*/
public static HTTPResponse sendPost(String url, String params, String charset,boolean isAutoRedirect, Integer timeout,Proxy proxy) {
HTTPResponse result=new HTTPResponse();
PrintWriter out = null;
BufferedReader in = null;
HttpURLConnection conn=null;
try {
URL realUrl = new URL(url);
// 打開和URL之間的連接

HttpURLConnection.setFollowRedirects(isAutoRedirect);
if(proxy!=null)
conn = (HttpURLConnection)realUrl.openConnection(proxy);
else
conn = (HttpURLConnection)realUrl.openConnection();
if(timeout!=null) conn.setConnectTimeout(timeout);
// 設置通用的請求屬性
conn.setRequestProperty("accept", "*/*");
conn.setRequestProperty("connection", "Keep-Alive");
conn.setRequestProperty("user-agent",
"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)");
// 發送POST請求必須設置如下兩行
conn.setDoOutput(true);
conn.setDoInput(true);
// 獲取URLConnection對象對應的輸出流
out = new PrintWriter(conn.getOutputStream());
// 發送請求參數
if(params!=null) out.print(params);
// flush輸出流的緩衝
out.flush();
result.setHtml(HTTPParseFunc.inputStream2String(conn.getInputStream(), charset));

} catch (Exception e) {
e.printStackTrace();
result=null;
}
// 使用finally塊來關閉輸出流、輸入流
finally {
try {
if (conn != null){
if(conn.getResponseCode()!=HttpURLConnection.HTTP_OK)
result=null;
result.setStatusCode(conn.getResponseCode());
}
} catch (Exception e) {
}
try {
if (out != null) {
out.close();
}
if (in != null) {
in.close();
}
} catch (IOException ex) {
ex.printStackTrace();
}
if (conn != null){
conn.disconnect();
}
}
return result;
}


}

使用範例程式

package common.test;

import java.util.HashMap;


import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import common.HTTPBaseIO2;
import common.model.HTTPResponse;

public class HTTPBaseIO2Test {

@Before
public void setUp() throws Exception {
}

@After
public void tearDown() throws Exception {
}

@Test
public void testDoGet() {
String urltest = "http://localhost:8080/Test/Test1" ;
String charset="UTF-8";
HashMap<String, String> params=new HashMap<String, String>();
try {
params.put("val", "測試&測試");
HTTPResponse html=HTTPBaseIO2.doSend(urltest, HTTPBaseIO2.Method.get, params, charset,true,null,null);
System.out.println("testDoGet:"+html.getHtml());
} catch (Exception e) {
e.printStackTrace();
}
}

@Test
public void testDoPost() {
String urltest = "http://localhost:8080/Test/Test1" ;
String charset="UTF-8";
HashMap<String, String> params=new HashMap<String, String>();
try {
params.put("val", "測試1&測試1");
HTTPResponse html=HTTPBaseIO2.doSend(urltest, HTTPBaseIO2.Method.post, params, charset,true,null,null);
System.out.println("testDoPost:"+html.getHtml());
params.clear();
params.put("val", "測試2&測試2");
html=HTTPBaseIO2.doSend(urltest, HTTPBaseIO2.Method.post, params, charset,true,null,null);
System.out.println("testDoPost:"+html.getHtml());
} catch (Exception e) {
e.printStackTrace();
}
}

}

[Java]Http Get與Post函式(Apache HttpClient)

序言

要達到基本的Http get與post方法取得網站內容,Java內的URLConnection就可以達成
但我實作時發現它太過於底層,以致於我不知該如何做到關於Session狀態的延續,也就是不能夠在完成登入後,保留住登入狀態然後進入下個動作。
於是就尋求別的作法,最後發現了Apache有提供了HttpClient套件的JAR檔,能讓我們用更簡單的方式達成工作。
在此參考網路上的文章:用HttpClient來模擬瀏覽器GET,POST,加上可自動轉導的功能,包成函式。
另外再加上可用三種類型來設定傳遞的參數,並控制其連線與中斷狀態。

下載套件

HttpClient Download下載套件,選擇Binary with dependencies的版本(如4.0.1.zip),
我們需要的JAR在該壓縮檔中的lib目錄下。會用到的JAR檔有三個:
  • commons-logging-1.1.1.jar
  • httpclient-4.0.1.jar
  • httpcore-4.0.1.jar

函式原始碼

參數轉換函式
package common.control;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.UnsupportedEncodingException;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.HashMap;

public class HTTPParseFunc {

/**
* hashMapToString
*
* @param map
* @param charset編碼
* ,如HTTP.UTF_8
* @return
* @throws UnsupportedEncodingException
*/
@SuppressWarnings("unchecked")
public static String hashMapToString(HashMap<String, String> map,
String charset) throws UnsupportedEncodingException {
StringBuffer result = new StringBuffer();
java.util.Iterator it = map.entrySet().iterator();
boolean isfirst = true;
while (it.hasNext()) {
java.util.Map.Entry entry = (java.util.Map.Entry) it.next();
if (isfirst) {
isfirst = false;
} else {
result.append("&");
}
result
.append(URLEncoder.encode(entry.getKey().toString(),
charset));
result.append("=");
result.append(URLEncoder.encode(entry.getValue().toString(),
charset));
}
return result.toString();
}
/**
* 將inputStream轉為String
*
* @param is
* inputStream
* @param charset
* 編碼,如HTTP.UTF_8
* @return inputStream的內容
* @throws UnsupportedEncodingException
*/
public static String inputStream2String(InputStream is, String charset)
throws UnsupportedEncodingException {
BufferedReader in = new BufferedReader(new InputStreamReader(is,
charset));
StringBuffer buffer = new StringBuffer();
String line = "";
try {
boolean isfirst = true;
while ((line = in.readLine()) != null) {
if (!isfirst) {
buffer.append("\n");
} else {
isfirst = false;
}
buffer.append(line);
}
} catch (IOException e) {
e.printStackTrace();
}

return buffer.toString();
}

/**
* HTTP 傳輸參數分割
* @param param 如name1=value1&name2=value2
* @return
* @throws UnsupportedEncodingException
*/
public static ArrayList<String[]> paramToArray(String param)
throws UnsupportedEncodingException {
ArrayList<String[]> arr = null;
String[] p = param.split("&");
if (param.toLowerCase().contains("&amp;")) {
ArrayList<String> p2 = new ArrayList<String>();
int j = 0;
for (int i = 0; i < p.length; i++) {
if (p[i].toLowerCase().startsWith("amp;")) {
p2.set(j - 1, p2.get(j - 1) + "&amp;" + p[i].substring(4));
j--;
}
p2.add(p[i]);
j++;
}
p2.toArray(p);
}

for (int i = 0; i < p.length; i++) {
String[] item = p[i].split("=");
if (item.length == 2) {
if (arr == null)
arr = new ArrayList<String[]>();
// item[0]=URLDecoder.decode(item[0],charset);
// item[1]=URLDecoder.decode(item[1],charset);
arr.add(item);
}
}
return arr;
}
}

HTTP連線函式
package common;

import java.io.IOException;
import java.io.UnsupportedEncodingException;
import java.util.HashMap;
import java.util.List;

import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.HttpHost;
import org.apache.http.HttpResponse;
import org.apache.http.HttpStatus;
import org.apache.http.NameValuePair;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpUriRequest;
import org.apache.http.conn.params.ConnRoutePNames;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.params.HttpConnectionParams;
import org.apache.http.params.HttpParams;

import common.control.HTTPParseFunc;

public class HTTPBaseIO {
public enum Method {
get, post
}

private DefaultHttpClient httpclient = null;
private boolean isClosedConn = false;
private String newuri = null;
private int statuscode = HttpStatus.SC_NO_CONTENT;

private HttpHost proxy = null;

private Integer timeout=null;

public HTTPBaseIO(){

}
public HTTPBaseIO(String proxyIP,int proxyPort){
setProxy(proxyIP,proxyPort);
}
public HTTPBaseIO(HttpHost proxy){
setProxy(proxy);
}

/**
* 取得使用的proxy
*/
public HttpHost getProxy() {
return proxy;
}

/**
* 設定proxy
*/
public void setProxy(HttpHost proxy) {
this.proxy = proxy;
}

/**
* 設定proxy
*
* @param ip
* proxy的IP(hostname)
* @param port
* proxy的Port
*/
public void setProxy(String ip, int port) {
if(ip!=null)
proxy = new HttpHost(ip, port);
}

/**
* 取得回應後所得到的代碼,可參考org.apache.http.HttpStatus類別
*
* @return org.apache.http.HttpStatus
*/
public int getStatuscode() {
return statuscode;
}

/**
* 如果是轉導的狀態所得到的URI
*
* @return
*/
public String getNewuri() {
return newuri;
}

public void resetNewuri() {
newuri = null;
}

/**
* 取得連線物件
*
* @return
*/
public DefaultHttpClient getHttpclient() {
return httpclient;
}

/**
* 設定連線物件
*
* @param httpclient
*/
public void setHttpclient(DefaultHttpClient httpclient) {
this.httpclient = httpclient;
}

/**
* 是否已關閉連線
*
* @return
*/
public boolean isClosedConn() {
return isClosedConn;
}

/**
* 關閉連線
*/
public void closeConn() {
closeConn(true);
}

/**
* 關閉連線
*
* @param isCloseConn
* 是否關閉
*/
public void closeConn(boolean isCloseConn) {
if (isCloseConn && httpclient != null && !isClosedConn) {
httpclient.getConnectionManager().shutdown();
httpclient = null;
isClosedConn = true;
}
}

public void setHttpConnectionFactoryTimeout(Integer milliseconds){
timeout=milliseconds;
}

/**
* 取得網頁內容
*
* @param urlpath
* 網址
* @param method
* Method.get or Method.post
* @param params
* 參數
* @param charset
* 編碼,如HTTP.UTF_8
* @param isAutoRedirect
* 如果網頁回應狀態為轉導到新網頁,且Header的location有值,則自己以location所指網址取得內容
* @param isCloseConn
* 是否關閉連線
* @return 失敗回傳null,成功回傳網頁HTML
* @throws ClientProtocolException
* @throws IOException
*/
public String doSend(String urlpath, Method method, String params,
String charset, boolean isAutoRedirect, boolean isCloseConn)
throws ClientProtocolException, IOException {
return doSendBase(urlpath, method, StringToHttpEntity(params, charset),
charset, isAutoRedirect, isCloseConn);
}

/**
* 取得網頁內容
*
* @param urlpath
* 網址
* @param method
* Method.get or Method.post
* @param params
* 參數
* @param charset
* 編碼,如HTTP.UTF_8
* @param isAutoRedirect
* 如果網頁回應狀態為轉導到新網頁,且Header的location有值,則自己以location所指網址取得內容
* @param isCloseConn
* 是否關閉連線
* @return 失敗回傳null,成功回傳網頁HTML
* @throws ClientProtocolException
* @throws IOException
*/
public String doSend(String urlpath, Method method,
List<NameValuePair> params, String charset, boolean isAutoRedirect,
boolean isCloseConn) throws ClientProtocolException, IOException {
return doSendBase(urlpath, method, ListToHttpEntity(params, charset),
charset, isAutoRedirect, isCloseConn);
}

/**
* 取得網頁內容
*
* @param urlpath
* 網址
* @param method
* Method.get or Method.post
* @param params
* 參數
* @param charset
* 編碼,如HTTP.UTF_8
* @param isAutoRedirect
* 如果網頁回應狀態為轉導到新網頁,且Header的location有值,則自己以location所指網址取得內容
* @param isCloseConn
* 是否關閉連線
* @return 失敗回傳null,成功回傳網頁HTML
* @throws ClientProtocolException
* @throws IOException
*/
public String doSend(String urlpath, Method method,
HashMap<String, String> params, String charset,
boolean isAutoRedirect, boolean isCloseConn)
throws ClientProtocolException, IOException {
return doSendBase(urlpath, method,
HashMapToHttpEntity(params, charset), charset, isAutoRedirect,
isCloseConn);
}

/**
* 取得網頁內容
*
* @param urlpath
* 網址
* @param method
* Method.get or Method.post
* @param params
* 參數
* @param charset
* 編碼,如HTTP.UTF_8
* @param isAutoRedirect
* 如果網頁回應狀態為轉導到新網頁,且Header的location有值,則自己以location所指網址取得內容
* @param isCloseConn
* 是否關閉連線
* @return 失敗回傳null,成功回傳網頁HTML
* @throws ClientProtocolException
* @throws IOException
*/
public String doSendBase(String urlpath, Method method, HttpEntity params,
String charset, boolean isAutoRedirect, boolean isCloseConn)
throws ClientProtocolException, IOException {
String responseBody = null;
HttpUriRequest httpgetpost = null;

statuscode = HttpStatus.SC_NO_CONTENT;
try {
if (httpclient == null || isClosedConn())
httpclient = new DefaultHttpClient();

if (proxy != null)
httpclient.getParams().setParameter(
ConnRoutePNames.DEFAULT_PROXY, proxy);

if(timeout!=null){
HttpParams param = httpclient.getParams();
HttpConnectionParams.setConnectionTimeout(param, timeout);
HttpConnectionParams.setSoTimeout(param, timeout);
}


if (method == Method.post) {
httpgetpost = new HttpPost(urlpath);
if (params != null) {
((HttpPost) httpgetpost).setEntity(params);
}
} else {
if (params != null) {
urlpath += "?"
+ HTTPParseFunc.inputStream2String(params.getContent(), charset);
}
httpgetpost = new HttpGet(urlpath);
}

HttpResponse response = httpclient.execute(httpgetpost);
statuscode = response.getStatusLine().getStatusCode();
if ((statuscode == HttpStatus.SC_MOVED_TEMPORARILY)
|| (statuscode == HttpStatus.SC_MOVED_PERMANENTLY)
|| (statuscode == HttpStatus.SC_SEE_OTHER)
|| (statuscode == HttpStatus.SC_TEMPORARY_REDIRECT)) {
Header header = response.getFirstHeader("location");

if (header != null) {
newuri = header.getValue();
if ((newuri == null) || (newuri.equals("")))
newuri = "/";
if (isAutoRedirect) {
httpgetpost.abort();
httpgetpost = null;
responseBody = doSendBase(newuri, Method.get, null,
charset, true, false);
}
}
} else if (statuscode == HttpStatus.SC_OK) {
responseBody = HTTPParseFunc.inputStream2String(response.getEntity()
.getContent(), charset);
}
} catch (ClientProtocolException e) {
throw e;
} catch (IOException e) {
throw e;
} catch (Exception e) {
e.printStackTrace();
} finally {
if (httpgetpost != null) {
httpgetpost.abort();
httpgetpost = null;
}
closeConn(isCloseConn);
}
return responseBody;
}

/**
* List<NameValuePair>轉為HttpEntity
*
* @param nvps
* @param charset
* 編碼,如HTTP.UTF_8
* @return
* @throws UnsupportedEncodingException
*/
public static HttpEntity ListToHttpEntity(List<NameValuePair> nvps,
String charset) throws UnsupportedEncodingException {
HttpEntity result = null;
if (nvps != null && nvps.size() > 0) {
result = new UrlEncodedFormEntity(nvps, charset);

}
return result;
}

/**
* String to HttpEntity(
*
* @param nvps
* @param charset
* 編碼,如HTTP.UTF_8
* @return
* @throws UnsupportedEncodingException
*/
public static HttpEntity StringToHttpEntity(String nvps, String charset)
throws UnsupportedEncodingException {
HttpEntity result = null;
if (nvps != null) {
StringEntity reqEntity = new StringEntity(nvps, charset);
reqEntity.setContentType("application/x-www-form-urlencoded");
result = reqEntity;
}
return result;
}

/**
* HashMap To HttpEntity
*
* @param nvps
* @param charset
* 編碼,如HTTP.UTF_8
* @return
* @throws UnsupportedEncodingException
*/
public static HttpEntity HashMapToHttpEntity(HashMap<String, String> nvps,
String charset) throws UnsupportedEncodingException {
HttpEntity result = null;
if (nvps != null) {
result = new StringEntity(HTTPParseFunc.hashMapToString(nvps, charset), charset);
try {
result = StringToHttpEntity(HTTPParseFunc.inputStream2String(result
.getContent(), charset), charset);
} catch (IllegalStateException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}
return result;
}

}

使用範例程式

package common.test;

import static org.junit.Assert.*;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;

import org.apache.http.NameValuePair;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.protocol.HTTP;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import common.HTTPBaseIO;
import common.control.HTTPParseFunc;

public class HTTPBaseIOTest {

@Before
public void setUp() throws Exception {
}

@After
public void tearDown() throws Exception {
}

@Test
public void testDoGet() {
String urltest = "http://www.google.com.tw";
String charset = "UTF-8";
HTTPBaseIO.Method method = HTTPBaseIO.Method.get;
HTTPBaseIO reqClient = new HTTPBaseIO();
try {
String html = reqClient.doSendBase(urltest, method, null, charset,
false, false);
System.out.println(html);
if (html == null)
fail("Client get nothing");
} catch (Exception e) {
e.printStackTrace();
} finally {
reqClient.closeConn();
}
}

@Test
public void testDoPost() {
String urlserv1 = "http://localhost/Test1/Serv1";
String urlserv2 = "http://localhost/Test1/Serv2";
String charset = HTTP.UTF_8;
HTTPBaseIO.Method method = HTTPBaseIO.Method.post;
HTTPBaseIO reqClient = new HTTPBaseIO();
try {
HashMap<String, String> map = new HashMap<String, String>();
map.put("jobname", "login");
map.put("id", "aaa");
map.put("pswd", "1234");
String html = reqClient.doSend(urlserv1, method, map, charset,
false, false);
System.out.println(html);

ArrayList<String[]> arr = HTTPParseFunc.paramToArray(html);
for (int i = 0; i < arr.size(); i++)
System.out.println(arr.get(i)[0] + "=" + arr.get(i)[1]);
if (html == null)
fail("Login get nothing");

String param = "jobname=getname";
html = reqClient.doSend(urlserv1, method, param, charset, false,
false);
System.out.println(html);
if (html == null)
fail("getname get nothing");

List<NameValuePair> nvps = new ArrayList<NameValuePair>();
nvps.add(new BasicNameValuePair("jobname", "loadserv2"));
nvps.add(new BasicNameValuePair("name", "aaa"));
reqClient.resetNewuri();
html = reqClient.doSend(urlserv1, method, nvps, charset, true,
false);
if (html == null)
fail("loadserv2 get nothing");
nvps.clear();
if (reqClient.getNewuri() != null && html != null) {
nvps.add(new BasicNameValuePair("jobname", "getname2"));
html = reqClient.doSend(urlserv2, method, nvps, charset, false,
false);
System.out.println(html);
if (html == null)
fail("Name get nothing");
nvps.clear();
}
nvps.clear();

nvps.add(new BasicNameValuePair("jobname", "Logout"));
html = reqClient.doSend(urlserv1, method, nvps, charset, false,
false);
System.out.println(html);
if (html == null)
fail("Logout get nothing");
nvps.clear();
} catch (Exception e) {
e.printStackTrace();
} finally {
reqClient.closeConn();
}
}

}

總結

在範例的地方我假設了兩個servlet用來接收傳遞的參數,這部份可能不能直接執行,不過函式我測試過是沒問題的。
使用時可選擇用Hashmap、String、或是List作為參數設定,我分別在範例中都使用過了。
轉跳時可用getNewuri函式來確認是否有取得轉跳的網址,用回傳的html來確認是否有抓到回應的內容。
PS.我發現了Apache提供的這個Client在Windows下會有連線數上限的問題(聽說好像調整XP連線數可以解決)。

[程式]VB.Net2.0網頁原始碼取得函式

序言

我因為學術需求,要取得某些網址的網頁原始碼進行分析,因此寫了以下函式,貼出來與大家分享吧~

*2009/10/29 更新函式,加強切割網頁函式、抓網頁標頭。

開發環境

  • VB.Net 2.0 (VS2005)

專案設定

  • 建立專案後預設的專案屬性下,參考的元件如下圖:

  • 下面的程式中會用到【Web】這個物件,所以我們需要加入【System.Web】這個元件:

    點選加入,在.Net分頁下找到【System.Web】這個元件

網頁原始碼函式類別

Imports System.Net
Imports System.io
Imports System.Reflection

Public Class WebPageGenFunc
    Private Shared rd As New Random(Now.Second)
    Private Shared sec As Integer
    Private Shared lastsec As Integer = -1
    Public Shared Function sleepTime(Optional ByVal minimatime As Integer = 2600, Optional ByVal rangeSecond As Integer = 5) As Integer
        sec = (rd.Next Mod rangeSecond) * 1000 + minimatime
        While sec = lastsec
            sec = (rd.Next Mod rangeSecond) * 1000 + minimatime
        End While
        lastsec = sec
        System.Threading.Thread.Sleep(sec)
        Return sec
    End Function


    Private Shared Sub SetAllowUnsafeHeaderParsing20()
        Dim a As New System.Net.Configuration.SettingsSection
        Dim aNetAssembly As System.Reflection.Assembly = Assembly.GetAssembly(a.GetType)
        Dim aSettingsType As Type = aNetAssembly.GetType("System.Net.Configuration.SettingsSectionInternal")
        Dim args As Object() = Nothing
        Dim anInstance As Object = aSettingsType.InvokeMember("Section", BindingFlags.Static Or BindingFlags.GetProperty Or BindingFlags.NonPublic, Nothing, Nothing, args)
        Dim aUseUnsafeHeaderParsing As FieldInfo = aSettingsType.GetField("useUnsafeHeaderParsing", BindingFlags.NonPublic Or BindingFlags.Instance)
        aUseUnsafeHeaderParsing.SetValue(anInstance, True)
    End Sub


    ''' <summary>
    ''' 使用Get方法取得網頁內容
    ''' </summary>
    ''' <param name="url">網址</param>
    ''' <param name="noCashe">不使用快取</param>
    ''' <returns>網頁的HTML</returns>
    ''' <remarks>使用Get方法取得網頁內容</remarks>
    Public Shared Function getHTMLGet(ByVal url As String, Optional ByVal noCashe As Boolean = False) As String
        getHTMLGet = Nothing
        SetAllowUnsafeHeaderParsing20()
        Dim wRs As HttpWebResponse
        Dim wRq As HttpWebRequest
        ' Create the request using the WebRequestFactory.

        wRq = CType(WebRequest.Create(url), HttpWebRequest)
        With wRq
            .UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
            .ContentType = "application/x-www-form-urlencoded"
            .Headers.Add("Accept-Language", "zh-tw")
            .Method = "GET"
            .Timeout = 10000
            If noCashe Then
                Dim policy As New Cache.HttpRequestCachePolicy(Cache.HttpRequestCacheLevel.NoCacheNoStore)
                .CachePolicy = policy
                .Headers.Add("Cache-Control", "no-cache")
            End If
        End With

        Try
            ' Return the response stream.
            wRs = CType(wRq.GetResponse(), HttpWebResponse)
            Dim streamResponse As Stream = wRs.GetResponseStream()
            Dim streamRead As New StreamReader(streamResponse)
            Dim responseString As String = streamRead.ReadToEnd()
            getHTMLGet = responseString
            ' Close Stream object.
            streamResponse.Close()
            streamRead.Close()
            ' Release the HttpWebResponse.
            wRs.Close()
        Catch ex As Exception
            Console.WriteLine(ex.ToString)
        End Try
    End Function

    ''' <summary>
    ''' 使用Post方法取得網頁內容
    ''' </summary>
    ''' <param name="url">網址</param>
    ''' <param name="postdata">傳遞參數,如a=123&b=456</param>
    ''' <param name="noCashe">不使用快取</param>
    ''' <returns>網頁的HTML</returns>
    ''' <remarks>使用Post方法取得網頁內容</remarks>
    Public Shared Function getHTMLPost(ByVal url As String, Optional ByVal postdata As String = Nothing, Optional ByVal noCashe As Boolean = False) As String
        getHTMLPost = Nothing
        SetAllowUnsafeHeaderParsing20()
        Dim wRs As HttpWebResponse
        Dim wRq As HttpWebRequest
        ' Create the request using the WebRequestFactory.
        wRq = CType(WebRequest.Create(url), HttpWebRequest)
        With wRq
            .UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
            .Headers.Add("Accept-Language", "zh-tw")
            .Method = "POST"
            .Timeout = 10000
            .KeepAlive = False
            If noCashe Then
                Dim policy As New Cache.HttpRequestCachePolicy(Cache.HttpRequestCacheLevel.NoCacheNoStore)
                .CachePolicy = policy
                .Headers.Add("Cache-Control", "no-cache")
            End If
            If Not postdata Is Nothing Then
                .Timeout = 60000
                Dim encoding As New System.Text.ASCIIEncoding()
                Dim byte1 As Byte() = encoding.GetBytes(postdata)
                .ContentType = "application/x-www-form-urlencoded"
                .ContentLength = byte1.Length
                .GetRequestStream().Write(byte1, 0, byte1.Length)
            End If
        End With
        wRq.GetRequestStream().Close()
        Try
            ' Return the response stream.
            wRs = CType(wRq.GetResponse(), HttpWebResponse)
            Dim streamResponse As Stream = wRs.GetResponseStream()
            Dim streamRead As New StreamReader(streamResponse)
            Dim responseString As String = streamRead.ReadToEnd()
            getHTMLPost = responseString
            ' Close Stream object.
            streamResponse.Close()
            streamRead.Close()
            ' Release the HttpWebResponse.
            wRs.Close()

        Catch ex As Exception
            Console.WriteLine(ex.ToString)
        End Try
    End Function
    ''' <summary>
    ''' 使用WebClient取得網頁內容
    ''' </summary>
    ''' <param name="url">網址</param>
    ''' <param name="postdata">傳遞的參數</param>
    ''' <param name="method">使用的方法,預設為POST</param>
    ''' <param name="noCashe">不使用快取</param>
    ''' <returns>網頁的HTML</returns>
    ''' <remarks>使用WebClient取得網頁內容</remarks>
    Public Shared Function getHTMLWebClient(ByVal url As String, ByRef postdata As Specialized.NameValueCollection, Optional ByVal method As String = "POST", Optional ByVal noCashe As Boolean = False) As String
        getHTMLWebClient = Nothing
        Try
            SetAllowUnsafeHeaderParsing20()
            Dim myWebClient As New WebClient()
            If noCashe Then
                Dim policy As New Cache.HttpRequestCachePolicy(Cache.HttpRequestCacheLevel.NoCacheNoStore)
                myWebClient.CachePolicy = policy
                myWebClient.Headers.Add("Cache-Control", "no-cache")
            Else
                Dim rheaders As WebHeaderCollection = myWebClient.ResponseHeaders
                If Not rheaders Is Nothing Then
                    Dim header As String = rheaders("Set-Cookie")
                    If Not header Is Nothing Then
                        myWebClient.Headers.Add("Cookie", header)
                    End If
                End If
            End If
            myWebClient.Headers.Add("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)")
            myWebClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded")
            myWebClient.Headers.Add("Accept-Language", "zh-tw")
            If postdata Is Nothing Then postdata = New Specialized.NameValueCollection
            Dim responseArray As Byte() = myWebClient.UploadValues(url, method, postdata)
            Dim encoding As New System.Text.UTF8Encoding
            getHTMLWebClient = encoding.GetString(responseArray)
        Catch ex As Exception
            Console.WriteLine(ex.ToString)
        End Try
    End Function

    ''' <summary>
    ''' 取得網頁的Body區段
    ''' </summary>
    ''' <param name="url">網址</param>
    ''' <param name="postdata">傳遞的參數,若有值會使用getHTMLPost取得</param>
    ''' <param name="postValue">傳遞的參數,若有值會使用getHTMLWebClient取得</param>
    ''' <param name="noCashe">不使用快取</param>
    ''' <returns>網頁的HTML</returns>
    ''' <remarks>取得網頁的Body區段</remarks>
    Public Shared Function getHTMLBody(ByVal url As String, Optional ByVal postdata As String = Nothing, Optional ByVal postValue As Specialized.NameValueCollection = Nothing, Optional ByVal noCashe As Boolean = False, Optional ByVal useWebClient As Boolean = False) As String
        getHTMLBody = Nothing
        Dim html As String
        If Not postValue Is Nothing Then
            html = getHTMLWebClient(url, postValue, , noCashe)
        ElseIf Not postdata Is Nothing Then
            html = getHTMLPost(url, postdata, noCashe)
        Else
            If useWebClient Then
                html = getHTMLWebClient(url, Nothing, "GET", noCashe)
            Else
                html = getHTMLGet(url, noCashe)
            End If

        End If
        If Not html Is Nothing Then
            getHTMLTagContain(html, "body", getHTMLBody)
        End If
    End Function
    ''' <summary>
    ''' 取得HTML中第一個符合的標籤內容的指標,並將取得的標籤內容寫入參數中
    ''' </summary>
    ''' <param name="html">HTML</param>
    ''' <param name="tag">標籤</param>
    ''' <param name="contain">取得的標籤內容</param>
    ''' <param name="indexEnd">此標籤結束於HTML的指標</param>
    ''' <returns>標籤內容的指標</returns>
    ''' <remarks>取得HTML中第一個符合的標籤內容的指標,並將取得的標籤內容寫入參數中</remarks>
    Public Shared Function getHTMLTagContain(ByVal html As String, ByVal tag As String, Optional ByRef contain As String = Nothing, Optional ByRef indexEnd As Integer = -1) As Integer
        tag = tag.ToLower
        contain = Nothing
        indexEnd = -1
        Dim indexBegin As Integer = -1
        Dim indexbBegin As Integer = -1
        If Not html Is Nothing Then
            indexBegin = html.ToLower.IndexOf("<" & tag)
            If indexBegin > -1 Then
                indexbBegin = html.IndexOf(">", indexBegin)
                If indexbBegin > -1 Then
                    indexbBegin += 1
                End If
                Dim findTag As Boolean = False
                indexEnd = indexbBegin
                Dim lastStart As Integer = indexbBegin
                Dim stopLimit2 As Integer = 9999
                Do
                    indexEnd = html.ToLower.IndexOf("</" & tag, indexEnd)
                    If indexEnd > -1 Then
                        If html.Substring(lastStart, indexEnd - lastStart).IndexOf("<" & tag) > -1 Then
                            lastStart = indexEnd
                            indexEnd = indexEnd + ("</" & tag).Length
                            findTag = True
                        Else
                            findTag = False
                        End If
                    Else
                        findTag = False
                    End If
                    stopLimit2 -= 1
                Loop While findTag And stopLimit2 > 0

                If indexEnd > -1 Then
                    contain = html.Substring(indexbBegin, indexEnd - indexbBegin)

                    indexEnd = html.IndexOf(">", indexEnd)
                    If indexEnd > -1 Then
                        indexEnd += 1
                    End If
                End If

            End If
        End If
        Return indexbBegin
    End Function
    ''' <summary>
    ''' 取得HTML中第一個符合的標籤屬性的指標,並將取得的標籤內容與該屬性內容寫入參數中
    ''' </summary>
    ''' <param name="html">HTML</param>
    ''' <param name="tag">標籤</param>
    ''' <param name="attName">屬性名稱</param>
    ''' <param name="att">取得的屬性內容</param>
    ''' <param name="contain">取得的標籤內容</param>
    ''' <param name="indexEnd">此標籤結束於HTML的指標</param>
    ''' <returns></returns>
    ''' <remarks>取得HTML中第一個符合的標籤屬性的指標,並將取得的標籤內容與該屬性內容寫入參數中</remarks>
    Public Shared Function getHTMLTagAtt(ByVal html As String, ByVal tag As String, ByVal attName As String, Optional ByRef att As String = Nothing, Optional ByRef contain As String = Nothing, Optional ByRef indexEnd As Integer = -1) As Integer
        att = Nothing
        contain = Nothing
        tag = tag.ToLower
        attName = attName.ToLower
        indexEnd = -1
        Dim indexBegin As Integer = -1
        Dim indexaBegin As Integer = -1
        Dim indexbEnd As Integer = -1
        Dim findTag As Boolean = False
        Dim sign As String

        If Not html Is Nothing Then
            indexBegin = 0
            Dim stopLimit1 As Integer = 9999
            Do
                indexbEnd = -1
                indexBegin = html.ToLower.IndexOf("<" & tag, indexBegin)
                If indexBegin > -1 Then
                    indexbEnd = html.IndexOf(">", indexBegin)
                    If indexbEnd > -1 Then
                        indexaBegin = html.Substring(0, indexbEnd).Replace("""", "'").ToLower.IndexOf(attName & "='", indexBegin)
                        If indexaBegin > -1 Then
                            sign = html.Substring(indexaBegin + (attName & "=").Length, 1)
                            indexaBegin += (attName & "='").Length
                            Dim indexaEnd As Integer = html.Substring(0, indexbEnd).IndexOf(sign, indexaBegin)
                            If indexaEnd > -1 Then
                                att = html.Substring(indexaBegin, indexaEnd - indexaBegin)
                            End If
                        Else
                            indexBegin = indexbEnd + 1
                            Continue Do
                        End If
                        indexbEnd += 1
                    End If

                    findTag = False
                    indexEnd = indexbEnd
                    Dim lastStart As Integer = indexbEnd

                    Dim stopLimit2 As Integer = 9999
                    Do
                        indexEnd = html.ToLower.IndexOf("</" & tag, indexEnd)
                        If indexEnd > -1 Then
                            If html.Substring(lastStart, indexEnd - lastStart).IndexOf("<" & tag) > -1 Then
                                lastStart = indexEnd
                                indexEnd = indexEnd + ("</" & tag).Length
                                findTag = True
                            Else
                                findTag = False
                            End If
                        Else
                            findTag = False
                        End If
                        stopLimit2 -= 1
                    Loop While findTag And stopLimit2 > 0

                    If indexEnd > -1 Then
                        contain = html.Substring(indexbEnd, indexEnd - indexbEnd)
                        indexEnd = html.IndexOf(">", indexEnd)
                        If indexEnd > -1 Then
                            indexEnd += 1
                        End If
                    End If
                Else
                    Exit Do
                End If
                stopLimit1 -= 1
            Loop While att Is Nothing And stopLimit1 > 0
        End If
        Return indexbEnd
    End Function
    ''' <summary>
    ''' Url參數值編碼
    ''' </summary>
    ''' <param name="value">參數值</param>
    ''' <returns>編碼結果</returns>
    ''' <remarks>Url參數值編碼</remarks>
    Public Shared Function getEncodeStr(ByVal value As String)
        Return Web.HttpUtility.UrlEncode(value)
    End Function
    ''' <summary>
    ''' Url參數值解碼
    ''' </summary>
    ''' <param name="value">參數值</param>
    ''' <returns>解碼結果</returns>
    ''' <remarks>Url參數值解碼</remarks>
    Public Shared Function getDecodeStr(ByVal value As String)
        Return Web.HttpUtility.UrlDecode(value)
    End Function
End Class

使用範例程式

        Dim indexEnd As Integer
        Dim valueAttribute As String
        Dim valueContain As String
        Dim url As String = "http://allen080.blogspot.com/2009/05/vbnet20.html"
        '使用WebRequest的Get方法取得整個網頁的HTML
        'Dim htmlAll As String = WebPageGenFunc.getHTMLGet(url)
        '取得網頁Body的部份
        Dim htmlBody As String = WebPageGenFunc.getHTMLBody(url)
        If Not htmlBody Is Nothing Then
            indexEnd = htmlBody.IndexOf("<span class='item-control blog-admin'>")
            valueAttribute = Nothing
            valueContain = Nothing
            If indexEnd > 0 Then
                htmlBody = htmlBody.Substring(indexEnd)
                '取得a標籤1的內容
                WebPageGenFunc.getHTMLTagContain(htmlBody, "span", valueContain, indexEnd)
                Console.WriteLine("span標籤的內容: " & valueContain)
                'htmlBody = htmlBody.Substring(indexEnd)
                '取得a標籤2的內容與href的屬性
                WebPageGenFunc.getHTMLTagAtt(htmlBody, "a", "onclick", valueAttribute, valueContain, indexEnd)
                Console.WriteLine("a標籤的內容與onclick的屬性: " & valueAttribute & vbTab & ",內容:" & valueContain)
            End If
        End If
        

        '使用WebClient的Post方法取得資料
        url = "http://al080.summerhost.info/invoiceMe/index.php"
        Dim searchStr As String = "123"
        Dim postdata As New Specialized.NameValueCollection
        postdata.Add("ddlYear", "2009")
        postdata.Add("ddlMonth", "7-8")
        postdata.Add("txtInvNO", searchStr)
        postdata.Add("btnMatchInv", "對獎")
        htmlBody = WebPageGenFunc.getHTMLBody(url, , postdata)
        If Not htmlBody Is Nothing Then
            indexEnd = htmlBody.IndexOf("<div class=""DivRight"">")
            If indexEnd > 0 Then
                htmlBody = htmlBody.Substring(indexEnd)
                '取得div標籤的內容
                WebPageGenFunc.getHTMLTagContain(htmlBody, "div", valueContain, indexEnd)
                Console.WriteLine("使用WebClient的Post方法取得資料: " & valueContain)
            End If
        End If
        
        '使用WebRequest的Post方法取得資料
        searchStr = String.Empty
        '在組合參數時要同時編碼
        For i As Integer = 0 To postdata.Count - 1
            searchStr &= "&" & postdata.GetKey(i) & "=" & WebPageGenFunc.getEncodeStr(postdata(i))
        Next
        searchStr = searchStr.Substring(1)
        htmlBody = WebPageGenFunc.getHTMLBody(url, searchStr)
        If Not htmlBody Is Nothing Then
            indexEnd = htmlBody.IndexOf("<div class=""DivRight"">")
            If indexEnd > 0 Then
                htmlBody = htmlBody.Substring(indexEnd)
                '取得div標籤的內容
                WebPageGenFunc.getHTMLTagContain(htmlBody, "div", valueContain, indexEnd)
                Console.WriteLine("使用WebRequest的Post方法取得資料: " & valueContain)
            End If
        End If

執行結果

span標籤的內容:
<a class='quickedit' href='http://www.blogger.com/rearrange?blogID=2698062899592178296&widgetType=HTML&widgetId=HTML1&action=editWidget' onclick='return _WidgetManager._PopupConfig(document.getElementById("HTML1"));' target='configHTML1' title='編輯'>
<img alt='' height='18' src='http://img1.blogblog.com/img/icon18_wrench_allbkg.png' width='18'/>
</a>

a標籤的內容與onclick的屬性: return _WidgetManager._PopupConfig(document.getElementById("HTML1"));    ,內容:
<img alt='' height='18' src='http://img1.blogblog.com/img/icon18_wrench_allbkg.png' width='18'/>

使用WebClient的Post方法取得資料: <span style="font-weight:bold;">對獎結果:</span><br>
                <div class="DivResult">
            無中獎發票...<br>
                </div>
           
使用WebRequest的Post方法取得資料: <span style="font-weight:bold;">對獎結果:</span><br>
                <div class="DivResult">
            無中獎發票...<br>
                </div>

相關連結

這裡是關於技術的手札~

也歡迎大家到

倫與貓的足跡



到噗浪來

關心一下我唷!
by 倫
 
Copyright 2009 倫倫3號Beta-Log All rights reserved.
Blogger Templates created by Deluxe Templates
Wordpress Theme by EZwpthemes