how to get html page source by C#

Question

I want to save complete web page asp in local drive by .htm from url or url but I did not success.

Code

public StreamReader Fn_DownloadWebPageComplete(string link_Pagesource)
{
     //--------- Download Complete ------------------
     //  using (WebClient client = new WebClient()) // WebClient class inherits IDisposable
     //   {

     //client
     //HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(link_Pagesource);

                    //webRequest.AllowAutoRedirect = true;
                    //var client1 = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(link_Pagesource);
                    //client1.CookieContainer = new System.Net.CookieContainer();


                 //   client.DownloadFile(link_Pagesource, @"D:\S1.htm");

              //  }
         //--------- Download Page Source ------------------
 HttpWebRequest URL_pageSource = (HttpWebRequest)WebRequest.Create("https://www.digikala.com");

                    URL_pageSource.Timeout = 360000;
                    //URL_pageSource.Timeout = 1000000;
                    URL_pageSource.ReadWriteTimeout = 360000;
                   // URL_pageSource.ReadWriteTimeout = 1000000;
                    URL_pageSource.AllowAutoRedirect = true;
                    URL_pageSource.MaximumAutomaticRedirections = 300;

                    using (WebResponse MyResponse_PageSource = URL_pageSource.GetResponse())
                    {

                        str_PageSource = new StreamReader(MyResponse_PageSource.GetResponseStream(), System.Text.Encoding.UTF8);
                        pagesource1 = str_PageSource.ReadToEnd();
                        success = true;
                    }

Error :

Too many automatic redirections were attempted.

Attemp by this codes but not successful.

many url is successful with this codes but this url not successful.

Hakan Fıstık · Accepted Answer · 2020-03-04T12:48:52.333

10

here is the way

string url = "https://www.digikala.com/";

using (HttpClient client = new HttpClient())
{
   using (HttpResponseMessage response = client.GetAsync(url).Result)
   {
      using (HttpContent content = response.Content)
      {
         string result = content.ReadAsStringAsync().Result;
      }
   }
}

and result variable will contains the page as HTML then you can save it to a file like this

System.IO.File.WriteAllText("path/filename.html", result);

NOTE you have to use the namespace

using System.Net.Http;

Update if you are using legacy VS then you can see this answer for using WebClient and WebRequest for the same purpose, but Actually updating your VS is a better solution.

edited Mar 04 '20 at 12:48

answered Jan 21 '17 at 10:22

Hakan Fıstık

14,367
10
94
117

Thanks,must update to vs2013 and install System.Net.Http and test. – RedArmy Jan 21 '17 at 10:49
@RedArmy Glad to help you, and please consider to accept this answer if it is solved your problem. – Hakan Fıstık Jan 21 '17 at 10:50
@RedArmy I updated my answer to solve your problem without updating Visual Studio and without Installing any `dll` (although this is not recommended) – Hakan Fıstık Jan 21 '17 at 10:55
1

While using the `.Result` does work, it converts the async call into a blocking sync call. It is better to `await` the async method call (no need to use .Result then) to benefit from the async nature. – Hans Keﬆing Jan 21 '17 at 11:01
dear @HakamFostok , i try witout update with webclient but not successful but update vs and successful your solution but when save the page, no show product lis . – RedArmy Jan 21 '17 at 11:02
in url : `https://www.digikala.com/search/category-motherboard/#%21/category-computer-devices/category-computer-parts/category-electronic-devices/category-motherboard/category-motherboard#%21category-computer-devicescategory-computer-partscategory-electronic-devicescategory-motherboard/category-motherboard#%21category-computer-devicescategory-computer-partscategory-electronic-devicescategory-motherboardcategory-motherboard#%21category-computer-devicescategory-computer-partscategory-electronic-devicescategory-motherboard/` – RedArmy Jan 21 '17 at 11:02
1

@HansKesting your are correct, but this depends on the requirement of the user itself. Any way I am resolving this issue in my other expanded answer which I linked. you can consider that expanded answer to handle most of the cases of `HttpClient` – Hakan Fıstık Jan 21 '17 at 11:04
use the AJax For show of Product , so i want to save complete page with c# but not successful. – RedArmy Jan 21 '17 at 11:05
1

Good solution for save source code dear @HakamFostok but this site used , ajax or etc for show product,please see url – RedArmy Jan 21 '17 at 11:08

score 2 · Answer 2 · answered Jan 21 '17 at 15:58

2

using (WebClient client = new WebClient ())
{
    string htmlCode = client.DownloadString("https://www.digikala.com");
}

answered Jan 21 '17 at 15:58

score 2 · Answer 3 · answered Jan 21 '17 at 16:07

2

using (WebClient client = new WebClient ())
{
    client.DownloadFile("https://www.digikala.com", @"C:\localfile.html");
}

answered Jan 21 '17 at 16:07

Thanks @imbot359 , but used this code and not work for this link.i dont know that why. – RedArmy Jan 22 '17 at 06:34

how to get html page source by C#

3 Answers3

Linked