Extracting data from webpage

[#1]

Hi,

I am trying to extract the data from the webpage i am getting the error as not aurthorized to access.I understood the actual problem is Internet is accessed based on credentials and the data in the webpage also accessed through passing the credentials.

(a)Websites are accessed by providing the username and Password.

(b)Data in the website are accessed by logining in to the specific website.

How the same can be implemented. Please suggest.

[stephencredmond]

QlikView, I believe, uses IE libraries to connect over http. This means that it passes Windows authentication to "intranet" sites (e.g. Sharepoint) correctly. However, if there is some sort of challenge/response or forms authentication, then that can't be handled.

The alternative is to use an external tool to connect to the website and download the data locally and then consume it in QlikView. For example, I have previously written a simple c# application to connect to a sharepoint site and download the contents locally to load into QlikView. There are many freeware applications available that can do this type of thing.

Regards,

Stephen

[gauravkhare]

Hi XXX,

you can find the solution in one of the sample applications given in QlikView 9.0 under currency conversion, in which they are extracting data from a websites containing the exchange rates for different countries. Here in this example they are first making the qvd for exchange rates and then performing scheduled reload as usual...Ofcourse we require some credentials for accessing the data froma website but this problem is solved when you contact your IT/Networking team concerning this issue.

So what I feel that your administrator will be granting you certain rights to get the work done!

[gauravkhare]

The best way being that you first extract data and make a qvd of it...and then perform weekly/monthly etc reload, it is because what I have seen that in many cases it takes many hours before a qvd is finally made while accessing data from a website!

So make a qvd first..................!

In this way internet authentication over the server can be given during different time periods beacuse some companies don't allow internet access over the production server..So in this case the IT Team grants special rights for extracting data from the website!!

[#5]

Thanks stephen and Gaurav,

i was trying to figure out the solution from the post

http://community.qlik.com/forums/t/38794.aspx

but not able.

[#6]

Hi,

Find cURL and WGET on Google, Wikipedia etc . Here is some sample usage: http://community.qlik.com/forums/p/44155/175663.aspx#175663

-Alex

[stevedark]

Hi there,

I've also run into the problem of getting data from a website that is protected with user credentials.

However, if anyone is looking to scrape data from a page where this isn't an issue I have created this short video:

Regards,

Steve

[#8]

Steve,

thanks for posting that video!!

It helped me greatly with a project that requires data extraction from websites to be added to an existing app.

[stevedark]

Glad to be able to help!  Cheers.

[alec1982]

I tried to follow the same way but I get 403 Forbidden as data only.

not sure if that because of the credentials..

Anybody can help..Please

Thxs,