Winnipeg Harvest donations from December meeting - thank you!
Thanks to the generosity of our members, we were able to donate three large boxes’ worth of non-perishable food items to Winnipeg Harvest in December 2010. The text of the thank-you letter we just received (see attached) is as follows: January 5, 2011 Manitoba UNIX User Group Attn: Adam Thompson Dear Friends of Harvest, We here at Winnipeg Harvest want to thank the Manitoba UNIX User Group for your support of helping those less fortunate in our community. Over the last few months 743 pounds of food was donated to the food bank. Your donations will go into hampers that will help Winnipeg Harvest feed more than 58,000 people each month, half of those being children. On behalf of the entire team at Winnipeg Harvest, thank you for helping us fight hunger and feed hope in Manitoba. We look forward to working with you again in the future. If I can ever be of assistance, please don’t hesitate to contact me by phone at 982-3670 or by email at tori@winnipegharvest.org. Yours truly, Tori Webber Development Associate Winnipeg Harvest Inc. So, thanks again. See you all in another six (6) days, at our next meeting on January 11th, 2011. -Adam Thompson athompso@muug.mb.ca
Problem: A web page / application presents information needed, but in the wrong format. The user can interact with the web page using check boxes and data fields. When the information appears complete, the user presses a "submit" button and the information is formatted in pdf to be printed out (call this format 1). This printout is needed, but in addition the same information (or possibly a few things added) must be printed in a different format (format 2). The solution I envision: run this web app in the browser as usual, and interact with it until the information is correct. The user can then launch a simple app which will find the open web page and scrape it, producing output in format 2. The user can then proceed by pressing the submit button on the page to output in format 1. At home on a Unix based system, I would use firewatir and ruby code to do this, starting an older version of firefox with -jssh. At work, this would be awkward. I saw a demonstration of PHP to do both scraping and presentation, but it seems to have server-side orientation. Running Apache or other server software would be out of the question. I need a client app simple to install, runable on Windows. A programmable browser would be ideal. Does anyone know of one that is multi-platform and can be installed without special services / privileges? Has anyone used XUL for something like this? Any suggestions are appreciated. -Dan Dan Martin GP Hospital Practitioner Computer Scientist ummar143@shaw.ca (204) 831-1746 answering machine always on
I'm looking at the Firefox Add-on SDK. I hope that will have the functionality I want. On 2012-03-22, at 3:04 PM, Dan Martin wrote:
Problem: A web page / application presents information needed, but in the wrong format.
The user can interact with the web page using check boxes and data fields. When the information appears complete, the user presses a "submit" button and the information is formatted in pdf to be printed out (call this format 1).
This printout is needed, but in addition the same information (or possibly a few things added) must be printed in a different format (format 2).
The solution I envision: run this web app in the browser as usual, and interact with it until the information is correct. The user can then launch a simple app which will find the open web page and scrape it, producing output in format 2. The user can then proceed by pressing the submit button on the page to output in format 1.
At home on a Unix based system, I would use firewatir and ruby code to do this, starting an older version of firefox with -jssh. At work, this would be awkward.
I saw a demonstration of PHP to do both scraping and presentation, but it seems to have server-side orientation. Running Apache or other server software would be out of the question. I need a client app simple to install, runable on Windows.
A programmable browser would be ideal. Does anyone know of one that is multi-platform and can be installed without special services / privileges? Has anyone used XUL for something like this?
Any suggestions are appreciated.
-Dan
Dan Martin GP Hospital Practitioner Computer Scientist ummar143@shaw.ca (204) 831-1746 answering machine always on
_______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Dan Martin GP Hospital Practitioner Computer Scientist ummar143@shaw.ca (204) 831-1746 answering machine always on
On 2012-03-22 Dan Martin wrote:
A programmable browser would be ideal. Does anyone know of one that is multi-platform and can be installed without special services / privileges? Has anyone used XUL for something like this?
I do this type of thing all the time: perl's WWW::Mechanize::Firefox on cpan. It remote controls FF using MozRepl. You could probably have it run on Windows in cygwin (MozRepl should run natively but perl w/weird modules is tougher). I've had to hack the suite a touch to get it to work with complicated pages that don't always fire their "done loading" event, though that should not be required and in most cases it works well OOTB. (And if you need some help, my programming time can be for-hire.)
Thanks for your reply, Trevor. I re-read your article from Feb 2011. Mechanize + MozRepl looks like a good combination - Mechanize (normally headless) to allow scripts, and MozRepl (appears to allow brief commands telnet style) to show the browser working. I'm going to try it some time. I'm not much of a perl programmer - I have a project written in perl, 5 lines of comments for each line of code, and I can't maintain it other than very minor fixes. I prefer Ruby, and I don't think it would be hard to code in python or javascript. I completed a working prototype using Ruby and Watir and I was able (with difficulty) to port it to a XP machine. I was not able to get the behaviour I wanted. I wanted the user to launch my program after the browser displayed the information with all the correct checkboxes, etc were selected (a la Ajax). The program would then attach to the open page and scrape it, producing output (possibly onto a newly launched second page). I have been able to do this in the past using Firewatir, which uses jssh to run Firefox, but this only works on Firefox 3 or earlier. I have nearly the same result. The user must launch the program first, which in turn launches the browser. Then the user interacts with the browser as usual while the program 'watches'. An event triggered by the user (in this case leaving the target page then coming back to it) triggers the scraping. One nice thing is, it should work equally well (or better) with IE. But I haven't tried it. The Ruby Watir API is by far the easiest way to scrape that I've seen. No need for XPath - I can get anywhere on a complex page I'm scraping in 2 hops, in part of a line of code. For this reason I plan to use Watir even when I don't need to drive the browser. -Dan On 2012-04-02, at 8:15 AM, Trevor Cordes wrote:
On 2012-03-22 Dan Martin wrote:
A programmable browser would be ideal. Does anyone know of one that is multi-platform and can be installed without special services / privileges? Has anyone used XUL for something like this?
I do this type of thing all the time:
perl's WWW::Mechanize::Firefox on cpan. It remote controls FF using MozRepl. You could probably have it run on Windows in cygwin (MozRepl should run natively but perl w/weird modules is tougher).
I've had to hack the suite a touch to get it to work with complicated pages that don't always fire their "done loading" event, though that should not be required and in most cases it works well OOTB.
(And if you need some help, my programming time can be for-hire.) _______________________________________________ Roundtable mailing list Roundtable@muug.mb.ca http://www.muug.mb.ca/mailman/listinfo/roundtable
Dan Martin GP Hospital Practitioner Computer Scientist ummar143@shaw.ca (204) 831-1746 answering machine always on
participants (3)
-
athompso＠muug.mb.ca -
Dan Martin -
Trevor Cordes