So in the intro theory post I mentioned a few times about scripts "calling home". This is the way I handle scaling so it's pretty important to at least understand the concepts behind it. For simple iMacros jobs, it's probably not useful. Since everything I do it to scale, having a mothership to satelite relatinoship (I like using alien terminology) happening from a project's conception is solid ground for me to build my automation on.
If you're big web 2.0 company, you'd call this system an "API". I'm not into labels so I just refer to it as "calling home".
The most basic components to this system are
Apache Mothership - This is just a basic webhost (any will do, domain name not required) that is used for storing all the information necessary for a project. Typically this consists of a webform and a database. (code samples for setting a demo up below)
Windows VPS servers - For iMacros work, I use windows servers because they're cheap, easy to RDC into and I'm very comfortable with the windows environment. Any computer (desktop, laptop or server) will do with some hacking.
The mothership just acts like any other website, sitting around and waiting to serve up web pages.
The VPS servers (workers) wait around for a period of time then requests a webpage from the mothership to see if there is work to do.
Simple & scalable
Here's a very basic setup for our first demo.
If you're big web 2.0 company, you'd call this system an "API". I'm not into labels so I just refer to it as "calling home".
The most basic components to this system are
Apache Mothership - This is just a basic webhost (any will do, domain name not required) that is used for storing all the information necessary for a project. Typically this consists of a webform and a database. (code samples for setting a demo up below)
Windows VPS servers - For iMacros work, I use windows servers because they're cheap, easy to RDC into and I'm very comfortable with the windows environment. Any computer (desktop, laptop or server) will do with some hacking.
The mothership just acts like any other website, sitting around and waiting to serve up web pages.
The VPS servers (workers) wait around for a period of time then requests a webpage from the mothership to see if there is work to do.
Simple & scalable
Here's a very basic setup for our first demo.
CREATE TABLE `database_name`.`imacros_table` ( `id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY , `name` VARCHAR( 255 ) NOT NULL , `password` VARCHAR( 255 ) NOT NULL, `complete` INT NOT NULL ) ENGINE = MYISAM ;
This is a really simple table that stores your name & password for logging into something. Setting up a database with your hosting is beyond the scope of this tutorial, once it's up and running dumping that code into the "SQL" tab of phpMyAdmin will get your table setup.
Aside from name & password, there's also 2 additional fields ... id (auto incrementing database reference for fast queries) & complete (your scripts have to know which record is complete and which are ready to work, yeah?).
HTML FORM
Aside from name & password, there's also 2 additional fields ... id (auto incrementing database reference for fast queries) & complete (your scripts have to know which record is complete and which are ready to work, yeah?).
HTML FORM
<?php
if (isset($_POST['name'])) {
include ('database.php') ;
mysql_query("INSERT INTO imacros_table (name,password) VALUES ('".mysql_real_escape_string($_POST['name'])."','".mysql_real_escape_string($_POST['password'])."')") ;
}
?><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=windows-1250">
</head>
<body>
<form action="" method="POST">
Name: <input name="name" /><br />
Password: <input name="password" /><br /><br />
<input />
</form>
</body>
</html>
So that's the most simple html page with database insertion I could come up with. Basic php is beyond the scope of this tutorial so if you have other questions about this, google it. A successful form submission inserts a username and password into a database.
Now that that's all setup, here comes the fun part .. iMacros. I'm going to break this down into a two chunks so I can explain a little bit about what is going on but the end result will include all of the below code in a .iim file.
Now that that's all setup, here comes the fun part .. iMacros. I'm going to break this down into a two chunks so I can explain a little bit about what is going on but the end result will include all of the below code in a .iim file.
VERSION BUILD=8300326 RECORDER=FX
SET !EXTRACT_TEST_POPUP NO
SET data_url "domain.com"
' Get necessary variables
TAB OPEN
URL GOTO=http://{{data_url}}/data.php?datatype=name
TAG POS=1 ATTR=* EXTRACT=TXT
SET name {{!EXTRACT}}
SET !EXTRACT NULL
URL GOTO=http://{{data_url}}/data.php?datatype=password
TAG POS=1 ATTR=* EXTRACT=TXT
SET password {{!EXTRACT}}
SET !EXTRACT NULL
URL GOTO=http://{{data_url}}/data.php?datatype=id
TAG POS=1 ATTR=* EXTRACT=TXT
SET login_id {{!EXTRACT}}
SET !EXTRACT NULL
TAB CLOSEALLOTHERS
Ok, the above code is just basic setup stuff. The top chunk just uses the imacros SET function to set some variables. The data_url is your "mothership" domain. If you were hosting the mothership on a domain that line would look like this.
SET data_url "mydomain.com"
Since there's no public facing GUI for our automation, putting the IP address there is perfectly fine, and often beneficial as nameservers have tendencies to be flaky
The 2nd chunk of iMacros code above is our script "calling home" to get some information. That information is what we inserted into our database previously. Each "call" is a 4 line chunk, the breakdown is below.
You can copy and paste that over and over again in your imacros code to get as many variables as you need from the mothership. In this example, we're also pulling the "password" entered from that form & the row's "id" so we can mark it as complete when done.
Here's some more iMacros
SET data_url "mydomain.com"
Since there's no public facing GUI for our automation, putting the IP address there is perfectly fine, and often beneficial as nameservers have tendencies to be flaky
The 2nd chunk of iMacros code above is our script "calling home" to get some information. That information is what we inserted into our database previously. Each "call" is a 4 line chunk, the breakdown is below.
- URL GOTO=http://{{data_url}}/data.php?datatype=name (this will literally load up the webpage "data.php" from your mothership with a value of "name" for "data_type". The resulting page will display in firefox)
- TAG POS=1 ATTR=* EXTRACT=TXT (take everything on that page, which is just your name, and save it)
- SET name {{!EXTRACT}} (set the extracted data from the last line and put it into a variable called 'name')
- SET !EXTRACT NULL (empty that variable so it's free to be used again. I've had some quirky experiences with remote data loading in the past, this is kind of a safeguard)
You can copy and paste that over and over again in your imacros code to get as many variables as you need from the mothership. In this example, we're also pulling the "password" entered from that form & the row's "id" so we can mark it as complete when done.
Here's some more iMacros
' LOGIN URL GOTO=wickedfire.com TAG POS=1 :TEXT FORM=ACTION:http://www.wickedfire.com/login.php?do=login ATTR=ID:navbar_username CONTENT={{name}} WAIT SECONDS=2 SET !ENCRYPTION NO TAG POS=1 :PASSWORD FORM=ACTION:http://www.wickedfire.com/login.php?do=login ATTR=ID:navbar_password CONTENT={{password}} WAIT SECONDS=2 TAG POS=1 :SUBMIT FORM=ACTION:http://www.wickedfire.com/login.php?do=login ATTR=* ' LOGOUT WAIT SECONDS=10 TAG POS=1 ATTR=TXT:Log<SP>Out ' PING HOME COMPLETE WAIT SECONDS=2 URL GOTO=http://{{data_url}}/data.php?data_&id={{login_id}}
This part of the script logs in and out of wickedfire (if you entered your real ID/PW into your form. If not, iMacros will stop when it errs out. Again, basic imacros help is beyond the scope of this tutorial. Suffice it to say that everything I learned about this I found searchuing google.
The last part of this long ass post is the PHP required to serve up the data. Here it is in an easy to digest format.
The last part of this long ass post is the PHP required to serve up the data. Here it is in an easy to digest format.
<?php
include ('database.php') ;
$q = mysql_query("SELECT * FROM imacros_table WHERE complete = 0 ORDER BY id ASC LIMIT 0,1") ;
if ($_GET['data_type'] == 'complete') {
mysql_query("UPDATE imacros_table SET complete = 1 WHERE id = ".mysql_real_escape_string($_GET['id'])) ;
exit ('complete') ;
}
if ($_GET['data_type'] == 'name') {
exit (mysql_result($q,0,'name')) ;
} elseif ($_GET['data_type'] == 'password') {
exit (mysql_result($q,0,'password')) ;
} elseif ($_GET['data_type'] == 'id') {
exit ((string)(mysql_result($q,0,'id'))) ;
} else {
exit ('err') ;
}
?>
You should save that PHP as "data.php" and upload it to your mothership. From there, you're good to go playing around with the result. This tutorial is setup to run through a list of usernames/passwords and mark as compete when it's done. Remove or comment out (') the last part of the imacro if you want to be able to run the imacro more than once ... if not, it'll err out somehow.
Obviously, this is a very basic example but hopefully it'll show you how stupid simple this method of automation can be. If you automate this way, your scripts are immune to javascript changes and other wacky shit webmasters will throw your way all in the name of stopping spam. Good luck chumps
Obviously, this is a very basic example but hopefully it'll show you how stupid simple this method of automation can be. If you automate this way, your scripts are immune to javascript changes and other wacky shit webmasters will throw your way all in the name of stopping spam. Good luck chumps