Sol Core Library - Built-Ins
dew.getURL
dew.getURL(string url)
dew.getURL(string url, function callback)
dew.getURL(string url, table headers)
dew.getURL(string url, table headers, function callback)
Description
Perform an HTTP GET request.
Parameters
string url
URL to request; must always include protocol (http/https)
table headers
List of HTTP header variables as strings in a table
{
'Accept: */*';
'Content-Encoding: gzip';
}
function callback(url, data, headers)
Callback for asynchronous requests
url - orignal request url
data - HTTP body (nil on HTTP failure)
headers - HTTP headers (nil on HTTP failure)
Return (No Callback)
string data
string headers
Remarks
When the HTTP request has failed for any reason both data and headers will be nil, otherwise they will both be set. The url parameter will always be set on the callback.
If Content-Encoding header is set to accept compression, the payload must be decompressed if the server returns a compression header as well. Otherwise the return will be garbled looking.
See dew.add_ip to set the source-IP of an outgoing request.
dew.infoURL
dew.infoURL(string url)
dew.infoURL(string url, function callback)
dew.infoURL(string url, table headers)
dew.infoURL(string url, table headers, function callback)
Description
Perform an HTTP HEAD request.
Parameters
string url
URL to request; must always include protocol (http/https)
table headers
List of HTTP header variables as strings in a table
{
'Accept: */*';
'Content-Encoding: gzip';
}
function callback(url, headers)
Callback for asynchronous requests
url - orignal request url
headers - HTTP headers (nil on HTTP failure)
Return (No Callback)
string headers
Remarks
When the HTTP request has failed for any reason headers will be nil, otherwise it will be set. The url parameter will always be set on the callback.
If Content-Encoding header is set to accept compression, the payload must be decompressed if the server returns a compression header as well. Otherwise the return will be garbled looking.
See dew.add_ip to set the source-IP of an outgoing request.
dew.deleteURL
dew.deleteURL(string url)
dew.deleteURL(string url, function callback)
dew.deleteURL(string url, table headers)
dew.deleteURL(string url, table headers, function callback)
Description
Perform an HTTP DELETE request. (identical to HTTP INFO but using DELETE verb)
Parameters
string url
URL to request; must always include protocol (http/https)
table headers
List of HTTP header variables as strings in a table
{
'Accept: */*';
'Content-Encoding: gzip';
}
function callback(url, headers)
Callback for asynchronous requests
url - orignal request url
headers - HTTP headers (nil on HTTP failure)
Return (No Callback)
string headers
Remarks
When the HTTP request has failed for any reason headers will be nil, otherwise it will be set. The url parameter will always be set on the callback.
If Content-Encoding header is set to accept compression, the payload must be decompressed if the server returns a compression header as well. Otherwise the return will be garbled looking.
See dew.add_ip to set the source-IP of an outgoing request.
dew.putURL
dew.putURL(string payload, string url)
dew.putURL(string payload, string url, function callback)
dew.putURL(string payload, string url, table headers)
dew.putURL(string payload, string url, table headers, function callback)
Description
Perform an HTTP PUT request.
The HTTP PUT operation will attempt to upload and store payload at the remote location specified by url
Parameters
string payload
Data sent to the webserver in the PUT operation.
> Could be anything, from contents of a file to a simple string.
string url
URL to request; must always include protocol (http/https)
table headers
List of HTTP header variables as strings in a table
{
'Accept: */*';
'Content-Encoding: gzip';
}
function callback(url, data, headers)
Callback for asynchronous requests
url - orignal request url
data - HTTP response body (nil on HTTP failure, often empty on success)
headers - HTTP response headers (nil on HTTP failure)
Return (No Callback)
string data
string headers
Remarks
The headers HTTP status code will indicate any webserver related errors - network errors will cause data and headers to both be nil. The url parameter will always be set on the callback.
For PUT and POST operations the HTTP return body is often just an empty string: do not mistake this for an error.
See dew.add_ip to set the source-IP of an outgoing request.
dew.postURL
dew.postURL(string payload, string url)
dew.postURL(string payload, string url, function callback)
dew.postURL(string payload, string url, table headers)
dew.postURL(string payload, string url, table headers, function callback)
Description
Perform an HTTP POST request.
The HTTP POST operation will ask a webserver to handle payload using the endpoint specified by url
Parameters
string payload
Data sent to the webserver in the POST operation.
> Could be anything, from contents of a file to a simple string.
string url
URL to request; must always include protocol (http/https)
table headers
List of HTTP header variables as strings in a table
{
'Accept: */*';
'Content-Encoding: gzip';
}
function callback(url, data, headers)
Callback for asynchronous requests
url - orignal request url
data - HTTP response body (nil on HTTP failure, often empty on success)
headers - HTTP response headers (nil on HTTP failure)
Return (No Callback)
string data
string headers
Remarks
The headers HTTP status code will indicate any webserver related errors - network errors will cause data and headers to both be nil. The url parameter will always be set on the callback.
For PUT and POST operations the HTTP return body is often just an empty string: do not mistake this for an error.
See dew.add_ip to set the source-IP of an outgoing request.
dew.patchURL
dew.patchURL(string payload, string url)
dew.patchURL(string payload, string url, function callback)
dew.patchURL(string payload, string url, table headers)
dew.patchURL(string payload, string url, table headers, function callback)
Description
Perform an HTTP PATCH request. (identical to HTTP POST but using PATCH verb)
The HTTP PATCH operation will ask a webserver to handle payload using the endpoint specified by url
Parameters
string payload
Data sent to the webserver in the PATCH operation.
> Could be anything, from contents of a file to a simple string.
string url
URL to request; must always include protocol (http/https)
table headers
List of HTTP header variables as strings in a table
{
'Accept: */*';
'Content-Encoding: gzip';
}
function callback(url, data, headers)
Callback for asynchronous requests
url - orignal request url
data - HTTP response body (nil on HTTP failure, often empty on success)
headers - HTTP response headers (nil on HTTP failure)
Return (No Callback)
string data
string headers
Remarks
The headers HTTP status code will indicate any webserver related errors - network errors will cause data and headers to both be nil. The url parameter will always be set on the callback.
For PUT and POST operations the HTTP return body is often just an empty string: do not mistake this for an error.
See dew.add_ip to set the source-IP of an outgoing request.
dew.add_ip
dew.add_ip(string ip_address)
Description
Adds an IP address for random selection.
All IP addresses added are stored in a list and used at random for any network related activities performed by Sol Core Libraries
Parameters
string ip_address
IPv4 or IPv6 address to be added (See format below):
> dew.add_ip('host!50.2.191.82')
> dew.add_ip('host!2607:ff28:2:1f:be5a:39b9:f5be:dda3')
Remarks
If dew.add_ip is never called then the default IP-address on the default interface is used.
The IP protocol version defaults to IPv4 but can be set by dew.preferip to prioritize IPv6.
All addresses added with dew.add_ip are stored in an array and are chosen at random for network operations performed by a Sol instance.
dew.preferip
dew.preferip(number ip_protocol_version)
Description
Sets the DNS resolver preference to always prioritize IPv4 or IPv6 DNS lookups.
Parameters
number ip_protocol_version
> Valid inputs are 4 and 6
Remarks
Sol contains its own DNS resolver, which defaults to IPv4 if dew.preferip is not called.
Many hostnames only have records for IPv4 or IPv6.
If a requested URL's hostname contains only one such DNS record Sol resolver will first try its preferred protocol but will fall back onto the other upon lookup failure.
If IPv6 is selected but no valid addresses have been added through dew.add_ip, the IPv6 DNS request may properly resolve but the connection will fail.
At least one address must be added before the corresonding protocol can be used.
Most interface will generally have at least one default IPv4 and IPv6 address, however.
dew.dir
dew.dir(string directory_path)
dew.dir(string directory_path, boolean deep)
Description
dew.dir returns a list of every file and folder in a directory, or directory chain.
Will recurse child directories if deep is true
Parameters
string directory_path
Any valid directory path:
> "my_folder"
> "/root/images"
> "C:\\books"
boolean deep
true = recursively list subdirectories
false = list current folder only
Return
table files
Remarks
dew.dir has a simple rule for directories:
On Windows: Directories will end in a \character
On Linux: Directories will end in a /character
dew.setDNS
dew.setDNS(string dns_resolver_ip)
Description
dew.setDNS sets the DNS resolver server for all DNS lookups
Parameters
string dns_resolver_ip
> "192.168.0.1"
Remarks
Default: 8.8.8.8
All DNS lookups in Sol are handled by Sol framework.
dew.exec
dew.exec(string path, function callback)
dew.exec(string path, function callback, boolean async)
Description
dew.exec executes a command line arguement on Windows/Linux and provides a handle to stdin, and a callback for the stdout/stderr. This lets Sol to recieve output from, and send messages to, the process. Enabling easy automation of command line applications.
Parameters
string path
This is the path or command line argument for the process being run.
On Linux, this parameter is processed with /bin/dash
> "ping yahoo.com"
> "systeminfo"
function callback(string msg, pointer hPipe)
This is a callback that receives all messages written to the created application's stdout/stderr pipes.
The msg variable contains the contents of each read operation on the stdout/stderr pipe of the executed application. This means a single write operation from the host application may not translate into a single callback event!
callback also contains hPipe which can be used with dew.tell to write messages to a program's stdin interface.
Return (Only Async)
pointer hPipe
Remarks
If you are building an application specifically for use with dew.exec, there is a function in the Sol Framework's core lua libraries which is setup to handle fragmented messages and provide callbacks with the full payload. This makes the extension of Sol's features in another language effortless.
This function was added to provice a simplified way of interfacing with the operating system by parsing the output of common console commands. Sol Framework provides a number of utility functions extending and simplifying the use for certain cases.
It is not necessary to call dew.close on hPipe as the pipe is closed automatically when the called application terminates.
dew.tell
dew.tell(pointer hPipe, string msg)
Description
dew.tell writes msg to the stdin interface hPipe. Allowing input as if a user was typing into the application on the command line.
Parameters
pointer hPipe
This pointer is a handle to the stdin of a program executed with dew.exec
string msg
The message to transmit to the program that owns hPipe
Remarks
Some applications like wc on linux expect the stdin handle to be closed before they will continue, and thus dew.close was added for this purpose. For applications expecting input only once, this is often required.
The parameter hPipe is both returned, and passed to, the callback function of dew.exec async call.
dew.close
dew.close(pointer handle)
Description
Closes a handle to an object. Currently this is used for dew.exec only, but may be used as a general handle close for other handles exposed to Lua in future Sol updates.
Note: May replace dew.freeHTML(undocumented) and other free/close calls in the future; allowing a simple to maintain branch table optimized function on the backend.
Parameters
pointer handle
This pointer is a handle representing an object allocated in C
Remarks
This must be called whenever it is stated as a requirement for another API; otherwise a memory leak will occur. This API will free any memory allocated by C, where it is required.
std.wait
std.wait()
Description
Waits for all asyncronouse calls to finish before returning. This function is integral for syncronizing a group of calls that must finish before beginning another group.
Remarks
This must only be called inside the main thread! If this function is called inside one of the asynchronous callbacks, it will stall forever, thus freezing the program.
std.sleep
std.sleep(number milliseconds)
Description
Pauses the execution of a Lua routine for milliseconds number of milliseconds, while yeilding execution to other threads. This sleep must be used if the goal is not to block all multi-threaded code.
(Calls NtDelayExecution system call on Windows)
(Calls nanosleep system call on Linux)
Parameters
number milliseconds
The value in milliseconds to sleep (Windows/Linux supported)
Remarks
While sleeping, any other asyncronous Lua functions, like dew.getURL, are still able to execute their callbacks.
Sol has a global lock on the base Lua state to prevent stack corruption, but has highly multi-threaded and performance oriented C code behind the Lua calls allowing high performance parallelism.
Sol Core Library - sol.lua
table.dump
table.dump(table tbl, boolean human_readable)
Library
sol.lua
Description
Special Function for debugging table datastructures.
Recursively converts table into a valid lua string representing the table structure.
Parameters
table tbl
boolean human_readable
Set to true if return characters should be added
Set to false if entire string should be one line
Return
string table_string
Remarks
Performance-wise this function is very slow and should not be used in production code, except for the purpose of debug and/or crash output.
This function attempts to generate a valid lua-syntax table, but this is not always possible.
There are currently a few bugs converting tables with strings containing newline characters and other things which are invalid if printed without being escaped.
This function will also print tables containing keys or values with types like function, which will appear as their pointer values, and cannot be used as valid Lua input.
Example
local tbl = {'taco', [{3,2,1}]=92, 'burito', price={dollars=12, cents=24}}
print(table.dump(tbl, true))
{
'taco',
'burito',
[{
3,
2,
1
}]=92,
price={
cents=24,
dollars=12
}
}
string:enc64
string:enc64()
Library
sol.lua
Description
String method to convert any string into it's base64 representation.
Base64 is a popular technique for binary encoding on the web, and is likely the primary method for embedding images directly in HTML documents and storing other binary information.
Return
string encoded_data
Remarks
This function will never return nil.
If string has a length of 0, the output is also an empty string and not a valid base64 sequence.
All input with more than length 0 will return a valid base64 sequence.
Example
> msg = 'Hello World'
> print(msg:enc64())
SGVsbG8gV29ybGQ=
string:dec64
string:dec64()
Library
sol.lua
Description
String method to convert any base64 string into its original byte representation.
Base64 is a popular technique for binary encoding on the web, and is likely the primary method for embedding images directly in HTML documents and storing other binary information.
Return
string decoded_data
Remarks
This function will never return nil.
If string has a length of 0, the output is also an empty string
Input to this function is not checked for validity, and will run dec64 data permutations on anything passed.
It is up to the programmer to keep track and verify that string is a base64 string.
While providing invalid input shouldn't crash, it will produce equally invalid output.
Example
> secret_msg = "RG9jdG9yIGRvY3RvciE="
> print(secret_msg:dec64())
Doctor doctor!
string:gzip
string:gzip()
string:gzip(number level)
Library
sol.lua
Description
String method to compress input string using gzip compression algorithm with a specified compression level.
Parameters
number level
Valid compression levels are 1-9, higher being better compression.
If no argument is passed, level defaults to 9.
Return
string compressed_data
Remarks
It is reccomended that the string data be at least 100 bytes or it is likely to be uncompressable.
If uncompressable data is passed, a nil value is returned.
If the input is not known to be compressable then the return must always be checked to see if it has returned nil or a string value.
This is the only way to handle uncompressable data without creating more problems by assuming the return is always compressed.
string:gunzip
string:gunzip()
Library
sol.lua
Description
String method to uncompress input data encoded with gzip compression.
Return
string uncompressed_data
Remarks
This function may return nil upon various errors, and only returns a string on total success.
This is one of the most volatile funtions, and it is incredibly important to pass only valid gzip information or this function may have unknown effect.
Handling errors when passing invalid data to this function is one of the top priorities for future patches and tweaks.
string:query
string:query(string query)
Library
sol.lua
Description
Comprehensive method for processing HTML5 documents.
Query syntax based on css selector is used to match nodes in an HTML document and provide flexable mechanisms for efficiently extracting information and navigation.
CSS Selector Reference: http://www.w3schools.com/cssref/css_selectors.asp
Return
table QueryResult
Structures
QueryResult
{
number [1]; Starting position of tag start. Eg: <b>Hello World</b>
number [2]; Starting position of content start. Eg: <b>Hello World</b>
number [3]; Starting position of content end. Eg: <b>Hello World</b>
number [4]; Starting position of tag end. Eg: <b>Hello World</b>
string tag; name of tag
string txt; Sanitized text (html removed) inside QueryResult tag region
cdata parent; Symbolic unique identifier for parent node of QueryResult
number order; Child node order in relation to other child nodes under the same parent
}
Remarks
Not all CSS selectors are supported; generally more complicated or less common selectors will not work.
If a desired selector fails: try to simplify the query and or programmatically filter the results (sol is really fast, and this is never a bad option).
parent and order are symbolic only, and are not references to objects or tables.
These exist to provide information required for the reconstruction of part or all of the source document from its nodes.
They may also be useful analytically when searching for a node which is unable to be identified with a simple CSS selector query.
string:literalize
string:literalize()
Library
sol.lua
Description
Escapes all lua pattern special-characters for use in string.gsub without being processed as a pattern expression.
This should be used whenever gsub input is taken from a varaible.
Return
string literal_string
Remarks
Replaces ^ $ ( ) % . [ ] * + - ? \0 with %^ %$ %( %) %% %. %[ %] %* %+ %- %? %z
string:hex
string:hex()
string:hex(number type)
Library
sol.lua
Description
Convert string into one of the supported hex representations.
Argument is not required. Default sequence is used.
Parameters
number type
Valid types:
0 - Normal Hex (default)
1 - Percent Encoded
2 - Escaped Sequence
Return
string hex_string
Example
> local msg = 'Hello World'
> print(msg:hex())
48656C6C6F20576F726C64
> print(msg:hex(0))
48656C6C6F20576F726C64
> print(msg:hex(1))
%48%65%6C%6C%6F%20%57%6F%72%6C%64
> print(msg:hex(2))
\x48\x65\x6C\x6C\x6F\x20\x57\x6F\x72\x6C\x64
ENV
ENV(string locals, function func)
Library
sol.lua
Description
Passes a callback function variables from the current calling environment as a table injected into the first argument of the callback; other arguments are unchanged and will follow table env.
This is a thread compatability function. Any callbacks relying on local scope data at the time of being registered must be passed the local data using ENV.
Return
function ModifiedCallback
Remarks
fix me: This is a critical function but hard to explain without example. Probably will add numerous examples in lieu of an english elaboration.
Sol URL Library - url.lua
urtl.absolute
urtl.absolute(string base_url, string anchor_url)
Library
url.lua
Description
Convert relative URL to full URL.
When scraping a website base_url is the page URL,
and anchor_url would come from <a href="anchor_url">
Return
string full_url
Remarks
There is one exception to this. The HTML base tag may specify an alternate base_url.
The URL specified by the base tag will take precident over the page URL.
urtl.parse
urtl.parse(string url)
Library
url.lua
Description
Opposite of urlutil.build
Parses URL and returns a table of its elements.
According to:
RFC 1738: https://tools.ietf.org/html/rfc1738
RFC 1808: https://tools.ietf.org/html/rfc1808
RFC 3968: https://tools.ietf.org/html/rfc3986
Return
table UrlInfo
Structures
UrlInfo
{
string scheme;
string authority;
string path;
string query;
string fragment;
string user;
string password;
string userinfo;
string host;
string port;
string params;
}
Remarks
foo://example.com:8042/over/there?name=ferret#nose
\_/ \______________/\_________/ \_________/ \__/
| | | | |
scheme authority path query fragment
user password
| |
/--\ /------\
wtf://user:[email protected]:8042/apath;type=a?name=ferret#nose
\___________/ \_________/ \__/ \____/
| | | |
userinfo host port params
urtl.build
urtl.build(table UrlInfo)
Library
url.lua
Description
Opposite of urlutil.parse
(Not every field needs a value)
Takes UrlInfo structure and builds a url string from its fields.
According to:
RFC 1738: https://tools.ietf.org/html/rfc1738
RFC 1808: https://tools.ietf.org/html/rfc1808
RFC 3968: https://tools.ietf.org/html/rfc3986
Return
string url
Structures
UrlInfo
{
string scheme;
string authority;
string path;
string query;
string fragment;
string user;
string password;
string userinfo;
string host;
string port;
string params;
}
Remarks
foo://example.com:8042/over/there?name=ferret#nose
\_/ \______________/\_________/ \_________/ \__/
| | | | |
scheme authority path query fragment
user password
| |
/--\ /------\
wtf://user:[email protected]:8042/apath;type=a?name=ferret#nose
\___________/ \_________/ \__/ \____/
| | | |
userinfo host port params
urtl.escape
urtl.escape(string str)
Library
url.lua
Description
Escape string segment containing special characters as their percent encoded values
Return
string escaped_str
Example
> local url = "~ Hello World@!(ROFL)"
> print(urlutil.escape(url))
%7e%20Hello%20World%40%21%28ROFL%29
urtl.unescape
urtl.unescape(string str)
Library
url.lua
Description
Convert escaped characters in a url back into their raw values.
Return
string unescaped_str
Example
> local url = %7e%20Hello%20World%40%21%28ROFL%29"
> print(urlutil.escape(url))
~ Hello World@!(ROFL)