Sol Core Library - Built-Ins

 

dew.getURL

dew.getURL(string url)

dew.getURL(string url, function callback)

dew.getURL(string url, table headers)

dew.getURL(string url, table headers, function callback)

 

Description

Perform an HTTP GET request.

 

Parameters

          string url

   URL to request; must always include protocol (http/https)

   'https://purple.com/'

 

          table headers

          List of HTTP header variables as strings in a table

          {

             'Accept: */*';

             'Content-Encoding: gzip';

          }

 

          function callback(url, data, headers)

          Callback for asynchronous requests

            url     - orignal request url

            data    - HTTP body     (nil on HTTP failure)

            headers - HTTP headers  (nil on HTTP failure)

 

Return (No Callback)

          string data

          string headers

 

Remarks

When the HTTP request has failed for any reason both data and headers will be nil, otherwise they will both be set.  The url parameter will always be set on the callback.

 

If Content-Encoding header is set to accept compression, the payload must be decompressed if the server returns a compression header as well.  Otherwise the return will be garbled looking.

 

See dew.add_ip to set the source-IP of an outgoing request.

 

 

 

 

 

dew.infoURL

dew.infoURL(string url)

dew.infoURL(string url, function callback)

dew.infoURL(string url, table headers)

dew.infoURL(string url, table headers, function callback)

 

Description

Perform an HTTP HEAD request.

 

Parameters

          string url

   URL to request; must always include protocol (http/https)

   'https://purple.com/'

 

          table headers

          List of HTTP header variables as strings in a table

          {

             'Accept: */*';

             'Content-Encoding: gzip';

          }

 

          function callback(url, headers)

          Callback for asynchronous requests

            url     - orignal request url

            headers - HTTP headers  (nil on HTTP failure)

 

Return (No Callback)

          string headers

 

Remarks

When the HTTP request has failed for any reason headers will be nil, otherwise it will be set.  The url parameter will always be set on the callback.

 

If Content-Encoding header is set to accept compression, the payload must be decompressed if the server returns a compression header as well.  Otherwise the return will be garbled looking.

 

See dew.add_ip to set the source-IP of an outgoing request.

 

 

 

 

 

dew.deleteURL

dew.deleteURL(string url)

dew.deleteURL(string url, function callback)

dew.deleteURL(string url, table headers)

dew.deleteURL(string url, table headers, function callback)

 

Description

Perform an HTTP DELETE request. (identical to HTTP INFO but using DELETE verb)

 

Parameters

          string url

   URL to request; must always include protocol (http/https)

   'https://purple.com/'

 

          table headers

          List of HTTP header variables as strings in a table

          {

             'Accept: */*';

             'Content-Encoding: gzip';

          }

 

          function callback(url, headers)

          Callback for asynchronous requests

            url     - orignal request url

            headers - HTTP headers  (nil on HTTP failure)

 

Return (No Callback)

          string headers

 

Remarks

When the HTTP request has failed for any reason headers will be nil, otherwise it will be set.  The url parameter will always be set on the callback.

 

If Content-Encoding header is set to accept compression, the payload must be decompressed if the server returns a compression header as well.  Otherwise the return will be garbled looking.

 

See dew.add_ip to set the source-IP of an outgoing request.

 

 

 

 

 

dew.putURL

dew.putURL(string payload, string url)

dew.putURL(string payload, string url, function callback)

dew.putURL(string payload, string url, table headers)

dew.putURL(string payload, string url, table headers, function callback)

 

Description

Perform an HTTP PUT request.

The HTTP PUT operation will attempt to upload and store payload at the remote location specified by url

 

Parameters

          string payload

   Data sent to the webserver in the PUT operation.

   > Could be anything, from contents of a file to a simple string.

 

          string url

   URL to request; must always include protocol (http/https)

   'https://purple.com/'

 

          table headers

          List of HTTP header variables as strings in a table

          {

             'Accept: */*';

             'Content-Encoding: gzip';

          }

 

          function callback(url, data, headers)

          Callback for asynchronous requests

            url     - orignal request url

            data    - HTTP response body     (nil on HTTP failure, often empty on success)

            headers - HTTP response headers  (nil on HTTP failure)

 

Return (No Callback)

          string data

          string headers

 

Remarks

The headers HTTP status code will indicate any webserver related errors - network errors will cause data and headers to both be nil.  The url parameter will always be set on the callback

 

For PUT and POST operations the HTTP return body is often just an empty string: do not mistake this for an error. 

 

See dew.add_ip to set the source-IP of an outgoing request.

 

 

 

 

 

dew.postURL

dew.postURL(string payload, string url)

dew.postURL(string payload, string url, function callback)

dew.postURL(string payload, string url, table headers)

dew.postURL(string payload, string url, table headers, function callback)

 

Description

Perform an HTTP POST request.

The HTTP POST operation will ask a webserver to handle payload using the endpoint specified by url

 

Parameters

          string payload

   Data sent to the webserver in the POST operation.

   > Could be anything, from contents of a file to a simple string.

 

          string url

   URL to request; must always include protocol (http/https)

   'https://purple.com/'

 

          table headers

          List of HTTP header variables as strings in a table

          {

             'Accept: */*';

             'Content-Encoding: gzip';

          }

 

          function callback(url, data, headers)

          Callback for asynchronous requests

            url     - orignal request url

            data    - HTTP response body     (nil on HTTP failure, often empty on success)

            headers - HTTP response headers  (nil on HTTP failure)

 

Return (No Callback)

          string data

          string headers

 

Remarks

The headers HTTP status code will indicate any webserver related errors - network errors will cause data and headers to both be nil.  The url parameter will always be set on the callback

 

For PUT and POST operations the HTTP return body is often just an empty string: do not mistake this for an error. 

 

See dew.add_ip to set the source-IP of an outgoing request.

 

 

 

 

 

dew.patchURL

dew.patchURL(string payload, string url)

dew.patchURL(string payload, string url, function callback)

dew.patchURL(string payload, string url, table headers)

dew.patchURL(string payload, string url, table headers, function callback)

 

Description

Perform an HTTP PATCH request. (identical to HTTP POST but using PATCH verb)

The HTTP PATCH operation will ask a webserver to handle payload using the endpoint specified by url

 

Parameters

          string payload

   Data sent to the webserver in the PATCH operation.

   > Could be anything, from contents of a file to a simple string.

 

          string url

   URL to request; must always include protocol (http/https)

   'https://purple.com/'

 

          table headers

          List of HTTP header variables as strings in a table

          {

             'Accept: */*';

             'Content-Encoding: gzip';

          }

 

          function callback(url, data, headers)

          Callback for asynchronous requests

            url     - orignal request url

            data    - HTTP response body     (nil on HTTP failure, often empty on success)

            headers - HTTP response headers  (nil on HTTP failure)

 

Return (No Callback)

          string data

          string headers

 

Remarks

The headers HTTP status code will indicate any webserver related errors - network errors will cause data and headers to both be nil.  The url parameter will always be set on the callback

 

For PUT and POST operations the HTTP return body is often just an empty string: do not mistake this for an error. 

 

See dew.add_ip to set the source-IP of an outgoing request.

 

 

 

 

 

dew.add_ip

dew.add_ip(string ip_address)

 

Description

Adds an IP address for random selection.

All IP addresses added are stored in a list and used at random for any network related activities performed by Sol Core Libraries

 

Parameters

          string ip_address

   IPv4 or IPv6 address to be added (See format below):

   > dew.add_ip('host!50.2.191.82')

   > dew.add_ip('host!2607:ff28:2:1f:be5a:39b9:f5be:dda3')

 

Remarks

If dew.add_ip is never called then the default IP-address on the default interface is used.

The IP protocol version defaults to IPv4 but can be set by dew.preferip to prioritize IPv6.

 

All addresses added with dew.add_ip are stored in an array and are chosen at random for network operations performed by a Sol instance.

 

 

 

 

 

dew.preferip

dew.preferip(number ip_protocol_version)

 

Description

Sets the DNS resolver preference to always prioritize IPv4 or IPv6 DNS lookups.

 

Parameters

          number ip_protocol_version

            > Valid inputs are 4 and 6

 

Remarks

Sol contains its own DNS resolver, which defaults to IPv4 if dew.preferip is not called.

 

Many hostnames only have records for IPv4 or IPv6.

If a requested URL's hostname contains only one such DNS record Sol resolver will first try its preferred protocol but will fall back onto the other upon lookup failure.

 

If IPv6 is selected but no valid addresses have been added through dew.add_ip, the IPv6 DNS request may properly resolve but the connection will fail.

At least one address must be added before the corresonding protocol can be used.

Most interface will generally have at least one default IPv4 and IPv6 address, however.

 

 

 

 

 

dew.dir

dew.dir(string directory_path)

dew.dir(string directory_path, boolean deep)

 

Description

dew.dir returns a list of every file and folder in a directory, or directory chain.

Will recurse child directories if deep is true

 

Parameters

          string directory_path

       Any valid directory path:

       > "my_folder"

       > "/root/images"

       > "C:\\books"

 

          boolean deep

       true  = recursively list subdirectories

       false = list current folder only

 

Return

          table files

 

Remarks

dew.dir has a simple rule for directories:

    On Windows: Directories will end in a \character

    On Linux: Directories will end in a /character

 

 

 

 

 

dew.setDNS

                dew.setDNS(string dns_resolver_ip)

 

Description

dew.setDNS sets the DNS resolver server for all DNS lookups

 

Parameters

          string dns_resolver_ip

       > "192.168.0.1"

 

Remarks

Default: 8.8.8.8

All DNS lookups in Sol are handled by Sol framework.

 

 

 

 

 

dew.exec

dew.exec(string path, function callback)

dew.exec(string path, function callback, boolean async)

 

Description

dew.exec executes a command line arguement on Windows/Linux and provides a handle to stdin, and a callback for the stdout/stderr.  This lets Sol to recieve output from, and send messages to, the process. Enabling easy automation of command line applications.

 

Parameters

          string path

       This is the path or command line argument for the process being run.

       On Linux, this parameter is processed with /bin/dash

       > "ping yahoo.com"

       > "systeminfo"

 

          function callback(string msg, pointer hPipe)

This is a callback that receives all messages written to the created application's stdout/stderr pipes.

The msg variable contains the contents of each read operation on the stdout/stderr pipe of the executed application.  This means a single write operation from the host application may not translate into a single callback event!

callback also contains hPipe which can be used with dew.tell to write messages to a program's stdin interface.

 

Return (Only Async)

          pointer hPipe

 

Remarks

If you are building an application specifically for use with dew.exec, there is a function in the Sol Framework's core lua libraries which is setup to handle fragmented messages and provide callbacks with the full payload.  This makes the extension of Sol's features in another language effortless.

 

This function was added to provice a simplified way of interfacing with the operating system by parsing the output of common console commands.  Sol Framework provides a number of utility functions extending and simplifying the use for certain cases.

 

It is not necessary to call dew.close on hPipe as the pipe is closed automatically when the called application terminates.

 

 

 

 

 

dew.tell

                dew.tell(pointer hPipe, string msg)

 

Description

dew.tell writes msg to the stdin interface hPipe.  Allowing input as if a user was typing into the application on the command line.

 

Parameters

          pointer hPipe

       This pointer is a handle to the stdin of a program executed with dew.exec

 

          string msg

       The message to transmit to the program that owns hPipe

 

Remarks

Some applications like wc on linux expect the stdin handle to be closed before they will continue, and thus dew.close was added for this purpose.  For applications expecting input only once, this is often required.

The parameter hPipe is both returned, and passed to, the callback function of dew.exec async call.

 

 

 

 

 

dew.close

                dew.close(pointer handle)

 

Description

Closes a handle to an object.  Currently this is used for dew.exec only, but may be used as a general handle close for other handles exposed to Lua in future Sol updates.

 

Note: May replace dew.freeHTML(undocumented) and other free/close calls in the future; allowing a simple to maintain branch table optimized function on the backend.

 

Parameters

          pointer handle

       This pointer is a handle representing an object allocated in C

 

Remarks

This must be called whenever it is stated as a requirement for another API; otherwise a memory leak will occur.  This API will free any memory allocated by C, where it is required.

 

 

 

 

 

std.wait

                std.wait()

 

Description

Waits for all asyncronouse calls to finish before returning.  This function is integral for syncronizing a group of calls that must finish before beginning another group.

 

Remarks

This must only be called inside the main thread!  If this function is called inside one of the asynchronous callbacks, it will stall forever, thus freezing the program.

 

 

 

 

 

 

std.sleep

                std.sleep(number milliseconds)

 

Description

Pauses the execution of a Lua routine for milliseconds number of milliseconds, while yeilding execution to other threads.  This sleep must be used if the goal is not to block all multi-threaded code.

       (Calls NtDelayExecution system call on Windows)

       (Calls nanosleep system call on Linux)

 

Parameters

          number milliseconds

       The value in milliseconds to sleep (Windows/Linux supported)

 

Remarks

While sleeping, any other asyncronous Lua functions, like dew.getURL, are still able to execute their callbacks.

Sol has a global lock on the base Lua state to prevent stack corruption, but has highly multi-threaded and performance oriented C code behind the Lua calls allowing high performance parallelism.

 

 

 

 

 

 

Sol Core Library - sol.lua

 

table.dump

table.dump(table tbl, boolean human_readable)

 

Library

          sol.lua

 

Description

Special Function for debugging table datastructures.

Recursively converts table into a valid lua string representing the table structure.

 

Parameters

          table tbl

 

          boolean human_readable

                Set to true if return characters should be added          

                Set to false if entire string should be one line

 

Return

          string table_string

 

Remarks

Performance-wise this function is very slow and should not be used in production code, except for the purpose of debug and/or crash output.

 

This function attempts to generate a valid lua-syntax table, but this is not always possible.

There are currently a few bugs converting tables with strings containing newline characters and other things which are invalid if printed without being escaped.

 

This function will also print tables containing keys or values with types like function, which will appear as their pointer values, and cannot be used as valid Lua input.

               

Example

local tbl = {'taco', [{3,2,1}]=92, 'burito', price={dollars=12, cents=24}}

print(table.dump(tbl, true))

{

  'taco',

  'burito',

  [{

    3,

    2,

    1

  }]=92,

  price={

    cents=24,

    dollars=12

  }

}

 

 

 

 

 

string:enc64

string:enc64()

 

Library

          sol.lua

 

Description

String method to convert any string into it's base64 representation.

Base64 is a popular technique for binary encoding on the web, and is likely the primary method for embedding images directly in HTML documents and storing other binary information.

 

Return

          string encoded_data

 

Remarks

This function will never return nil.

If string has a length of 0, the output is also an empty string and not a valid base64 sequence.

All input with more than length 0 will return a valid base64 sequence.

 

Example

> msg = 'Hello World'

> print(msg:enc64())

SGVsbG8gV29ybGQ=

 

 

 

 

 

string:dec64

string:dec64()

 

Library

          sol.lua

 

Description

String method to convert any base64 string into its original byte representation.

Base64 is a popular technique for binary encoding on the web, and is likely the primary method for embedding images directly in HTML documents and storing other binary information.

 

Return

          string decoded_data

 

Remarks

This function will never return nil.

If string has a length of 0, the output is also an empty string

Input to this function is not checked for validity, and will run dec64 data permutations on anything passed.

 

It is up to the programmer to keep track and verify that string is a base64 string.

While providing invalid input shouldn't crash, it will produce equally invalid output.

 

Example

> secret_msg = "RG9jdG9yIGRvY3RvciE="

> print(secret_msg:dec64())

Doctor doctor!

 

 

 

 

 

string:gzip

string:gzip()

string:gzip(number level)

 

Library

          sol.lua

 

Description

String method to compress input string using gzip compression algorithm with a specified compression level.

 

Parameters

          number level

   Valid compression levels are 1-9, higher being better compression.

   If no argument is passed, level defaults to 9.

 

Return

          string compressed_data

 

Remarks

It is reccomended that the string data be at least 100 bytes or it is likely to be uncompressable.

If uncompressable data is passed, a nil value is returned. 

 

If the input is not known to be compressable then the return must always be checked to see if it has returned nil or a string value.

This is the only way to handle uncompressable data without creating more problems by assuming the return is always compressed.

 

 

 

 

 

string:gunzip

string:gunzip()

 

Library

          sol.lua

 

Description

String method to uncompress input data encoded with gzip compression.

 

Return

          string uncompressed_data

 

Remarks

This function may return nil upon various errors, and only returns a string on total success.

 

This is one of the most volatile funtions, and it is incredibly important to pass only valid gzip information or this function may have unknown effect.

Handling errors when passing invalid data to this function is one of the top priorities for future patches and tweaks.

 

 

 

 

 

string:query

string:query(string query)

 

Library

          sol.lua

 

Description

Comprehensive method for processing HTML5 documents.

Query syntax based on css selector is used to match nodes in an HTML document and provide flexable mechanisms for efficiently extracting information and navigation.

CSS Selector Reference: http://www.w3schools.com/cssref/css_selectors.asp

 

Return

          table QueryResult

 

Structures

 

QueryResult

{

      number [1];            Starting position of tag start.         Eg: <b>Hello World</b>

      number [2];            Starting position of content start. Eg: <b>Hello World</b>

      number [3];            Starting position of content end.  Eg: <b>Hello World</b>

      number [4];            Starting position of tag end.          Eg: <b>Hello World</b>

      string tag;           name of tag

      string txt;            Sanitized text (html removed) inside QueryResult tag region

        cdata parent;     Symbolic unique identifier for parent node of QueryResult

      number order;       Child node order in relation to other child nodes under the same parent

}

 

Remarks

Not all CSS selectors are supported; generally more complicated or less common selectors will not work.

If a desired selector fails: try to simplify the query and or programmatically filter the results (sol is really fast, and this is never a bad option).

 

parent and order are symbolic only, and are not references to objects or tables.

These exist to provide information required for the reconstruction of part or all of the source document from its nodes.

They may also be useful analytically when searching for a node which is unable to be identified with a simple CSS selector query.

 

 

 

 

 

string:literalize

string:literalize()

 

Library

          sol.lua

 

Description

Escapes all lua pattern special-characters for use in string.gsub without being processed as a pattern expression.

This should be used whenever gsub input is taken from a varaible.

 

Return

          string literal_string

 

Remarks

Replaces ^ $ ( ) % . [ ] * + - ? \0 with %^ %$ %( %) %% %. %[ %] %* %+ %- %? %z

 

 

 

 

 

string:hex

string:hex()

string:hex(number type)

 

Library

          sol.lua

 

Description

Convert string into one of the supported hex representations.

Argument is not required. Default sequence is used.

 

Parameters

          number type

   Valid types:

       0 - Normal Hex (default)

       1 - Percent Encoded

       2 - Escaped Sequence

Return

          string hex_string

 

Example

> local msg = 'Hello World'

> print(msg:hex())

48656C6C6F20576F726C64

> print(msg:hex(0))

48656C6C6F20576F726C64

> print(msg:hex(1))

%48%65%6C%6C%6F%20%57%6F%72%6C%64

> print(msg:hex(2))

\x48\x65\x6C\x6C\x6F\x20\x57\x6F\x72\x6C\x64

 

 

 

 

 

ENV

ENV(string locals, function func)

 

Library

          sol.lua

 

Description

Passes a callback function variables from the current calling environment as a table injected into the first argument of the callback; other arguments are unchanged and will follow table env.

This is a thread compatability function. Any callbacks relying on local scope data at the time of being registered must be passed the local data using ENV.

 

Return

          function ModifiedCallback

 

Remarks

fix me:  This is a critical function but hard to explain without example.  Probably will add numerous examples in lieu of an english elaboration.

 

 

 

 

 

Sol URL Library - url.lua

 

urtl.absolute

urtl.absolute(string base_url, string anchor_url)

 

Library

          url.lua

 

Description

Convert relative URL to full URL.

 

When scraping a website base_url is the page URL,

and anchor_url would come from <a href="anchor_url">

 

Return

          string full_url

 

Remarks

There is one exception to this.  The HTML base tag may specify an alternate base_url.

The URL specified by the base tag will take precident over the page URL.

 

 

 

 

 

urtl.parse

urtl.parse(string url)

 

Library

          url.lua

 

Description

Opposite of urlutil.build

Parses URL and returns a table of its elements.

According to:

RFC 1738: https://tools.ietf.org/html/rfc1738

RFC 1808: https://tools.ietf.org/html/rfc1808

RFC 3968: https://tools.ietf.org/html/rfc3986

 

Return

          table UrlInfo

 

Structures

 

UrlInfo

{

      string scheme;

      string authority;

      string path;

      string query;

      string fragment;

      string user;

      string password;

      string userinfo;

      string host;

      string port;

      string params;

}

 

Remarks

         foo://example.com:8042/over/there?name=ferret#nose

         \_/   \______________/\_________/ \_________/ \__/

          |           |            |            |        |

       scheme     authority       path        query   fragment

 

 

               user password

                |      |

               /--\ /------\

         wtf://user:[email protected]:8042/apath;type=a?name=ferret#nose

               \___________/ \_________/ \__/       \____/

                   |              |       |           |

                userinfo         host    port        params

 

 

 

 

 

urtl.build

urtl.build(table UrlInfo)

 

Library

          url.lua

 

Description

Opposite of urlutil.parse

 (Not every field needs a value)

Takes UrlInfo structure and builds a url string from its fields.

According to:

RFC 1738: https://tools.ietf.org/html/rfc1738

RFC 1808: https://tools.ietf.org/html/rfc1808

RFC 3968: https://tools.ietf.org/html/rfc3986

 

Return

          string url

 

Structures

 

UrlInfo

{

      string scheme;

      string authority;

      string path;

      string query;

      string fragment;

      string user;

      string password;

      string userinfo;

      string host;

      string port;

      string params;

}

 

Remarks

         foo://example.com:8042/over/there?name=ferret#nose

         \_/   \______________/\_________/ \_________/ \__/

          |           |            |            |        |

       scheme     authority       path        query   fragment

 

 

               user password

                |      |

               /--\ /------\

         wtf://user:[email protected]:8042/apath;type=a?name=ferret#nose

               \___________/ \_________/ \__/       \____/

                   |              |       |           |

                userinfo         host    port        params

 

 

 

 

 

urtl.escape

urtl.escape(string str)

 

Library

          url.lua

 

Description

Escape string segment containing special characters as their percent encoded values

 

Return

          string escaped_str

 

Example

> local url = "~ Hello [email protected]!(ROFL)"

> print(urlutil.escape(url))

%7e%20Hello%20World%40%21%28ROFL%29

 

 

 

 

 

urtl.unescape

urtl.unescape(string str)

 

Library

          url.lua

 

Description

Convert escaped characters in a url back into their raw values.

 

Return

          string unescaped_str

 

Example

> local url = %7e%20Hello%20World%40%21%28ROFL%29"

> print(urlutil.escape(url))

~ Hello [email protected]!(ROFL)