Showing posts with label sqlcl. Show all posts
Showing posts with label sqlcl. Show all posts

Wednesday, November 30, 2016

SQLcl custom Input prompt and validations

Another quick twitter inspired blog post inspired by the SQLcl account itself.



ACCEPT is a great way to get input but validation has to happen elsewhere as there's no hooks to do something like validate a number in a range without running a sql or plsql that does the validation.

As I've covered in many examples on this blog and in our github repo of examples, SQLcl has the ability to do scripting.  Using a few lines of javascript, I can prompt, validate, loop waiting for a better answer from the user.  This is a very simple example that loops waiting for the correct answer printing a message with each failure.  While this is trivial example it shows how easy it could be to have a custom script that prompts for values then with some scripting validate those values and put it into SQLcl's list of variable used for substitutions.












Thursday, November 17, 2016

ECMA Script 6 / Nashorn / Java 9 and SQLcl

This blog post brought to you by the letter M as in Martin .  . He asked me on twitter if SQLcl via Nashorn could use ECMA Script 6 features yet.  The answer is yes.  So, for the brave that want to try out the latest greatest ECMA Script 6 features it can be done with SQLcl.

This is a good reference for what's in version 6 that could be useful.  The only caution is not everything is built into nashorn quite yet.  For Example destructuring is not there.
   http://es6-features.org/#StringInterpolation

The Template Literals are probably to me the most interesting thing since it cleans up all things like string concats and multi line string.


Here's how to get it working:
Step 1: Get Java 9 https://jdk9.java.net/download/
Step 2: Set the enviroment variable : APP_VM_OPTS="-Dnashorn.args=--language=es6"
Step 3: Go.

Complete side note as you can see in my default PS1, apple doesn't and has never understood you don't add an 's to a noun ending in a s.

kriss-MacBook-Pro:bin klrice$ export APP_VM_OPTS="-Dnashorn.args=--language=es6"
kriss-MacBook-Pro:bin klrice$ ./sql klrice/klrice

SQLcl: Release 4.2.0 Production on Thu Nov 17 09:22:35 2016

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Last Successful login time: Thu Nov 17 2016 09:22:36 -05:00

Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> script /Users/klrice/workspace/raptor_common/examples/sql.js
Using Binds:[object Object]

  ***************************************************************
  *****    SIMPLE LOOP OF LIST OF LIST                ***********
  ***************************************************************
 OBJECT_TYPE    OBJECT_NAME 
 TABLE    EMPLOYEES 



  ***************************************************************
  *****    SIMPLE LOOP OF LIST OF NAMES WITH BINDS     **********
  ***************************************************************
 TABLE    EMPLOYEES  
SQL> 


Now this probably looks just like the script from my first blog on how to use javascript http://krisrice.blogspot.in/2015/10/sqlcl-oct-13th-edition.html


However the script itself is much nicer to read ( to me anyway ) . What will stand out is multi line string with using a back tick.  Also is the evaluating of variables inside the quoted string. No more string concatenations!




Wednesday, November 16, 2016

Import APEX apps now easier with SQLcl

Hopefully by now everyone knows there's the ability in SQLcl to use Javascript and the same javascript can be used to make custom commands extending what SQLcl offers out of the box.

Also, I hope everyone knows we are posting examples of this to github as a base for anyone to learn and try it out.  Just in case here's the link to the GitHub location https://github.com/oracle/oracle-db-tools/tree/master/sqlcl

The last example I did was for ApexExport which took the tried and true ApexExporter java utility that ships with Apex and converted it to JavaScript. http://krisrice.blogspot.com/2016/10/export-apex-application-with-sqlcl.html

One of first questions was from Vito about an import counterpart.




Here's that counterpart which is leveraging apex_application_install for setting options to be used. This screenshot is showing loading the command straight from github which I wouldn't recommend if there is ever a need to be offline using the command.  There's a Download/Clone button right on the github space home page then all the file are local and be used anytime regardless of network connectivity.





When the command is called it echos out all the options being used. In the case of multiple workspaces and none being specified right or wrong it's going to use this sql to determine which to use:
  select min(workspace_id) wsID  from  apex_workspace_schemas where schema = user

If that's not correct, no problems just pass in -workspaceid



KLRICE@orcl >apximp  -file f102.sql -offset 123 -name Kris -alias rice  -installSupportingObjects
*** USING DEFAULT WORKSPACE *** 
*** KLRICE *** 
** Importing with the following options **
Application Alias :rice
Workspace ID :1930920493850173
Application Name :Kris
Offset :123
****


The script is here:

Tuesday, November 15, 2016

SQLcl as a library in existing programs

I got a question over the weekend if SQLcl could be leveraged as library from inside a Jython program. This may seem like it's the same thing as adding jython to SQLcl to use instead of Javascript but it's a significant difference. This method allows for the use of SQLcl in any existing program. That means when the program needs to interact with the database, all the code in SQLcl that is tried and true can be leveraged.

This is a simple example to prove it's possible.  First the jar files need to be added to Jython's command line.

# all files in sqlcl/lib/*
# adjust to your install ( unzip ) of sqlcl
LIB=/Users/klrice/workspace/raptor_common/sqlcl/built/lib/
CP=
for  f in $(ls $LIB/*.jar); do
 echo $f
 CP=$CP:$f
done
echo --- $CP ---

#Start up jython 
jython -J-cp $CP 


Now that the needed libraries are in the path. It's quite easy to start using the same methods and classes that have been shown in various javascript examples.  This example gets a connection on it's own then instantiates SQLcl for programmatic use.


#import various things
from java.sql import DriverManager
from oracle.dbtools.db import DBUtil
from oracle.dbtools.raptor.newscriptrunner import *

#plain ol jdbc connection
conn  = DriverManager.getConnection('jdbc:oracle:thin:@//localhost:1521/orcl','klrice','klrice');

#get a DBUtil but won't actually use it in this example
util  = DBUtil.getInstance(conn);

#create sqlcl
sqlcl = ScriptExecutor(conn);

#setup the context
ctx = ScriptRunnerContext()

#set the context
sqlcl.setScriptRunnerContext(ctx)
ctx.setBaseConnection(conn);

#change the format
sqlcl.setStmt('set sqlformat json');
sqlcl.run();

#run the sql
sqlcl.setStmt('select * from emp');
sqlcl.run();


The output of this is simply the json output.








Monday, November 14, 2016

Adding Reserved command in SQLcl


I saw Stephen's example of checking reserved words in the database from Vertan's day and figured I'd do the same in SQLcl.


Checked if something is reserved seems like a nice add on for the tooling.  This example adds a SQL>reserved command that checks all words passed to see if they are in the single source of truth which is v$reserved_words


The outcome is something like this.





The code is in the normal location on github.  It follows the same CommandRegistry/CommandListener that was used in lots of the previous examples.


Monday, October 10, 2016

Export APEX application with SQLcl


APEXExport has been around a long time for exporting an application and anything else like images, feedback,websheets,.. into a file system so that they can be version controlled.  This is a must if there is ever a need to rollback or see what the application was X days ago.  This is a java program that is part of the apex distribution.  The catch for some folks is that it's a java program and being mostly DB / APEX / PLSQL people java isn't something that's normally done which makes it a tad cumbersome.


Now, enter my long flight today with nothing much to do.


SQLcl and Javascript


There's not much that can't be done with SQLcl and Javascript.  Here's the latest example that is probably the most useful example for anyone using APEX.

The benefit is that this is just a command in SQLcl just like the new INFO command or the DESC or anything else.  Just another command for developers to use.  I took the existing APEXExport which was all java around 800+ lines of code and rewrote into javascript which cut the size in half!  The reason for this is that as a java program the APEXExport had to create a connection, take db username|password|tns.  Then it also had to deal with the low lever database calls for looping resultsets and closing them after.

Just as an example, here's the normal JDBC vs SQLcl way to get a single column from a single row into a variable to use.

PreparedStatement stmt = conn.prepareStatement("select user from dual");
            ResultSet rset = stmt.executeQuery();
            rset.next();
            String userName = rset.getString(1);
            rset.close();
            stmt.close();

VS

var user = util.executeReturnOneCol('select user from dual');









To load this just grab the script from github and load it into SQLcl. Once this is done SQLcl will now have the "apxexp" command.  This is a first pass as the port of the code.  If anyone finds an issue, log a ticket or just ping me on twitter @krisrice

SQL>script apxexp


Next I'll port the APEXExportSplitter over.


Wednesday, June 22, 2016

Kill DB sessions the easy way with SQLcl

Seems I can not tweet these animated GIFs anymore. So this is another very short blog post to show real case for adding a new command into SQLcl which can be downloaded here.

It's hard annoying tedious to find SID and SESSION then alter to kill that when needed.  What if there was a simple kill command.

Here a sample one that takes in the 2 options:
  kill sid serial#
  Example :   kill 1 2
OR
  kill USERNAME
  Example: kill KLRICE








Thursday, June 16, 2016

SQLcl and Query Change Notification

The database has had Query Change Notification for a while but to use it required a custom program. Such as Tim outlines on his blog https://oracle-base.com/articles/10g/dbms_change_notification_10gR2


Since SQLcl has Nashorn, now it can be integrated with a few lines of javascript to get notified when changes happen.  The script is below and posted.  The catch is QCN only works on Varchars and Numbers.







Friday, March 04, 2016

DIY SQCL Commands



As mentioned once or twice or 100 times, sqlcl exposes javascript scripting with nashorn to make things very scriptable.  To learn more on Nashorn itself there's a lot of great write ups such as http://www.oracle.com/technetwork/articles/java/jf14-nashorn-2126515.html  So far, the scripting examples have been along the lines of conditional or looping of existing sqlcl commands.

Here's an example of creating a brand new command only from javascript.  This is a pretty simple one that for ALL command will snapshot the start time and print the elapsed time.  It also adds the new command "kris".

Just to show this is really nothing that new to sqlcl, here's a blog post from 2006 about how to make a Java based CommandListener in SQL Developer. This hasn't changed since then.

This all adds up to if we forget to add some feature, you want to override a command, perform something before or after commands, it's very simple to DIY your sqlcl.







// SQLCL's Command Registry
var CommandRegistry = Java.type("oracle.dbtools.raptor.newscriptrunner.CommandRegistry");

// CommandListener for creating any new command
var CommandListener =  Java.type("oracle.dbtools.raptor.newscriptrunner.CommandListener")

// Broke the .js out from the Java.extend to be easier to read
var cmd = {};

// Called to attempt to handle any command
cmd.handle = function (conn,ctx,cmd) {
   // Check that the command is what we want to handle
   if ( cmd.getSql().indexOf("kris") == 0 ){
       ctx.write("Hi Kris, what up?\n");

       // return TRUE to indicate the command was handled
       return true;
    }
   // return FALSE to indicate the command was not handled
   // and other commandListeners will be asked to handle it
   return false;
}

// fired before ANY command
cmd.begin = function (conn,ctx,cmd) {
   var start = new Date();

   // stash something for later like the start date
   ctx.putProperty("cmd.start",start);
}

// fired after ANY Command
cmd.end = function (conn,ctx,cmd) {
   var end = new Date().getTime();
   var start = ctx.getProperty("cmd.start");
   if ( start ) {
      start = start.getTime();
      // print out elapsed time of all commands
      ctx.write("Elapsed Time:" + (end - start) + "\n");
   }
}

// Actual Extend of the Java CommandListener

var MyCmd2 = Java.extend(CommandListener, {
        handleEvent: cmd.handle ,
        beginEvent:  cmd.begin  ,
        endEvent:    cmd.end
});

// Registering the new Command
CommandRegistry.addForAllStmtsListener(MyCmd2.class);

Monday, February 29, 2016

SQLCL Monitoring itself with Longops

Longops is a great way to monitor things that take some time to do work.  There's an easy example of using longops on oracle-base here.  I borrowed the script from there and put it into a file named my_slow_thing.sql.  Now here's a nice example of what's possible with sqlcl.




The easy way to using this is add it to your login.sql which is what I did. Of course the script could run at anytime instead of in the login.sql

script longops.js klrice klrice

The results


Monday, February 01, 2016

Putting SQL in the corner with Javascript in SQLCL

Here's a pretty small javascript file that allows for running sql in the background of your current sqlcl session.  This is a trivial example of a sql script that has a sleep in it to simulate something taking time. It also prints the SID to show it's a different connection than the base.

select 'DB SID ' ||sys_context('USERENV','SID') || ' is going to sleep' bye from dual;

begin 
  dbms_lock.sleep(15);
end;
/
select 'DB SID ' ||sys_context('USERENV','SID') ||' is  awake' hi from dual;







Here's the script as of now and it's in GitHub here


// declare the 2 java files needed
var DriverManager = Java.type("java.sql.DriverManager");
var ScriptExecutor  = Java.type("oracle.dbtools.raptor.newscriptrunner.ScriptExecutor");

var BGsql="";
for(var i=1;i>args.length;i++){
  BGsql = BGsql + " " + args[i];
}

// Create a new connection to use for monitoring
// Grab the connect URL from the base connection in sqlcl
var jdbc = conn.getMetaData().getURL();
var user = 'klrice';
var pass = 'klrice';

runme(BGsql);

//
// running the actual sql
//
function main(arg){
 function inner(){
    // make a new connect
  var conn2  = DriverManager.getConnection(jdbc,user,pass);  
  var sqlcl2 = new ScriptExecutor(conn2);  
  
  sqlcl2.setStmt(arg);
  // run it
  sqlcl2.run();
  conn2.close();
 }
 return inner;
};

// make a thread and start it up
// runs later
function runme(arg){
 // import and alias Java Thread and Runnable classes
 var Thread = Java.type("java.lang.Thread");
 var Runnable = Java.type("java.lang.Runnable");

 // declare our thread
 this.thread = new Thread(new Runnable(){
    run: main(arg)
 });

 // start our thread
 this.thread.start();
 return;
}

Thursday, January 07, 2016

Tuning SQL with Javascript in SQLCL

In case anyone missed it, #sqlcl has the ability to run javascript. This opens a lot of options.  Here's a simple example that shows how to using javascript.

  • Open a new Database Connection
  • Collect stats on the base connection
  • Do work on the main connection
  • Collect stats again
  • Only Print the stats that changed


In SQL Developer, the autotrace feature has for a long time selected the session statistics and display them.  Then you can diff those to another run and see what changes.  Jeff of course has blogs on this That's great but where's it leave a #sqlcl user? Enter JavaScript.

Here's the end result.  I ran a select count on all_objects filtered to DBA_%.  The things to notice is the Work Session's SID and Monitor Session's SID is different and printed to show I didn't cheat.  The command to execute is just passed as arguments to the script command.  The result is the count ran. I got the stats before and after.  Then it's just a simple loop comparing them to see what changed and print on the far right the actual difference.

The JavaSript is below and has comments enough to follow hopefully.  I'll post it to our #sqlcl github examples directory shortly.







// declare the 2 java files needed
var DriverManager = Java.type("java.sql.DriverManager");
var DBUtil  = Java.type("oracle.dbtools.db.DBUtil");


// Create a new connection to use for monitoring
var jdbc = conn.getMetaData().getURL();
var user = 'klrice';
var pass = 'klrice';
//connect
var conn2 = DriverManager.getConnection(jdbc,user,pass);
var util2 = DBUtil.getInstance(conn2);

// probably a better way to rpad a string
function pad(s,n){
   s = s.toString() + "                                                                    ";
   return ( s.substring(0,n));
}

//
// Simple function to grab the session stats and put them into JSON
//
function stats2json(sid){
     var binds = {};
     binds.SID = sid;

     var list = util2.executeReturnList("select name,value from v$statname sn,v$sesstat  s where sn.statistic# = s.STATISTIC# and s.SID= :SID",binds);
     var ret={};
     for (var i = 0; i < list.length; i++) {
             ret[list[i].NAME] = parseInt(list[i].VALUE);
     }
   return ret;
}

//
// Simple function to grab the session details and put them into JSON
//
function getSession(util){
      /* Current active SQL Connection */
      var results = util.executeReturnList("select sys_context('USERENV','session_user') u,sys_context('USERENV','SESSIONID') sessionID,sys_context('USERENV','SID') sid from dual",null);

      var ret = {};
          ret.user      = results[0].U;
          ret.sid       = results[0].SID;
          ret.sessionid = results[0].SESSIONID;

      ctx.write("\tUser:" + ret.user + "\n");
      ctx.write("\tSID:" + ret.sid + "\n");
      ctx.write("\tSession:" + ret.sessionid + "\n");

     return ret;
}

// rebuild the arguments which are the commands to run
var sql="";
for(var i=1;i<args.length;i++){
  sql = sql + " " + args[i];
}

// print the sessions to prove they are different
ctx.write("--Work Session--\n");
var session = getSession(util);

ctx.write("--Monitor Session--\n");
var session2 = getSession(util2);

// grabt the stats before
var before = stats2json(session.sid);

// rebuild the sql from the arguments
ctx.write("Command:\n\t"+ sql + "\n\n");
sqlcl.setStmt(sql);

// run it
sqlcl.run();

// get the after session stats
var after = stats2json(session.sid);

ctx.write("Session Stat Changes:\n\n");

// walk the stats and print the deltas
for(var key in before){
   // print only stats that changed
   if ( before[key] !=  after[key] ) {
     ctx.write( pad(key,64) + "\t\t "+ pad(before[key],12) + " --- "+ pad(after[key],12) + "\t "  );
     ctx.write( pad((after[key]-before[key]),12) + "\n");
   }
}

Tuesday, December 15, 2015

Carbonated SQLCL - aka SODA support

     Oracle Database 12.1.0.2 introduced JSON support.  This is supported in REST Data Services already and now in the latest SQLCL.  This means ORDS can serve up the access to client applications and developers can have a command line way to interact and test features/queries/... in a nut shell a shell for JSON document storage. To use this feature the database will be to have patch #20885778 applied.

KLRICE@orcl >soda

SODA
------

SODA allows schemaless application development using the JSON data model.
  SODA create  
       Create a new collection 

  SODA list 
       List all the collections 

  SODA get <collection_name> [-all | -f | -k | -klist] [{<key> | <k1> <k2> ... > | <qbe>}]
       List documents the collection 
       Optional arguments: 
      -all    list the keys of all docs in the collection 
      -k      list docs matching the specific  
      -klist  list docs matching the list of keys 
      -f      list docs matching the  

  SODA insert <collection_name> <json_str filename=""> 
       Insert a new document within a collection 

  SODA drop <collection_name> 
       Delete existing collection 

  SODA count <collection_name> [<qbe>] 
       Count # of docs inside collection. 
       Optional <qbe> returns # of matching docs

  SODA replace <collection_name> <oldkey> <new_ doc="" str=""> 
       Replace one doc for another 

  SODA remove <collection_name> [-k | -klist |-f] {<key> | <k1> <k2> ... | <qbe>} 
       Remove doc(s) from collection 
       Optional arguments: 
      -k     remove doc in collection matching the specific <key> 
      -klist remove doc in collection matching the list <key1> <key2> ... > 
      -f     remove doc in collection matching <qbe> 

KLRICE@orcl >

There's a lot in there for command so give them a try.  SODA is underpinned but tables with JSON data in them.  It's very easy to get going testing out the SODA features.  Here's a simple collection create, insert, and query.


KLRICE@orcl >soda create ODTUG
Successfully created collection: ODTUG

KLRICE@orcl >soda insert ODTUG {"content":"awesome"}
Json String inserted successfully.

KLRICE@orcl >soda insert ODTUG {"location":"chicago"}
Json String inserted successfully.

KLRICE@orcl >soda insert ODTUG {"best_of_week":"DB Symposium"}
Json String inserted successfully.

KLRICE@orcl >soda get ODTUG -f {"content":{"$startsWith":"a"}}

Key:      7A01B31EBD7C484C88F431B8871DD165
Content:  {"content":"awesome"}
-----------------------------------------
 1 row selected. 



JavaScript and SODA

Here's a quick example of using the SODA libraries from javascript.  Also in here is a new feature of scripting.  The javascript can be inlined into a sql script.  Notice that it's soda.sql <<< .sql that has javascript in it, let that sink in for a minute with the possibilities.

KLRICE@orcl >@soda.sql

Caffeinated SODA 
Dropping the old
Created the old
Inserted Document
Searching...
CBFE3D919E814094954BB217A2F6916E
{ "name" : "Alex", "friends" : "50" }


KLRICE@orcl >!cat soda.sql

script
ctx.write('Caffeinated SODA \n');


var OracleRDBMSClient = Java.type("oracle.soda.rdbms.OracleRDBMSClient");
var OracleDataSource = Java.type('oracle.jdbc.pool.OracleDataSource');
var cl = new OracleRDBMSClient();
var db = cl.getDatabase(conn);

ctx.write("Dropping the old\n")
var foo = db.admin().createCollection("foo");
 foo.admin().drop();
ctx.write("Created the old\n")
    foo = db.admin().createCollection("foo"); 


var doc = db.createDocumentFromString("{ \"name\" : \"Alex\", \"friends\" : \"50\" }");
ctx.write("Inserted Document\n")
 foo.insertAndGet(doc);

conn.commit();

var f = db.createDocumentFromString(JSON.stringify( {"name" : { "$startsWith" : "A" }}));
ctx.write("Searching...\n")
var c = foo.find().filter(f).getCursor();


while (c.hasNext()) {
 cDoc = c.next();
 ctx.write(cDoc.getKey() + "\n");
 ctx.write(cDoc.getContentAsString());
 ctx.write("\n\n"); 
}

ctx.write("\n\n");
/

KLRICE@orcl>


Gory Details of the release

Someone asked about change logs for these SQLCL updates.  Here's a filtered version of the gory details



Thursday, October 15, 2015

SQLCL - Blob loading ALL the files


The github repo for all the things the DB Tools team does will be posted here https://github.com/oracle/Oracle_DB_Tools This will include sqlcl , sqldev, modeler, ords , and who knows what other things we have up our sleeves to post.  This repo is going to be made up of  examples and getting started things like a new extention for sqldev, custom transformation for modeler, ords integration with handsontable, or sqlcl scripting.


Now onto more BLOB loading.

I'm slowly building out a helper.js which is now posted and I'll continue to update it on github here: https://github.com/oracle/Oracle_DB_Tools/tree/master/sqlcl/lib

There's a new function in there to exec host commands and capture the stdout, stderr, and return code of the command.  Now coupling that with the BLOB loading and you can load an entire directory of files into a table with a small amount of javascript.




New helper function

/*
*  Runs the passed in command and returns an Object with
*   .rc      - the return code
*   .stdout  - STDOUT
*   .stderr  - STDERR
*
*/
helpers.exec=function(cmd){
    var RunTime  = Java.type("java.lang.Runtime");
    var Scanner  = Java.type("java.util.Scanner");
    var p = RunTime.getRuntime().exec(cmd);

    var ret={};
        s = new Scanner(p.getInputStream()).useDelimiter("\\A");
        ret.stdout = s.hasNext() ? s.next().trim() : "";
        s = new Scanner(p.getErrorStream()).useDelimiter("\\A");
        ret.stderr = s.hasNext() ? s.next().trim() : "";

        p.waitFor();
        ret.rc = p.exitValue();
      p.destroy();
    return ret;
}


Putting it together with the BLOB loading example

load('https://raw.githubusercontent.com/oracle/Oracle_DB_Tools/master/sqlcl/lib/helpers.js');

/* Clean out the table just to prove it's not a baked demo */
sqlcl.setStmt("truncate table k");
sqlcl.run();

/* File name */
var files= helpers.exec('find . -maxdepth 1 -type f ').stdout.split('\n');

/* bind map for reuse */
var binds = helpers.getBindMap();

for(f in files ) {
   //  ctx.write("Loading : " + files[f] + "\n");
  /* load the blob */
  blob = helpers.getBlobFromFile(files[f]);

  /* assign the binds */
  binds.put("path",files[f]);
  binds.put("b",blob);

 /* Just do it */
  var ret = util.execute("insert into k(path,blob_content,when) values(:path , :b, sysdate)",binds);
}

/* print the results to see if it worked */
sqlcl.setStmt("select path,dbms_lob.getlength(blob_content) from k order by when desc;");
sqlcl.run();



Removing the truncate and select at the end, loading the entire directory of client side files is 7 lines of javascript.



Wednesday, October 14, 2015

AskTom's print_table - SQLCL version

There was an idea logged over on community.oracle.com to add AskTom style print_table to sqlcl.

Here's how easy it is now that there's scripting in sqlcl.  This is a simple js file that loops over the rows and columns and prints them.  When using the executeReturnListofList the first row is the name of the columns for reference.  With that data there it's quite simple to customize any output.



/* rebuild the sql passed in since args is split on spaces */

var sql="";
for(var i=1;i<args.length;i++){
  sql = sql + " " + args[i];
}
ctx.write(sql + "\n\n");


var ret = util.executeReturnListofList(sql,null);

/* loop the rows */
for (var i = 0; i < ret.size(); i++) {
    ctx.write('>ROW \n'); 
    /*loop the cols */
    for( var ii=0;ii<ret[i].size();ii++) {
        ctx.write("\t" + ret[0][ii] + " : " + ret[i][ii] + "\n");
    } 
}

ctx.write('\n\n');




Now running it is very simple.  Script printtab select......



SQLCL - Blob loading from a file

There is a ton of ways having new scripting in sqlcl can help.  One example is a simpler way to load a local file into a BLOB in a column.

Prior to this the only options I know of would be to write custom code or move the files to the server and use utl_file or xdb or some combination of things like this.

Enter SQLCL and javascript method #1.


Here's a simple script which loads a local file into a temp BLOB and uses that as a bind variable in the insert statement

/*
/*
*  Function to take in a filename and add or create it to a map
*  with bind variables
*/
function addBindToMap(map,bindName,fileName){
     /*  conn is the actual JDBC connection */
     var b = conn.createBlob();

     var out = b.setBinaryStream(1);

     var path = java.nio.file.FileSystems.getDefault().getPath(fileName);

     /* slurp the file over to the blob */
     java.nio.file.Files.copy(path, out);
     out.flush();

     if ( map == null ) {
         /* java objects as binds needs a hashmap */
         var HashMap = Java.type("java.util.HashMap");
         map = new HashMap();
     }
     /* put the bind into the map */
     map.put("b",b);
 return map;
}


/* File name */
var file = "/Users/klrice/workspace/raptor_common/10_5.log";

/* load binds */
binds = addBindToMap(null,"b",file);

/* add more binds */
binds.put("path",file);

/* exec the insert and pass binds */
var ret = util.execute("insert into k(path,blob_content,when) values(:path , :b, sysdate)",binds);

/* print the results */
sqlcl.setStmt("select path,dbms_lob.getlength(blob_content) from k order by when desc;");

sqlcl.run();



Running it is as expected.  One new row and showing the blob size.  The only difference here is that you can avoid typing s-c-r-i-p-t and save 2 letters with just scri.




Method #2 shared library

I'll be making a library intended to make some things easier such as file loading and creating the hashmap.  So taking out this blob loading as an example there first 2 things to put into such a library in helpers.js


/*
    This is a javascript library to make some of the common things
    in SQLCL simplier.
*/

var helpers = {} ;

/* for complex type a java hashmap is needed for binds */
helpers.getBindMap = function(){
   var HashMap = Java.type("java.util.HashMap");
   map = new HashMap();
   return map;
};

/* create a temp blob and load it from a local to sqlcl file */
helpers.getBlobFromFile=function (fileName){
     var b = conn.createBlob();
     var out = b.setBinaryStream(1);
     var path = java.nio.file.FileSystems.getDefault().getPath(fileName);
     java.nio.file.Files.copy(path, out);
     out.flush();
 return b;
};


Now the blob loading script can be reduced to something like this

load('helpers.js');

/* File name */
var file = "/Users/klrice/workspace/raptor_common/10_5.log";

/* load binds */
binds = helpers.getBindMap();

/* add more binds */
binds.put("path",file);

blob = helpers.getBlobFromFile(file);

binds.put("b",blob);


/* exec the insert and pass binds */
var ret = util.execute("insert into k(path,blob_content,when) values(:path , :b, sysdate)",binds);




I'll be cleaning up this helpers.js and hosting it soon with frequent additions to new functions.

Tuesday, October 13, 2015

SQLcl - Oct 13th Edition

Another build of SQLCL is out.  This one has lots of great things in it.  There's fix in there to work with Cygwin for the unlucky people that are still on windows.

Now the new stuff:  SCRIPT

This new command can radically change things you can do in the tool.  The script command runs a script that can be any language that is JSR-223 enabled.  The best list I can find is here: https://en.wikipedia.org/wiki/List_of_JVM_languages  So you want to program sql scripting in groovy,perl,javascript,pascal,....  All doable now.

Javascript is the only one built into Java by default so most example will be focused on that although I do have some Jython ones also to post.

To run a script simple type: script <script name>  If there is no file extension passed along it assume a .js  The language is loaded based on this extension.  If the language can't be loaded you get a message saying as much "Could not load Scripting Engine for :py"  More on how to add the other languages later.

Here's a simple script:

ctx.write("Hold my beer, I'm going to try something\n\n");

ctx.write('hi from javascript\n');

ctx.write('This is a count of my objects');
sqlcl.setStmt('select count(1) from user_objects; ');
sqlcl.run();

var ps1= 'set sqlprompt "@|blue  _USER|@@@|green _CONNECT_IDENTIFIER|@@|blue 🍼 >|@"'

ctx.write('Now javascript will change the sqlprompt\n');
sqlcl.setStmt(ps1);
sqlcl.run();

sqlcl.setStmt('@numbers.sql');

sqlcl.run();


This example while simple show you can run commands from inside the script that affects sqlcl itself like changing the prompt.  The simple example is that I connected to production so a baby bottle it swapped out for beer




There's a few globals available for use.  There'll be more formal doc to follow

    sqlcl
        Methods:  
             setStmt(<String of stuff to run>)
                 This can be a single statement, an entire script of stuff, or any sqlcl command such as "@numbers.sql"
             run()
                 Runs whatever is in the stmts

   ctx ( this has tons of methods but this is the single most important )
       Methods:
             write(<String>)

   util ( again tons of methods ) 
       Methods:
              execute(<string>,binds)
                   executes whatever is passed in with a boolean return for success/failure
              executeReturnOneCol(<string>,binds)
                   executes and returns the first row , first column
              executeReturnListofList(<string>,binds)
                   executes and returns an array(rows) of arrays(row).  
              executeReturnList(<string>,binds)
                   execute and returns and array ( rows ) of objects ( row )
              

Here is an example using some of these methods with the expectable output shown below.


/* Look up a single value to use in a bind later */
var user = util.executeReturnOneCol('select user from dual');

var binds = {};
binds.name = 'EMP';
binds.who = user;

ctx.write('Using Binds:'+ binds + '\n');

ctx.write('***************************************************************\n');
ctx.write('*****    SIMPLE LOOP OF LIST OF LIST                ***********\n');
ctx.write('***************************************************************\n');


var ret = util.executeReturnListofList('select object_name,object_type from  all_objects where object_name = :name and owner = :who ',binds);

for (var i = 0; i < ret.length; i++) {
    ctx.write( ret[i][1]  + "\t" + ret[i][0] + "\n");
}

ctx.write('\n\n');

ctx.write('***************************************************************\n');
ctx.write('*****    SIMPLE LOOP OF LIST OF NAMES WITH BINDS     **********\n');
ctx.write('***************************************************************\n');


ret = util.executeReturnList('select object_name,object_type from  all_objects where object_name = :name and owner = :who ',binds);

for (i = 0; i < ret.length; i++) {
    ctx.write( ret[i].OBJECT_TYPE  + "\t" + ret[i].OBJECT_NAME+ "\n");
}





Tuesday, October 06, 2015

SQLcl - Oct 5th Edition

It was time for another push of the new sql command line , sqlcl.  Here's the changes in the latest build.  There are a number of bugs fixed relating to connecting to idle databases, alias feedback, and picking up the $ORACLE_HOME's jdbc driver when available.  There is only a couple functional changes.


First is that now the jdbc connect string itself can be specified.  This opens up a number of options on how to connect. This is list of valid jdbc prefixes.  While I've not yet tested it as noted in this list we will be testing loading sqlcl into the DB and trying it out in the JVM there.
  • jdbc:oracle:thin
  • jdbc:oracle:oci8
  • jdbc:oracle:kprb
  • jdbc:default:connection
  • jdbc:oracle:kprb:
  • jdbc:default:connection:





Next is the ansiconsole formatting of numbers was changed up a little.  The new syntax as 3 options.  One piece of feedback was that the numbers auto formatting with separators made some things difficult.  Now just setting the console formatter will NOT format numbers.  Instead the default option needs to be specified.  Also is the option to specify any number format which has to be in the form of a java format.  The syntax can be read here.


  • set sqlformat ansiconsole
  • set sqlformat ansiconsole default
  • set sqlformat ansiconsole <number format>


This script show an example of all the new options.


The only caution when using the @|... |@ syntax is that only one thing can be used at a time as the, used to separate the formatting options gets read by the java formatter as a number format.  If this is something that folks end up using a lot, I'll come up with something to let it blink and be red at the same time!



Friday, September 11, 2015

SQLCL - More secure, now with REST !

A new SQLCL build was just posted go grab it and kick the tires.  There are well over 100 bug fixes in there so it's better than ever.  Also there's some new things.


More Secure 


  Imagine you have an api that is called and a password or something is used in the parameters.  We use exec MY_API(...)  and it works just fine.  However consider if someone with access to v$sql they just got anything you passed into the api and quite easily.  Enter SQLCL new thing #1 starting with this build now changes all "exec my_api(...)" into an anonymous block with binds so those literals do not show up.



Procedure SECURE_API compiled

KLRICE@orcl >exec secure_api('my password');


PL/SQL procedure successfully completed.

KLRICE@orcl >select sql_text from v$sql where sql_text like '%secure_api%';


SQL_TEXT                                                       
BEGIN secure_api(:ZSqlDevUnIq1); END;                          
select sql_text from v$sql where sql_text like '%secure_api%'  


KLRICE@orcl >




REST Support

Always check the help, the tool highlights what is new to make it easy.  It only took Alex 8 minutes from the download being available to seeing the new features !



 This build adds the ability to extract ORDS services.  Once a rest service is defined and it's time to move it from one system to another this will make it easier than ever.  Simple spool to a file, export the definition and instant install script.

KLRICE@orcl >help
    ....
    REST
    ...
KLRICE@orcl >rest

REST
------

REST allows to export ORDS 3.X services.

      REST export                     -  All modules
      REST export >module_name>    -  Export a specific module
      REST export >module_prefix>  -  Export a specific module related to the given prefix
      REST modules                    -  List the available modules
      REST privileges                 -  List the existing privileges
      REST schemas                    -  List the available schemas
KLRICE@orcl >rest modules

NAME   PREFIX  STATUS     ITEMS_PER_PAGE  
demos  demos/  PUBLISHED  25              
hr     hr      PUBLISHED  25              


KLRICE@orcl >rest export hr

declare
  l_module_id number;
  l_template_id number;
  l_handler_id number;
begin

  l_module_id := ORDS_METADATA.ORDS_SERVICES.create_module( p_name => 'hr' , p_uri_prefix => 'hr' , p_items_per_page => 25 , p_status => 'PUBLISHED' );
  l_template_id := ORDS_METADATA.ORDS_SERVICES.add_template( p_module_id => l_module_id, p_uri_template => 'everything' , p_priority => 0 , p_etag_type => 'HASH' );
  l_handler_id := ORDS_METADATA.ORDS_SERVICES.add_handler( p_template_id => l_template_id, p_source_type => 'json/query' , p_method => 'GET' , p_items_per_page => 0 , p_source => 'select d.*,' || unistr('\000a')
   || '' || unistr('\000a')
   || '   cursor( select e.*' || unistr('\000a')
   || '           from hr.employees e where e.department_id = d.department_id) emps           ' || unistr('\000a')
   || '   from hr.departments d');

commit;
end;
KLRICE@orcl >

Tuesday, June 16, 2015

SQLcl , yet again

By the Numbers


There's a new SQLcl out for download.  In case, there are too many to keep track of the build numbers are quite easy to tell if you have the latest.  The build posted today is  sqlcl-4.2.0.15.167.0827-no-jre.zip

Here's what we are doing
4.2.0 <-- doesn't matter at all
15     <- year
167   <- day in julian
0827 <- time the build was done

So yes, this build was done today at 8am !


Info on PL/SQL

I don't think I covered this before.  The info on any plsql gives you actual useful stuff.  It prints out a stub of pl/sql that can be copy and pasted to get going much quicker than looking at a describe tabular output of args.  I'll pick on dbms_metadata which is a huge package.



This can still be way too much 'info' if the packages are very large.  The other part of info is that you can get a subset.  Here the ".a" was added to filter it down.  I'll blog more on why I'm showing metadata as there's some pretty interesting usages that I just found how to use.



Show All

Not much to see here.  Just looks like it should.

Show All+

Much like the info/info+ we added lots more.  Just scroll over this or try it out and see how much more is there.  It should give you just about anything you may be interested in.




KLRICE@orcl🍺 >show all+

Show All:
========================================
appinfo is OFF and set to "SQL Command Line"
arraysize 15
autocommit OFF
autoprint OFF
autotrace OFF
BTitle OFF
colsep " "
concat "." (hex 5c)
copycommit 0
define "&"
echo OFF
escape OFF
FEEDBACK ON for 6 or more rows
heading ON
headsep "|" (hex 7c)
linesize 136
long 80
longchunksize 80
null null
numformat ""
numwidth 10
pagesize 44
PAUSE is OFF
release 1201000200
scan ON
serveroutput OFF
space " "
spool OFF
sqlcode 0
sqlprompt "@|red KLRICE|@@@|green orcl|@@|blue 🍺 >|@ "
suffix "sql"
termout ON
time OFF
timing OFF
trimout OFF
trimspool OFF
TTitle OFF
USER is "KLRICE"
verify ON
wrap : lines will be wrapped

Show All+ (Single Line):
========================================
_restrict 0
buffer SQL
CD : Default
commandline true
con_id 3
con_name ORCL
encoding "UTF-8"
LDAPCON : Default
net: ON
noverwrite: WARN
null null
SQL Format : ansiconsole
SQLPATH : /Users/klrice/workspace/raptor_common/:.
systemout "true"
Toplevel TRUE 
url top_base_url ""    base_url ""    node_url (Parent) "file:/Users/klrice/workspace/raptor_common"
xquery BASEURI "" CONTEXT "" NODE DEFAULT ORDERING DEFAULT

Show All+ (Multi Line):
========================================

CONNECTION:
========================================
CONNECTION:
 KLRICE@jdbc:oracle:thin:@localhost:1521/orcl 
CONNECTION_IDENTIFIER:
 orcl
CONNECTION_DB_VERSION:
 Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
 With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
NOLOG:
 false
PRELIMAUTH:
 false

DDL:
========================================
STORAGE : ON
INHERIT : ON
SQLTERMINATOR : ON
OID : ON
SPECIFICATION : ON
TABLESPACE : ON
SIZE_BYTE_KEYWORD : ON
PRETTY : ON
REF_CONSTRAINTS : ON
FORCE : ON
PARTITIONING : ON
CONSTRAINTS : ON
INSERT : ON
BODY : ON
CONSTRAINTS_AS_ALTER : ON
SEGMENT_ATTRIBUTES : ON

DEFINES:
========================================
DEFINE _DATE =  "16-JUN-15" (CHAR)
DEFINE _CONNECT_IDENTIFIER =  "orcl" (CHAR)
DEFINE _USER =  "KLRICE" (CHAR)
DEFINE _PRIVILEGE =  "" (CHAR)
DEFINE _SQLPLUS_RELEASE =  0401000000 (NUMBER)
DEFINE _EDITOR =  "vim" (CHAR)
DEFINE _O_VERSION =  "Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options" (CHAR)
DEFINE _O_RELEASE =  1201000000 (NUMBER)
DEFINE _PWD =  "/Users/klrice/workspace/raptor_common" (CHAR)

INSTANCE:
========================================
USERNAME KLRICE
INSTANCE_NAME cdb1
HOST_NAME klrice-nsvpn-dhcp-141-144-19-110.vpn.oracle.com
SID 21
VERSION 12.1.0.2.0
STARTUP_DAY 20150616

INTERNALERRORS:
========================================
sqldev.error "false"
sqldev.error.any.in "false"
script.runner.autocommit.errorflag "" 
sqldev.last.err.message "" 
sqldev.last.err.message.forsqlcode "" 

JDBC:
========================================
-- Database Info --
Database Product Name: Oracle
Database Product Version: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
Database Major Version: 12
Database Minor Version: 1
-- Driver Info --
Driver Name: Oracle JDBC driver
Driver Version: 12.1.0.2.0
Driver Major Version: 12
Driver Minor Version: 1
Driver URL: jdbc:oracle:thin:@localhost:1521/orcl

NLS:
========================================
DB_TIMEZONE +00:00
NLS_CALENDAR GREGORIAN
NLS_CHARACTERSET AL32UTF8
NLS_COMP BINARY
NLS_CURRENCY $
NLS_DATE_FORMAT DD-MON-RR
NLS_DATE_LANGUAGE AMERICAN
NLS_DUAL_CURRENCY $
NLS_ISO_CURRENCY AMERICA
NLS_LANGUAGE AMERICAN
NLS_LENGTH_SEMANTICS BYTE
NLS_NCHAR_CONV_EXCP FALSE
NLS_NUMERIC_CHARACTERS .,
NLS_SORT BINARY
NLS_TERRITORY AMERICA
NLS_TIMESTAMP_FORMAT DD-MON-RR HH.MI.SSXFF AM
NLS_TIMESTAMP_TZ_FORMAT DD-MON-RR HH.MI.SSXFF AM TZR
NLS_TIME_FORMAT HH.MI.SSXFF AM
NLS_TIME_TZ_FORMAT HH.MI.SSXFF AM TZR
SESSION_TIMEZONE America/New_York
SESSION_TIMEZONE_OFFSET -04:00

PDBS:
========================================
CON_ID  NAME  OPEN_MODE   RESTRICTED  
3       ORCL  READ WRITE  NO          


SGA:
========================================
Total System Global Area   838860800 bytes      
Fixed Size                   2929936 bytes      
Variable Size              545262320 bytes      
Database Buffers           285212672 bytes      
Redo Buffers                 5455872 bytes      

KLRICE@orcl🍺 >